Always figured it was Charles- he's too trapped in the worst aspects of what Harry called "the institution" to be able to do what he should do- just be a freaking grandparent who accepts his grandchildren unconditionally. He simply couldn't LET himself accept the fact that it doesn't matter what Archie's skin color is.