Yes, it's an inner struggle of his church doctrine vs. his lust for Esmerelda. The general distaste for the gypsies comes from the fact that they were the minority group that was easily blamed for all the city's problems. As an authority figure he used that prejudice to control the populace. He's obviously a really messed up guy, one of Disney's most tortured villains.