That's exactly how the Nazis, the individual people who were Nazis, were about the Jews in the Third Reich. Goebbels complains about it in his diary. When they passed the Nuremberg Laws, everyone came to him with their exception -- they hated the Jews as much as anyone, they said, but what about their neighbour that very nice Mr X, he was a Jew but he was different. Didn't stop him ending up in a camp, but that's how they felt.

This is why it's important not to go along with being accepted as an exception but to identify clearly as what you are and that there are lots of others too.