I am sure you must have heard of the word Stigma.
One entry says Stigma is - a strong feeling of disapproval that most people in a society have about something, especially when this is unfair
Another one is even more brutal - a mark of disgrace associated with a particular circumstance, quality, or person.