Attitude is everything! You must believe in yourself if you want to go out to the world. Others can sense your insecurity and that gives them the opening to demean you.
If you accept their judgements, then that is on you. I prefer to consider that their reaction to me reflects their insecurities and that allows me to stare them down.
To restate what Char has said, you can only control your actions.
As for safety, I stop being a lady once there is a physical threat.