its just so easy to yell things out when your in a group or far away from the person
This could happen even if your not out dressed. People do this sort of thing for absolutely no reason, especially with a group or a couple of friends, just to make themselves feel superior. They will never say it to your face or if they are alone. They are words, nothing more, you either make an issue out of it or you don't.
When I go out, if I am dressed, I am going to use a women's room or gender neutral restroom. I am not going into the men's room, that's like searching for problems, comments, or conflict. Thankfully I have never had to use a restroom while I am out. I understand your feeling uneasy about it though.
We all like to think the world is changing but the reality is it hasn't much. It's still the same BS you dealt with in high school because that is the mentality level of a lot of people out there. They have been born and bred to think a certain way and by god that is what they are going to do.
I guess you have 3 choices while you are out, use the right restroom as you present or deal with the BS, wait till you get home, or piss yourself. Your not going to change the world and I see no other way to deal with it. For the record though, while they may be a few, I don't think anyone here looks at using the women's room as any golden trophy, bragging rights, or anything along those lines.