Good question. I think there are three possible outcomes based on my reaction in this sort of situation -- 1) I can escalate the hostility by reacting in kind; 2) I can try to make them see the hurtfulness of their behavior in a non-aggressive way; or 3) I can (hopefully) defuse the situation entirely by simply ignoring the insult.
I nearly always elect to ignore the insult and go about my business because I perceive that nothing is to be gained by doing anything else. I strongly believe that the best way to advance the cause of acceptance is through one-on-one interaction, but the notion that I'm gonna change some stranger's mind with a few seconds of witty repartee is wildly optimistic, imo.
If someone is just glaring I am likely, like you, to make eye contact and respond with a smile, even when I don't feel like it, just cuz that's the mature thing to do. There was one incident where two people were making fun of me in voices intentionally loud enough for me to overhear and I basically told them that as difficult as it might be for them to believe, I am a person just like them and what they were saying really hurt my feelings. They just smirked, but they did lower their voices -- probably out of fear that I might start crying.
The only situations where I try to overcome the hostility in a gentle way -- and I've done this a number of times, usually at one of the clubs -- is choosing to overlook negative signals from someone I know disapproves of me and go out of my way to be nice and engage them. I don't challenge them or even mention their disapproval, I just try to help them see through normal interaction that I'm a real person, and I'm a nice person, and I'm much more "normal" than they might think. Over time, this works better than you might think. I've had several of them come around.
One tactic I know won't work -- and by "work" I mean change a person's attitude -- is to get angry and snarky. Impugning someone's masculinity, accusing them of latent homosexuality or implying they might be harboring latent feminine tendencies themselves isn't going to accomplish anything except piss people off and make them despise me more.
Only one time have I ever responded in anger. I was at a park one Sunday afternoon and this hillbilly-looking mf about 50 yards away started yelling insults at me. Really pissed me off. But I am in no way tough (as in fighting tough), so I couldn't do what I wanted to, which was knock the few teeth he had out of his hateful mouth. Instead, I got in my car, rolled down the window, shot him the finger and drove off. I could see him in my mirror about to have a stroke and I laughed. It didn't take me long to start feeling like a jerk for lowering myself to his level, and because I knew I had only succeeded in throwing gasoline on the fire. But there for a minute, I tell ya, it felt great. :D