Just for fun, I'm going assume that the title of this thread is a legitimate question, not a rhetorical one.
What is everyone's problem? Homophobia.
I'm not assuming you're gay. They are assuming that. And they're afraid of you because of that assumption.
In modern society, men have power. And like all power, people expect that power be exercised "properly." Men have a dress code, where women (who as a gender are not as powerful) do not have as much of one. Women can wear anything they like. Men have to dress a certain way, like people in uniform, because they have power. (Please don't take this as a slur on women. There are lots of individual women who have power because of occupation etc. and they have dress codes too.) If you violate your dress code, people feel they have a right to know why. And a man in a skirt violates the dress code, but it isn't clear enough why he's violating it. He's making a statement maybe, but not a clear one. If he dresses and acts more like a stereotype (hippie, effeminate gay, clown, guy who is so rough and tough he can dress however he wants, etc.) there won't be the stares.
Before I went full time, I used to dress androgynously a lot. I got stared at a lot too. I learned to stare back, with a kind of "What's your problem?" look in my face. I stared them down, in other words. It's a good skill to learn--it comes in handy for crossdressers. Women in miniskirts have to do the same thing. So do women who are dressed waaaaay out of fashion. In fact, I noticed that I had to do it less when I crossdressed than when I was dressed androgynously! When I went full time, life became easier (than dressing androgynously).
I figured out that by staring the starers down, I was conforming to the last of my stereotypes two paragraphs back--I was being the guy who is so rough and tough he can dress however he wants! It works. But wasn't I going round and round the gender tree?
Blessings,
Rikki