For quite a few years I pushed the envelope HARD. I looked like a colour blind gay peacock, and freely mixed what is considered acceptable male/female attire and demeanor.
Perhaps it is because I was in Toronto, one of the world's most liberal cities. Maybe people talked about me after the fact. When you look like Liberace on drugs, you tend to ignore things like that.
Bottom line...nobody commented (to my face), nobody cared, and, contrary to what I was attempting, I really don't think anybody noticed.
Caveat- I would not try this in some of the crazier states.