I don't think the analogy is valid. Women took a piece of the male wardrobe (pants) but converted it into a female item. Although there are unisex pants (and t-shirts, and jackets, etc.) there are also clearly female versions of them. Crossdressers try to look like a woman. Most women don't try to look like a man.
And where is your workplace? Is it a corporate office in New Yawk City or a construction site in Lagos, Nigeria?
I don't think the flavor of crossdressing we practice will ever be truly popular. Accepted, or tolerated, in urban environments, maybe. Now if men as a whole adopt a piece of feminine clothing -say skirts- and turn it into an ordinary piece of male clothing, then yes. That's what happened with small earrings in the 80s, and yet if you wear a long dangling earring today you'll be labeled as a cd.