I've seen women wear casual outfits which were a hybrid of male and female clothing. This means that they were technically crossdressing. I would love to be able to merge my wardrobe in such a way and not have to care about what's what. It seems that most intolerance comes from the perceived perversion of a Male, existing in public (out as opposed to closet) wearing anything that was intended specifically for a Female.
A kilt may technically fall under the definition of skirt, but isn't generally seen as one given it's past. Which makes it more accepted in society compared to a male wearing a pinkish floral skirt.
Back to women wearing casually mens clothing.
There are few things aside from maybe a jockstrap that are specifically made just for men or are considered to be male only. Womens pants, mens pants. Womens skirt, mens.... kilt. Womens dress, mens..... ???
I know that in the past it was unheard of for a woman to wear pants. This just as recent as the Dick Van Dyke show where there were a limited number of times Mrs. Petrie could wear pants. So, if the past is a precedent for the potential evolution of the future, tolerance of men wearing womens clothing is not far into the future provided the argument is sound.