I'm the exception, I'm sure. I have a distinctively female side and a distinctively male side, and each one likes equal time (oh Lord, I sound like Sybil).
Anyway, I used to enjoy dressing like a girl when I was younger and also dressed like a boy when I got the urge, but when I stopped wearing boys clothes (in my late 20's / early 30's) due to negative reactions, I also stopped wearing dresses, skirts, pantyhose and heels (although I've always hated pantyhose - I agree that they're torture devices).
The strange thing is that I didn't even make the connection. It wasn't until just recently when I decided to resume my cross-dressing and to hell with anyone who has a problem with it, that I realized I had stopped wearing distinctively female clothing just about the same time I stopped wearing boys clothes.
Now that I've slowly resumed cross-dressing, I've also started to think about getting exclusively feminine clothes again, although I don't feel ready to buy a skirt yet, but I can feel that the day will come eventually.
I hope that answers the question without much confusion - but I'm still confused about these things myself.