It is certainly going to vary among each of us and our tastes. I love to dress up too. A lot of women do also. It doesn't mean they do it every waking minute. Since I do dress probably >90% of my non-work time, it isn't going to be all skirts and dresses. There is a lot of enjoyment in putting together outfits that work for all occasions and environments. I would also think that another factor is going to be if and how much one of us gets out and about. You most certainly can wear dresses and skirts all the time and be fine but it for those who want to, it is easier to blend in some situations by being in pants.
The key from all of this? Be yourself and enjoy. Life is suppose to be about being yourself and being happy. The rest is noise.