Krisi, I would suggest that it is because they are NOT trying to pass as female.
What is funny about your comment is that it is written from a "wants to pass" cross dresser's perspective. If a woman were writing this, she may write, "I am curious why a man would walk around town in a skirt." It's just perspective.
