I think that our choice of wearing female clothing is a statement (or a declaration) to many people, whether we want it to be or not. Not necessarily OUR choice of statement, but none the less we seem to get tagged with it. In short, it's not just the clothing or our appearance, it's the other's interpretation of our preference of clothing or appearance.
In an old story (of which I cannot recall the name), there was a conflict between two fictional societies. One society cracked their boiled eggs in the morning on the big side, the other on the little side, and the side they cracked it on was placed up in their egg cup. Sometimes that's all it takes.
That's my take on it.
