If I thought that because I love to dress in female clothes every other male should do too, I would have to also accept the reverse: males thinking I'm a weirdo doing it since they would never have this idea for them.
But I want people to accept that I may be different from them. Well then, in turn I have to accept that they may be different from me.
So, no, I don't think every guy wants to or has worn female clothes, and like someone rightly said, even for those who did, their motivations may be entirely different from ours (as CDers).