Nobody gives a second thought when a lady is in the men's section because, for one, a lady in the men's section is generally shopping for a husband too lazy to do it himself, and two, it's a bit uncommon to see women acting as pervs in the men's section, while it happens regularly the other way around. Let's be realistic, there is no double standard here. Just simple facts of a woman's life that the average male simply has no idea about.
If, in male mode (always when I'm out), I sense that I am annoying a lady in the eponymous section, I back out, not because I'm not legitimate there, I am, but just because I don't want her to stress while picking undies or whatever. On the other hand, I don't think any male ever got a sweat because a lady was scouting the racks next to him.
This is very akin to the male/female point of view discussion in a recent thread from Patience. Males, CDers or not, don't seem to be able to see the world with a woman's eyes, which in my opinion explains all these "double standards" comments.