There can be more than one thing going on when you crossdress, and though it seems like a contradiction, it's really not.

A heterosexual male brought up with the imagery of modern society is prone to being turned on by sexy lingerie, the curve of a bosom etc. As a teen, when hormones are raging, these feelings and the inevitable consequence are intense. If, for whatever reason, this coincides with a desire to CD (and taking the opportunity to do so), then it makes a strong mental connection. This is the start of the classic fetish CD. Fetish by definition is the need for some specific additional feature in order to become aroused, though it is often used with a slightly looser meaning.

Now what if the cause of the desire to CD was not actually the sexual aspect at all. Gender roles can be quite restrictive depending on the community in which you are brought up. It's by no means one sided (oh, don't we guys have it tough?) but there are pressures on males to be physically and mentally tough, and repress gentler emotions. The degree to which this affects your actions should you find that you're not a neat fit with the stereotype vary too.

Do you enjoy watching chick-flicks? Do you feel able to say to your mates that you enjoy chick-flicks? Do you feel able to say to yourself that you enjoy chick-flicks? Why is it not OK for me to be pampered? Why can't I get dolled up and pretty too? Cross-dressing is a way of giving yourself permission to experience some of the things that females have by rights. The most obvious (duh!) is the clothing itself, it's pretty, it's tactile, and it represents femaleness. It flicks a switch in the mind that then gives you permission. It removes the barriers that you have placed there with the guiding hand of society to define what you should or shouldn't feel, enjoy, or participate in.
It may not be femininity, or how a female would actually feel, because you still have a male brain (assuming you're not TS) and male hormones (after all it's not entirely society's doing), but nevertheless it has a powerful effect.

So in conclusion, let your male self enjoy the release, then give your whole self permission to enjoy being you.