Very true. There's quite a spectrum. Some people get a huge sexual thrill out of it and would never identify as transgendered. Some people are very transgendered and would never say that it was a turn-on for them. Certainly most of us are somewhere in the middle.
I have nothing bad to say about the people who do totally sexualize it, except that I'm horribly offended by them being impolite to me at a club as they often are -- as though they think that since I'm dressed like I am I must share their feelings and be a total pervert.
A lot of people use crossdressing as a sexual outlet that amps them up then years later it becomes more of a zen-like thing that calms them down. I know I got into my mom's stuff when I was very young, like 3 to 6, and that was certainly not sexual, but every time I got into my mom's or GF's stuff from about 10 till 20 (which wasn't often) it was a huge turn-on. Now it's much less like that, but there's certainly excitement to feeling attractive and feminine.
The people who fascinate me the most are the Tri-Ess crowd who claim they are not at all transgendered but also claim that they aren't turned on by it. If that's the case, they sure have picked an interesting hobby!