I agree. The word transvestite when translated literally means "crossdresser." The only real difference between the two words is that transvestite is more often used in a derogatory sense, maybe because due to its latin origins it sounds more like a clinical term.
I don't think people should just assume that because someone considers themselves a transvestite it means their dressing is partially to get off. Transvestic Festishism and Transvestism aren't the same thing.
Though the only time I do see a distinction in the words is that I often view "Crossdressing" as more of an umbrella term for wearing clothing of the opposite sex, where as transvestism always to me seems like actually dressing as or behaving as one. I wear women's clothes as a man, and I view it as being a crossdresser but calling myself a transvestite never really has clicked for me.
But then Eddie Izzard blows that whole argument out of the water for me when he presents himself as a man in women's clothes but calls himself a transvestite. (ok, an "Executive transvestite.") so maybe it's a generational thing?