Caution - don't speak for all of us. I find the word transvestite offensive. I consider myself a crossdresser.
The words mean different things to different people, so to say they are the same is not accurate. "Transvestite" is an outdated word that sounds like a disease to me. Crossdresser more accurately describes what I do without the "added baggage."
I don't have issues with labels per se, but I do prefer to be the one labeling myself.
As a policy wonk and someone who deals in foreign affairs, I can assure you that words mean a great deal - parsing them up into pieces to trace origins is one thing, but words also have significant cultural meanings not always understood by everyone.