Somehow what may start out as an innocent term gathers a fat baggage of meaning and can become insulting to some. Why did the term transvestite fall out of use to be replaced by cross dresser. They mean the exact same thing philologically but one is felt to be somehow demeaning while the other, at least for now, is felt to be OK.
Remember when Gov. Shwarzeneger ( apologies for misspelling) called some people "girly men". That struck me as being insulting to men who might seem less masculine to some but also to women as the use of the term "girly" was intended as a pejorative. That was apparent in the tone of voice that he used. The strong implication was that being very feminine was somehow weak or bad or perhaps a sign of moral cowardice.
Language is a subtle and also a very powerful thing. We should always be sensitive to the nuances.