Agreed, but as a metaphor to explain to certain audiences what trans is about the statement can be of use. Eddie Izzard has used it in chat shows.
When I was challenged by a lesbian about why I didn't fancy men I had to point out to her that she didn't, so why should I?