Your best bet is to leave the restroom. Its neat that y'all have never been harassed in a public restroom, but this does actually happen fairly often. Passing privilege avoids it, but not everyone has it. And if you don't believe that a mother with a young daughter with her in a restroom can react with great hostility if she perceives someone to be male in the women's restroom, then you are kidding yourselves.
http://williamsinstitute.law.ucla.ed...mss-june-2013/
In the survey above, 70% of the trans respondents were harassed in a public restroom.
BTW, if you still think this is a trivial matter, many states are trying to pass legislation to criminalize trans access to public restrooms. Texas, where I live is one of them. If the law here passes, we could be fined up tto $2000 and spend a year in jail for using the women's room. Practically speaking, they'd never get me, as a cop who for some reason questioned me in a restroom would likely look no further than my ID, which says F. For a CD, though, y'all could be arrested. A number of trans people are lobbying to kill these horrible laws around the US.
@Isha - 7% of the US population knows someone who is openly trans. We are not understood or trusted by many. Good luck with relying on the kindness of strangers!
All of this stuff is highly location dependent. In Dallas, gender identity & expression are protected, and you have a right to use gender appropriate facilities that match your expression. In small towns outside of a metropolitan area - good luck with that!