If you are put in a position of defending yourself in the larger group, you can begin with Nemesis' allegation that you are "really a man." It is not precisely true, and it gives you an opening to explain who you are "really."
The whole gossipy event serves to prove that Nemesis is really not a lady.