From the above, I am not sure whether you present as female with makeup, wig, etc, or do you present 50/50? If the latter then I suspect that your best option is to use a gender neutral toilet such as separate disabled facilities.
If you are presenting female then I disagree with the advice you were given. It isn't about how you see yourself, i.e. as a male who happens to dress, but about how others perceive you. If you present female use the female facilities unless there is a law against it. No trophy will be awarded - just practical common sense and personal safety. Be thankful the men were 'cowards' - the alternative might have been more painful.