Most of the time we control how much hits the fan. When some gets by us, we can control what does hit it by either reacting truthfully, deflecting it with lies or jokes, or putting it off til we're ready.
The more we are comfortable with what we do, the less of a problem it is.
And we can not read something into everything every body says as being knowledge that they know all about our "secret" of dressing up. We can not use our "secret" as being the reason we are turned down for a job or rejected for a friendship.
Unless, we tell straight out that we are a crossdresser, most people would have no clue (unless we go around in a skirt, etc., as our male selves).
And, if we do that we have no need to be paranoid.