I think any realist will acknowledge that there are a wide variety of potential threats. But we often exagerate legitimate fears and conjure others from thin air. The question of probabilities comes into play. Some things are very likely to happen - such as an SO stumbling upon a person's "stash". Others, such as a violent encounter with a transphobe are really quite improbable. And of course there are always mitigating circumstances. For example, bad things are more likley to happen during late nite excursions through the seedier parts of town than during a trip to the mall.
In the end, fear is simply a reaction to one's perception of risks. But we have the capacity to manage risks. That's the logical response to fear.
