For me it's the risk of being seen by someone I (we) know who's opinion we might care about.
I recently came to terms with exactly who I am, after spending most of my life feeling that my dressing meant there was "something wrong with me", so the idea of going out and being seen - by strangers - actively excites me. I don't fear people's reactions, nor have any physical worries (I wouldn't put myself in a dark alley and can handle myself one-on-one).
My only concern is people that know me and how them knowing would effect the life of my wife and kids.
If that weren't a consideration I'd be going out fully dressed today. As it is, I need to be more careful, at least right now.