Well in my case, I worry about the neighbours finding out because I just don't want any hassle, I worry about friends finding out because I'd be afraid of losing them & I worry about my job because I need to make a living.
It's sad, but society in general is just not as accepting as we'd like it to be.