Ilene its really not that bad. While many of us have stories of bad experiences, we can all agree they are rare.
With that being said I do think some crossdressers bring it upon them selves and are "asking for it". By this I mean they do not dress appropriately for wherever they are going out in public. Instead they dress in a more fetishized manner. I.E. wearing 6 inch stiletto heels and an ultra short mini skirt or dress riding up their bum when they are just going out to the grocery store or to run errands. Now that is an exaggerated example but what I am trying to say is if you are not dressing appropriate for your age and the occassion, you are inviting negative reactions your way. While GGs will get the same negative reactions for not being dressed properly and appropriately, your negative reaction will be compounded.
As long as YOU do everything right, you are in good shape.