Why are you afraid? Simple -you're worried that something will happen that will harm you in some way. The actual question is what are you afraid of? It seems unlikely that if you get out of your car in a skirt all traffic on the street is going to come to a screeching halt, so that's probably not it. I'd say the two biggest problems are inexperience (you don't know what will happen, so you start thinking anything might happen.) Or internally you think you're doing something wrong and you're expecting divine lightning (or an agent of the divine) to strike you down. Technically that's called internalized transphobia and it's a product of your upbringing. You have to learn that you're OK and the people you trusted to help you form opinions while growing up, though perhaps loving, were wrong. That's why gender therapists are so popular -- you have to work your way out of that.
Best answer is do what makes you feel comfortable. If what's comfortable is not what makes you happy, figure out why and correct it. It's a life-long process.