Assuming you are not wearing a flashy, look at me blouse, I agree that other drivers would be oblivious to the style of your top. Just keep it blendable or androgynous and you will be fine. How many times have you paid attention to the shirt of the male driver in the car next to yours? And, on the off chance you noticed he was wearing a womans shirt?
Exactly.
Nothing would come of it.