If you think nobody cares, you are fooling yourself. Most people aren't going to make comments to your face, but they do notice and they do form an opinion of you. When they get home or to work, they will tell people that they saw a weirdo man wearing women's clothes. They might secretly take your photo and post it on Facebook.
Yes, you have the right to wear whatever you want, but you have to be willing to accept the consequences.
