In a word, yes it does matter. A few years ago I had to find some formal wear to wear to a company Christmas party. I went to several stores trying to find slacks, a button up shirt etc.. I just couldn't find anything that suited me.
I finally went to the women's section and found right away a pair of slacks and a nice button up blouse that could pass as male , unless you really looked at them, and knew what you were loking at.
I wasted alot of time and aggrivation, when I should have went straight to the women's section in the first store I went to.