That's effectively what I say to people about 2 or 3 times a week. I work retail, and amongst other things, we sell clothing. For the most part, you can consider it "technical" clothing. When we don't have the item in the men's section, I will sometimes suggest trying the female one. Often, they are the same thing, but with different cut. With stretch fabrics, the fit has enough "give" that it can work. Occasionally, it works better. Typically, the ladies stuff has shorter inseams and sleeves, so that can work in your favour. Happens the other way round as well, some of the very tall girls around here end up buying the guy's stuff, otherwise parts of their anatomy are not covered. Reactions vary, a lot just shrug it off, some say "No, don't think so..." or similar, but about half go "That's an idea.." One guy came back to me yesterday and told me the pants I gave him to try were a perfect fit, he was very happy.
Just struck me as funny the other day, that what I was suggesting was just, "You should try being a crossdresser"