Wow! Thanks for all the feedback! Just to clarify a bit...
When he made the comment "this must be for your wife" I didn't answer rudely, I just said it was for me (even though is actually wasn't). I think in the moment I just though what business is it of his. I still get nervous shopping, and if I had been buying it for myself his comment would have made me feel very uncomfortable.
I should have made it clearer too that it was his tone when he said "this is a woman's scarf" that really got under my skin!
I didn't respond in a rude tone, or raise my voice, I just made it exactly clear I knew what I was doing!