I agree, it does seem a bit weird that he's smiling all the time unless he's a total a$$hole (which it sounds like he is) and I also think that this is a rare thing to have happen. I have been underdressing at work for 12 years and I have never had a single thing ever happen, not even the slightest weird look or a casual comment never mind an accidental uncovering. I also think that the "it never happened" stance is a good one. It's just your word against his if he takes it further. Stay confident and assertive and insist you don't know what he's talking about and if he is a bit of an a$$hole, people will know that and he'll make himself look like a fool or a liar ... or both.