It would have been better not to answer any of his questions IMO. Every answer you gave sparked more of the same type of questions from him. Of course, if I were actually in that situation I can't say how I would have reacted.
One method is to answer a question with a question. "What are you doing working over here. Oh, well I suppose you wouldn't look bad in a dress". What makes you say that? Now the pressure is on him to answer. - Or change the subject completely, maybe back to command hooks.
I like the axiom, "don't complain, don't explain". Explaining why you're in that department isn't part of your job description. You didn't have to do it.