He knows you had to be buying them for a female acquaintance.
Don't worry just have an inplausible explanation next time you see him.
Tell him you are going upmarket in the fashion stakes. He won't believe that either.