It is amazing how we don't know how to handle male advances when we're enfemme. I know, I've been there and it scared the hell out of me.
First, YES he is hitting on you.
Just because you dress doesn't mean you are gay or interested. If you're interested fine, but if not politely tell him that. If he is as professional as you say he should accept your comment and continue in a professional relationship.
I understand your comments about improving your physique but you're right, it doesn't help your feminine image at all!
Good Luck!