I think that I would tell them that you sometimes are not dressed for company, so if they knock, it might be a few minutes before you answer. A solution might be if they were to call ahead. Just don't answer the door if you are not dressed to meet anyone that is there.
