To be fair to the OP, it has only been in the last couple of years that I have reached a comfort level that doesn't require "fire alarm" type nervous reactions when the doorbell rings or if my wife were to come home unexpectedly. Prior to that, all through my youth, teens, and adult life I would have jeans or sweatpants, oversized shirts and the like in "strategic" locations so I could get something quickly over whatever it was I was wearing if need be.
Perhaps the real game changer was when my wife deeply and honestly (with no strings attached) said she does not care at all what any of the neighbors think, so I don't bother to change for the door anymore. I do have kids, but we are working together on letting them slowly know that this is part of me, so for now I would pull the fire alarm if need be, just for the wish not to be shocking to them. My wife also prefers not to see me dressed (at least she never has asked or indicated any particular curiousity) so I would likely try to change for her, although I won't hurt myself or clothing over it.