I've driven dressed a few times. It's an enjoyable way of getting out into the world, as an observer at least, without being read.
The cars are parked on the back patio, so there's little chance of being observed getting in and out; neighbors are a long way away anyway.
Heels, I think, might be awkward for me; enfemme I choose to drive in low heels or flats. I had a 5-liter Mustang some years ago whose clutch was somewhat heavier than most, so I appreciated Exris's remarks about heels. However, what I think is important is for the shoe to fit firmly to the foot and not slip. In drab I don't usually wear shoes around the house and put on flipflops if I need to go outside. Sometimes I'll drive somewhere without putting on any "regular" shoes. However, I don't like to drive in flipflops because they can slip. I just kick them off and drive in bare feet, which I find far safer.
I recall hearing that the famous English rally driver of years gone by, Pat Moss, used to drive in bare feet, so obviously she found no problem with it. While I was trying to check that fact, Google came with an article from the British Daily Mail about driving in flipflops and other footwear, which readers here might find interesting.