My immediate reaction is if all the neighbors know then you have nothing left to lose? You can come and go as you please.
I have a parallel story.
My GG friend Michelle needed help moving. She organized some friends and I (guy me) went and helped.
I introduced myself to one of the others and said we previously met at a local bar. A bar I've only been to as Dee. Ok, own it.
Five days later, back to bar, this time as Dee. Same guy was there (and his girlfriend, who was also at move). Not an issue.
I'm finding people care less and less as time moves forward.
Or I care less.
Or both.