I lived in a third story flat (apartment building) a long time ago and had a real problem with one of the neighbours. I ended up having to move as it was just not worth losing my sanity over something which, looking back, was so trivial. I could have fought it out, but I am truly glad I moved.
Can I ask you a question though......Why are you allowing them to invade your privacy? I'm guessing from what you've written they are not friends/family!? So what gives? It sounds like you are allowing them to walk all over you!