I'd be telling the neighbor that HIS problems are HIS problems, and that they're NOT your problems.
I had one of these types across the street from me. He was a retired policeman, which only made his personality even worse.
I was building a custom 8x12 yard barn in my backyard (which he couldn't even SEE from his house), and he stormed into MY backyard, ranting and raving about it.
I had all of the necessary permits that I needed. I used the same materials/colors to build the yard barn that were used to build my house, and I even landscaped around the barn like I had around the house.
I stopped working and freely let him vent for 4 to 5 minutes. Then I picked up my cell phone and told him that he had 30 seconds to wrap it up, and then I would be calling the sheriff's department so they could send a deputy to help him find his way back to his house. Of course, with him being a retired cop, that had the desired effect, and he immediately found his way back home... and I never heard from him again. (*YAY!*)
Of course, less than two hours later, I received a friendly visit by the local municipal building inspector, who checked my permits and found everything in order.