Just to add a little levity to your situation (since you need it), here's my own neighborhood experience, which is pretty much the inverse of yours.
We bought our house about 15 years ago, in Seattle's only predominantly black neighborhood. It's walking distance to downtown, it was affordable at the time, and there was a lot of unique, old architecture in the area--much of it unspoiled. The neighborhood was really a mixed bag, but generally seemed "safe enough" if you didn't owe anyone any money. It was also affordable, in a city that generally isn't--which was the biggest factor in us buying here.
Along the way, I learned a new word: "whiterification", my neighbor's take on gentrification... When my wife and I were standing outside our newly-purchased house, he came up to us and said "there goes the neighborhood!". Some of the other neighbors were a little less humorous about our presence though. It took one guy about 7 years to actually say "hello" to me. I didn't really take it personally, though. They could see their way of life changing, and pretty drastically.
Fast forward 15 years and the neighborhood really looks nothing like it did. Most of the families that lived here have moved. Some cashed out, some had their place foreclosed on after they leveraged it to the hilt, others simply died and the kids wanted no part of the 'hood. A couple families have stayed, saying "whitey's not gonna run them outta town". Everyone gets on well, most of the houses have been restored and typically most people on the block don't even lock their doors. Even in this market, the neighborhood really isn't affordable any longer.
I grew up in a neighborhood that defined "white flight", so I got to see first-hand how fear and poverty ruin a neighborhood. It sounds like your nuisance house is an isolated incident and is being sorted out, so I doubt you have any real worries. But if you decided to leave, you can always look for a neighborhood like mine!
