A charging station has no cash or booze, odds are the people using a charging station have no cash, and odds are criminals aren't driving EV's either, so you have less chance of getting caught in a random gun fight... Also you sit in your locked car, so its some protection against random crime. Someone starts harassing you, and you feel in danger, drive away...
As a practical matter, it's not possible to drive away when connected. No modern EV will allow it, and connectors are typically locked - at least with Tesla. So it would require getting out of the vehicle to remove the connector.
However, I've had limited experience with EV charging since June and I can't think of any time when there was anything that looked out of place. Most people kept to themselves, although once at a Supercharger we did talk to a Tesla employee who was up from the Bay Area to offer a test drive to someone in Mendocino County. But for the most part I don't recall too many people chatting it up.
The oddest thing I saw was someone in a Ford F-150 Lightning who parked in a Tesla charging spot (a back-in spot but he went head-in). These spots were specifically reserved for vehicle charging; some spaces have signs saying that vehicle charging is preferred but not required. Not sure if he was engaging in some sort of protest against EVs, which I've heard of. Then I saw someone else get out of the pickup and get into a Tesla, so he was just dropping her off.
That may be something I worry about - the oddball, anti-EV people who think that they're making a statement by blocking charging stations and/or harassing EV drivers.