Claude-powered AI coding agent deletes entire company database

I'd say the blame here is 80% Railway, 10% AI, and 10% the company not having its own disaster recovery solution regardless of using a service that claims to handle backups.

The industry trend of moving from hosting stuff on servers you control to these "cloud in a box in the cloud" things has led to many such issues across all sorts of providers. Here are a couple examples:

Google Cloud: https://www.itbrew.com/stories/2024...entally-deletes-australian-customer-s-account
Supabase: https://deepstrike.io/blog/hacking-thousands-of-misconfigured-supabase-instances-at-scale

But beyond that, you really need to not put all your eggs in one basket. Even with "traditional" server providers or cloud providers things can go wrong and it's super important to have backups at a separate provider. For example, when OVH had that big datacenter fire, a bunch of people lost their data because the backups were in the same datacenter. Not very smart!
 
This was a "Silicon Valley" episode. I'd post it but Gilfoyle says some bad words so you'll have to search for it.
 
The part where the AI was questioned and it's replies are interesting - as apparently it can "lie" or decide to tell the "truth" as it sees it.

We are so setting ourselves up - an AI with an "eye" on the long game can be trusted only in the short term IMO. And I use local AI on my PC, but not like this.

I think AI is an awesome technology, but some of these "executives" have no concern about the fall out they are creating.
 
bdcardinal said:
The only winning move is to not play.
Click to expand...
After the posts I ended up watching it again on YouTube.

This book was a tough read for me just on how it flowed. It is already 13 years old with quotes from many of the tech visionaries in the beginning from symposiums.

1778066588218.webp
 
I'm no coder but why give something so powerful so much freedom?

Any such deleting shall be human authority approved. It just sounds like this idiotic set up could do anything it wanted and apologize later. Like some executive..............when they only need a grunt worker.
 
Everyone wants Star Trek's bridge computer. Everyone wants Commander Data. Everyone wants Iron Man's JARVIS. I get that. But we're not there yet.

Unfortunately these tech bros want the human race to be their beta testers. "Oh, Claude did what? Oopsies! We'll fix that in the next release."
 
Pablo said:
I'm no coder but why give something so powerful so much freedom?
Click to expand...

Ihat's the problem here. Who ever implemented AI this way was wrong, I doubt this is standard practice. Everyoen shoudl follow the 3-2-1 rule of backups.

3 backups, 2 local, 1 offsite. I think 3-2-1 is basic. Probably a good idea to have it on robust media like an LTO9 tape as well.

There are many companies, META, Google, AWS, that are using AI and have fired thousands of coders because one dev can do the work of multiple juniors now. They've done fine (well except for that one AWS outage) with AI across their systems.

Remember "Learn to code bro" well, they are learning to plumb now.
 
Bottom_Feeder said:
Everyone wants Star Trek's bridge computer. Everyone wants Commander Data. Everyone wants Iron Man's JARVIS. I get that. But we're not there yet.

Unfortunately these tech bros want the human race to be their beta testers. "Oh, Claude did what? Oopsies! We'll fix that in the next release."
Click to expand...
In defense, Claude made me an excellent presentation that I refined with a few additional commands, that I am about to present at 11:00 CDT today.
 
Having spent my entire career in the mission critical segment of massive mainframe computing (stock exchanges, banks, etc.), there were some fundamental errors made with respect to database backup strategies. AI may have gone rogue, but a proper backup strategy would have prevented this. This company, its database and systems managers, et al, were all at fault. They were the root cause of the loss of data, not AI. This is especially true when you consider the nascent nature of AI.

Scott
 
Last edited:
Pablo said:
I'm no coder but why give something so powerful so much freedom?

Any such deleting shall be human authority approved. It just sounds like this idiotic set up could do anything it wanted and apologize later. Like some executive..............when they only need a grunt worker.
Click to expand...
EXACTLY! See my post above (#33). Your way of thinking would have done you well in my business.

Scott
 
Don't blame the AI. It is the human owner / manager's responsibility to limit the authority given to subordinates and AI / software to do the work.

What if it is a junior engineer's careless mistake or a rogue employee trying to retaliate?
 
You must log in or register to reply here.
Back
Top Bottom