General Annoyance, work has no backup plans for IT

[Thermo1223]

This is just the continuation of the same lackadaisical attitude toward disaster planning with them. Another warehouse close by had its cable modem & SonicWall destroyed again by lightning.

No hot spare so we'll have to order one....

Like why it only serves less than 10 devices, it's barely a SOHO. Keep a hot spare. Yes the non-IT people(ME) have to go troubleshoot it for them in NY because it might just be a "bad power brick".

/rant

 

[Kira]

I suppose with details; this very interesting post would be much longer.
The warehouse is in New York State?
Would a lightening arrestor have sufficed?

And a "bad power brick" is just a bad transformer, yes?

 

[edyvw]

This is just the continuation of the same lackadaisical attitude toward disaster planning with them. Another warehouse close by had its cable modem & SonicWall destroyed again by lightning.

No hot spare so we'll have to order one....

Like why it only serves less than 10 devices, it's barely a SOHO. Keep a hot spare. Yes the non-IT people(ME) have to go troubleshoot it for them in NY because it might just be a "bad power brick".

/rant

When hackers breached Target, it was a guy who had a Master's in Public Administration who figured out what they used to enter the system (AC system).

 

[Pew]

I'm assuming your work has no cyber security insurance or they use a MSP for satellite locations?

We're required, by both cybersecurity insurance and by our public works clients to have proper documentation.

IT is almost always seen as a waste of money so many places don't have proper IT controls, procedures, or budgeting to allow for anything more than desktop break-fix dealings. Lots of things wrong from top -> down.

 

[CarbonToast]

What is the ROI for a backup? I can't seem to get any buy-in on contingency investments. If its still working we aren't buying a new one, backup, anything.

Unfortunately problems have to occur to "create the need" then suddenly everyone wants to solve it.

 

[Pew]

What is the ROI for a backup? I can't seem to get any buy-in on contingency investments. If its still working we aren't buying a new one, backup, anything.

Unfortunately problems have to occur to "create the need" then suddenly everyone wants to solve it.

Ugh that policy is the death of many good IT administrators. No room in the budget for IT DRP but then management is surprised when things go down and replacement parts are days or weeks away.

 

[Leo99]

My company is extremely risk averse. We have backups for everything and maintain a two year inventory of product and raw materials should something happen to one of our suppliers.

 

[Thermo1223]

I'm assuming your work has no cyber security insurance or they use a MSP for satellite locations?

We're required, by both cybersecurity insurance and by our public works clients to have proper documentation.

IT is almost always seen as a waste of money so many places don't have proper IT controls, procedures, or budgeting to allow for anything more than desktop break-fix dealings. Lots of things wrong from top -> down.

Nope it is frustrating to say the least. Family run company so it's 5 brothers arguing about things sometimes.

 

[Thermo1223]

What is the ROI for a backup? I can't seem to get any buy-in on contingency investments. If its still working we aren't buying a new one, backup, anything.

Unfortunately problems have to occur to "create the need" then suddenly everyone wants to solve it.

Well considering the warehouse cannot ship *IE make money* I mean I would assume it would pay for itself in an hour.

 

[Thermo1223]

I suppose with details; this very interesting post would be much longer.
The warehouse is in New York State?
Would a lightening arrestor have sufficed?

And a "bad power brick" is just a bad transformer, yes?

I was vague because details wouldn't matter. HQ is in NY state, warehouse is in PA.

The bad transformer idea was a pipe dream as the whole thing was fried. Lightning came through the coax into the modem, killed it and the SonicWall. 2nd time it happened actually.

Considering it's always the coax maybe Astound Broadband should install one.

 

[tburke]

Sonicwalls can operate in a high-availability configuration, so if this facility is as critical as it seems, they should really have an HA pair.

(on that note, why in the world is _anyone_ still using Sonicwalls in 2025?)

I have had good luck with these lightning arrestors: https://www.amazon.com/Proxicast-Lightning-Arrester-Suppressor-Connectors/dp/B0CBW5MM13/. Mount it at the point of entry to the building, bond it to ground, and you should be good to go.

 

[SC Maintenance]

Its not just IT. Industrial is the same. Stock no spares - when the part fails call the manufacturer and scream at them for not keeping one in stock. If your loosing $10K a minute stock your own spare.

Of course I have seen lots of industrial spares sit on the shelf and never gets used. Some accountant comes in, scraps the spares, writes a corrective action to not stock spares that were never used and save lots of money. Accountant gets promoted. Next time machine quits - rinse / repeat.

 

[CarbonToast]

Well considering the warehouse cannot ship *IE make money* I mean I would assume it would pay for itself in an hour.

Yes, but the people in charge dont think this way.

 

[JeffKeryk]

Up time is expensive. Up time is hard. And nowadays there are so many moving parts...

I built a disaster recovery fail over system and plan for my Forecast application that was deemed Mission Critical by the Exec Staff. It had to be tested periodically. Let's just say SAP failed every time; mine was able to be brought up in the DR site in 15 minutes by non-IT personell.

Lotta earthquakes around here, ya know...

 

[rijndael]

I find it's especially hard for mid-sized organizations. You're big enough to think you're a fortune 500 sized player, but are unwilling to spend that level of money.

I had customer that had their own facility/electrical crews, they do a lot of industrial power stuff. But, when it came to their own data center, they used the same crews ... as if keeping the power on to a campus of buildings is the same as doing data center power. But, you're too short sighted to realize it's specialized. Then you keep suffering data center power failures, mostly because this is outside of your expertise, but are unwilling to outsource it (either in crews or colo).

 

[dubber09]

SonicWall? What kinda business uses SonicWall? Bubba & Co?
Do they even care about DLP? Oh, that stands for Data Loss Prevention.
Remember the time Singing River (hospitals) lost not only facilities but a ton of patients' data? For a smaller business that would equal basically end of it.

 

[Thermo1223]

Sonicwalls can operate in a high-availability configuration, so if this facility is as critical as it seems, they should really have an HA pair.

(on that note, why in the world is _anyone_ still using Sonicwalls in 2025?)

I have had good luck with these lightning arrestors: https://www.amazon.com/Proxicast-Lightning-Arrester-Suppressor-Connectors/dp/B0CBW5MM13/. Mount it at the point of entry to the building, bond it to ground, and you should be good to go.

On the SonicWall bit I agree, why they are not sending us a better core router.

 

[Pew]

What's wrong with sonicwalls? We use them but I only have very limited experience with Fortinet.

Works great for an SMB and they have awesome upgrade specials for them. We went from an 2600 -> 2700 and 300 -> 370 for free with the 3-year cgss purchase. They're not *amazing* but they work well for SMBs.

 

[UncleDave]

My company is extremely risk averse. We have backups for everything and maintain a two year inventory of product and raw materials should something happen to one of our suppliers.

What do you build?

 

[Leo99]

What do you build?

We make drugs for rare diseases. Many of our suppliers are single source. Two years would give us time to bring another supplier online.