Ransomware Attack Affects Parts & Service Network for 15,000 Auto Dealerships

[D60]

^^^^
Well said and how I think.

Let’s say I need new brakes and rotors on my 2018 Mazda CX5.
Mazda parts dept is down and don’t have the parts on hand but they can open a Commercial account with AutoZone, NAPA, AAP, etc…

They buy all the parts and simply mark them up ____% plus the necessary labor, shop fee, tax etc….

As a consumer I have no idea what they did…. I just want my brakes done.

If Mazda dealer needs an air filter, NGK plugs, cabin filter, etc… they can still do tune ups even if they don’t have the necessary parts on hand.

Not only that but brakes are a staple of auto service and I'd argue the top-end stuff from O'Reilly or AutoZone (ie BrakeBest Pro/Import Direct or Duralast Gold) is basically just as good as OEM. Certainly good enough that no dealer should be ashamed of using it.

As you said, most consumers don't care. If they're quiet and have decent pedal feel, done. edit: and that customer returns NEXT TIME for more service

 

[JeffKeryk]

Dealerships across the USA are filing lawsuits against CDK.

With big software licenses, there is generally an up-time threshold agreement. The world is so dependent on these systems, when they falter companies are crippled.

For a world class cloud based platform, the weak link becomes your network. Of course the network is necessary regardless of cloud, on prem or hybrid.

CDK got caught with their pants down. Security is 1st, 2nd, 3rd importance; a critical component.

 

[bdcardinal]

^^^^
Well said and how I think.

Let’s say I need new brakes and rotors on my 2018 Mazda CX5.
Mazda parts dept is down and don’t have the parts on hand but they can open a Commercial account with AutoZone, NAPA, AAP, etc…

They buy all the parts and simply mark them up ____% plus the necessary labor, shop fee, tax etc….

As a consumer I have no idea what they did…. I just want my brakes done.

If Mazda dealer needs an air filter, NGK plugs, cabin filter, etc… they can still do tune ups even if they don’t have the necessary parts on hand.

We don't need a functional DMS to place orders with Ford or Mazda. They have their own websites where we go in to place the actual orders. It just makes it a lot easier to create an order in DealerTrack/CDK, save it as a CSV/TXT and upload it to that site. All the part numbers, quantities, and remarks transfer over automatically. It can all be done by just typing, just takes a lot longer. Ford even has 2 ways, the newer DOW or the older DOES II that looks like WarGames.

 

[bdcardinal]

Also to expand, having a functional DMS makes stock orders a lot easier. I just go in to create a daily stock order after doing special orders and it pulls up everything not at the stocking level we want based on the sales parameters we have set. I also run a "Daily out of stock" report that shows everything that was sold out with an empty spot on the shelf and edit the daily stock order accordingly. Sometimes the system won't catch parts that have phased in, like something for a new recall, so you manually increase the numbers until the system catches up.

 

[JeffKeryk]

Also to expand, having a functional DMS makes stock orders a lot easier. I just go in to create a daily stock order after doing special orders and it pulls up everything not at the stocking level we want based on the sales parameters we have set. I also run a "Daily out of stock" report that shows everything that was sold out with an empty spot on the shelf and edit the daily stock order accordingly. Sometimes the system won't catch parts that have phased in, like something for a new recall, so you manually increase the numbers until the system catches up.

In manufacturing that is akin to "MRP" which is Materials Requirement Planning. You probably do MRP using a Min-Max inventory bin level calculation. There is also blowing out BOMs to plan the shop, and MRP2 which adds in the labor requirement. Welcome to my world. Procurement is critical.

 

[Dave Hess]

We don't need a functional DMS to place orders with Ford or Mazda. They have their own websites where we go in to place the actual orders. It just makes it a lot easier to create an order in DealerTrack/CDK, save it as a CSV/TXT and upload it to that site. All the part numbers, quantities, and remarks transfer over automatically. It can all be done by just typing, just takes a lot longer. Ford even has 2 ways, the newer DOW or the older DOES II that looks like WarGames.

I used Mazda as an example cause I drive one, but it could be any manufacturer…

Good to hear Ford and Mazda can still order parts.

 

[D60]

Also to expand, having a functional DMS makes stock orders a lot easier. I just go in to create a daily stock order after doing special orders and it pulls up everything not at the stocking level we want based on the sales parameters we have set. I also run a "Daily out of stock" report that shows everything that was sold out with an empty spot on the shelf and edit the daily stock order accordingly. Sometimes the system won't catch parts that have phased in, like something for a new recall, so you manually increase the numbers until the system catches up.

I think (suspect) you're far more competent than the average employee, whether it's parts or ANYTHING else. It would be nice if I were wrong and everyone cared as much about their profession as you do, but I think we all know most people are just a warm body to collect a paycheck.

I guess my point ‐‐ if I had one -- is to remember others don't view the parts shelves and systems with the same level of competency as you. And you do it all while getting constant push notifications on the Kardashians (KIDDING!!!!)

 

[RazorsEdge]

Honestly from what I've seen, many big companies don't even want to spend money on updating their own computers,let alone invest in security measures. If you don't protect,then your vulnerable. Pay now or later, I guess we know what they chose in this matter.

 

[AZjeff]

As of this morning the local GMC dealer has some CDK functions back but can't print anything. Better than no function but frustrating.

 

[bdcardinal]

I think (suspect) you're far more competent than the average employee, whether it's parts or ANYTHING else. It would be nice if I were wrong and everyone cared as much about their profession as you do, but I think we all know most people are just a warm body to collect a paycheck.

I guess my point ‐‐ if I had one -- is to remember others don't view the parts shelves and systems with the same level of competency as you. And you do it all while getting constant push notifications on the Kardashians (KIDDING!!!!)

Technically it isn't parts on shelves. It is little stacks of the dealer owner's money slowly shrinking over time and then growing with each sale.

Fortunately my family has a strong work ethic that I got to see growing up with my grandpa working at Aero Spacelines then Tracor (Boeing before me) and my mom working at Raytheon for 44 years. Before doing parts I worked in the shop here and at Sears Auto and before that did the auto program at the local city college. When I started here I wanted to be like the old guys and have all this knowledge on the tip of my tongue and be the parts person that the cranky old guys asked for.

 

[bdcardinal]

Back to the memes (not mine)

 

[Dave Hess]

What does your boss say about the situation ?

Are coworkers getting antsy with the hit to paycheck ?

 

[Thermo1223]

Honestly from what I've seen, many big companies don't even want to spend money on updating their own computers,let alone invest in security measures. If you don't protect,then your vulnerable. Pay now or later, I guess we know what they chose in this matter.

This^^^

My job went through their own ransomware attacks about 5 or so years back. It was definitely pre-cvid.

All domain, and proprietary software functions was locked out. I even sent E-mail warnings about strange network activity to crickets. They also had the financial info so they knew asking for $250,000 was nothing but obviously it was the principal.

We had no off-site backups and the IT guy is scared of any type of Cloud Infrastructure.

Live and learn, it was a fun 2 weeks getting paid to do nothing.

 

[Pew]

Honestly from what I've seen, many big companies don't even want to spend money on updating their own computers,let alone invest in security measures. If you don't protect,then your vulnerable. Pay now or later, I guess we know what they chose in this matter.

I am lucky that my company takes it seriously. I have nearly a blank check for it, within reason of course. Within a day I got the green light for new sonicwalls. On my consulting side gig, the only firms that care are the ones that had to go through it before.

 

[SC Maintenance]

Security, or IT in general is a "cost". So its done away with.

When I got out of college in the later 90's, Outsourcing was really, really getting rocking, as was globalism, as was the internet. So we had lots of projects to look at what was a "core" function - ie one we couldn't possibly live without, and those which weren't - so we could outsource them.

A retailer would consider being able to transact business with their customers as being a core function. Not saying they could not use CDK, but the fact some had no backup - is really bad management. You put your ability to operate 100% In the hands of a third party.

 

[bdcardinal]

What does your boss say about the situation ?

Are coworkers getting antsy with the hit to paycheck ?

Thankfully we are on DealerTrack and only sales is getting impacted since CDK runs Ford's eLeads thing for leads on sales. My manager was on vacation all week and the only time we texted was when I told him I wouldn't let a shop that habitually bounces checks float a $6K part even after they said I had a bad attitude.

 

[SC Maintenance]

Decent article, no paywall, covers a little about some of the average people being affected like parts delivery drivers and sales.

Also some humor. It took a week to get a buyer a bill of sale because they had to (gasp) write it by hand.

https://abc7.com/post/cyberattack-affecting-car-dealerships-brings-chaos-sellers-buyers/15014304/

 

[bdcardinal]

I've heard a lot of dealers are slowly coming back online. Now comes the fun part of keypunching all of the hand written ROs and parts invoices as well as trying to post parts invoices and shippers and verifying on hand inventory.

 

[Owen Lucas]

I am lucky that my company takes it seriously. I have nearly a blank check for it, within reason of course. Within a day I got the green light for new sonicwalls. On my consulting side gig, the only firms that care are the ones that had to go through it before.

In the worst case scenario, despite all the top notch planning and equipment, if some ransomware makes it onto the system, what is the final resort?

I presume backups, having weeks or months of backups? Is it that easy? I've heard of scenarios where the backups have been corrupted because malware was in the system before it encrypted everything, but you can at least restore to a time before the infection.

I guess you lose some data between the safe backup and malware infection date, but that's not as bad as going completely down.

So what's that take, a few LTO-9 drives, maybe a library, some software, and 365 tapes? Store them offsite, tapes are obviously air gapped.

LTO-9 drive $10k and $32k for 18TB

18TB $42k
36TB $84k
54TB $129k
72TB $171k

 

[Pew]

In the worst case scenario, despite all the top notch planning and equipment, if some ransomware makes it onto the system, what is the final resort?

I presume backups, having weeks or months of backups? Is it that easy? I've heard of scenarios where the backups have been corrupted because malware was in the system before it encrypted everything, but you can at least restore to a time before the infection.

I guess you lose some data between the safe backup and malware infection date, but that's not as bad as going completely down.

So what's that take, a few LTO-9 drives, maybe a library, some software, and 365 tapes? Store them offsite, tapes are obviously air gapped.

LTO-9 drive $10k and $32k for 18TB

18TB $42k
36TB $84k
54TB $129k
72TB $171k

Complete shutdown of the system and restore from the backups would be the best bet but IT would have to figure out how far back they need to look in the backups to restore from. In my particular case, the ransomware was a zero-day and 'only' got 5 desktop computers but it also partially got into the backups. The desktops were easy to deal with - they just needed a fresh reimage. The backup drives had to be shipped to a data recovery company and was ~$30k+ in 2019 and they still couldn't get the data back, if I recall correctly. It took 11 hours myself to reimage the computers, trace it back to patient zero, and find out what was infected. The office was on the smaller side though, less than 25 employees and typically smaller offices like this can't or won't pay for services like off-site storage/backups and cannot afford or justify their own IT person so pretty much every employee had admin access to their computer because they didn't want to pay every time they needed to do anything that required admin access.