The great China Hack - Epic infiltration

[OVERKILL]

Originally Posted by PandaBear

Personal tie? No, but have done enough contract manufacturing related work that I know changes like this should not have been "caught off guard". PCB changes are usually big enough that you will not be able to fool people, because you have to change the test and the result interpretation, and the yield, and all sorts of other stuff. The design already has the pads in place to insert these chips, that's for sure, otherwise you can't fool people and change their designs. The customers probably asked them to not populate a few chips they only use for design / debugging in mass production, and then the manufacturers have some left and they just put them in anyways (to reduce variations or make manufacturing easier, i.e. more test to improve yield) at no charge. You see that a lot if you get penalized for yield loss (i.e. typically 98.5% when mature, 90% when launch). Throwing in a free chip is cheaper than a lost.

That makes sense. So the solder points are already on the board since these are generally cookie cutter designs picked from their extensive catalog and not all designs sharing the same board share the same component layout. Ergo, one could "include" this extra chip that wasn't in the design without drawing significant suspicion because a component in that location was spec'd on other variations of that design. Of course the other possibility was that somebody (or somebodies) at SuperMicro was/were aware of what was going on and these boards were modified to include a location for the chips, but I think your idea is more probable.

Originally Posted by PandaBear

I've also done security audit and know that you will ALWAYS find something, big or small, and you'll never catch everything.

For sure, but in this case it was ultimately the customer that had to find it, as SuperMicro was, apparently, completely unaware

Originally Posted by PandaBear

I'm more interested in why it isn't mentioned which chip they are talking about. My bet is a debug access chip on SMBus or UART, something useful for debugging, maybe even just the phy to the chip or a resistor to enable / disable the existing debug features. I have seen contract manufacturers throw in a few popcorns for free so they don't have 2^6 kind of configurations they have to keep inventories on, and get sloppy and didn't document these stuff.

From the article:

Originally Posted by Bloomberg

A Chinese military unit designed and manufactured microchips as small as a sharpened pencil tip. Some of the chips were built to look like signal conditioning couplers, and they incorporated memory, networking capability, and sufficient processing power for an attack.

Originally Posted by Bloomberg

Apple made its discovery of suspicious chips inside Supermicro servers around May 2015, after detecting odd network activity and firmware problems, according to a person familiar with the timeline. Two of the senior Apple insiders say the company reported the incident to the FBI but kept details about what it had detected tightly held, even internally.

And:

Originally Posted by Bloomberg

two people familiar with the chips' operation say. This happened at a crucial moment, as small bits of the operating system were being stored in the board's temporary memory en route to the server's central processor, the CPU. The implant was placed on the board in a way that allowed it to effectively edit this information queue, injecting its own code or altering the order of the instructions the CPU was meant to follow. Deviously small changes could create disastrous effects. Since the implants were small, the amount of code they contained was small as well. But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device's operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.

Now of course the article is dumbed down a bit, but it sounds like the chip placement is relative to the BBMC and exploits it and its functionality.

Quote

https://www.kgw.com/article/money/a...l-technologies-server-hack/283-600850935 Is it really a hack? Or just propaganda? My money is betting that Dell and HP are trying to smear Super Micro so they don't lose more customer to the cheaper sources. Let's be serious here. How are they going to access the server when it is in a data center, guarded by all sorts of security check on the network? Even if you open it all up you won't be able to access it without being seen.

The quote from Apple seems to indicate that it reached out to a computer on the internet in an inconspicuous manner, perhaps masquerading as other traffic, which is perhaps why that activity wasn't noticed immediately.

 

[Vikas]

Originally Posted by PandaBear

https://www.kgw.com/article/money/a...l-technologies-server-hack/283-600850935 Is it really a hack? Or just propaganda? My money is betting that Dell and HP are trying to smear Super Micro so they don't lose more customer to the cheaper sources. Let's be serious here. How are they going to access the server when it is in a data center, guarded by all sorts of security check on the network? Even if you open it all up you won't be able to access it without being seen.

Quote

An Amazon spokesperson countered the report: "As we shared with Bloomberg Businessweek multiple times over the last couple months, at no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in Super Micro motherboards in any Elemental or Amazon systems.‎ Additionally, we have not engaged in an investigation with the government."

Who do you believe? Bloomberg or Amazon? I would have expected Amazon to make "No comment" reply rather than specifically stating that there was no modified hardware was found. The denial is extremely strong. Actually, I would have expected "Amazon could not be reached" rather than getting the blanket denial. I am thinking somebody is making a mountain out of an ant :-)

 

[OVERKILL]

Originally Posted by Vikas

Originally Posted by PandaBear

https://www.kgw.com/article/money/a...l-technologies-server-hack/283-600850935 Is it really a hack? Or just propaganda? My money is betting that Dell and HP are trying to smear Super Micro so they don't lose more customer to the cheaper sources. Let's be serious here. How are they going to access the server when it is in a data center, guarded by all sorts of security check on the network? Even if you open it all up you won't be able to access it without being seen.

Quote

An Amazon spokesperson countered the report: "As we shared with Bloomberg Businessweek multiple times over the last couple months, at no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in Super Micro motherboards in any Elemental or Amazon systems.‎ Additionally, we have not engaged in an investigation with the government."

Who do you believe? Bloomberg or Amazon? I would have expected Amazon to make "No comment" reply rather than specifically stating that there was no modified hardware was found. The denial is extremely strong.

Also from the article:

Quote

One of those officials and two people inside AWS provided extensive information on how the attack played out at Elemental and Amazon; the official and one of the insiders also described Amazon's cooperation with the government investigation. In addition to the three Apple insiders, four of the six U.S. officials confirmed that Apple was a victim. In all, 17 people confirmed the manipulation of Supermicro's hardware and other elements of the attacks.

So employees of Amazon and Apple both confirmed it

 

[SonofJoe]

Of course we would NEVER do such a thing to the Chinese would we? It just wouldn't be cricket would it? LOL!

 

[Vikas]

No; you are comparing official vs "unnamed sources" response. No way official response would have been this strong if there was underlying case to be made. You are also trying to intentionally misled us by putting the "same article" in your comment which is rather a cheap trick.

 

[OVERKILL]

Originally Posted by Vikas

No; you are comparing official vs "unnamed sources" response. No way official response would have been this strong if there was underlying case to be made. You are also trying to intentionally misled us by putting the "same article" in your comment which is rather a cheap trick.

So your position here is that Amazon's media-facing department is more reputable than internal employees who have chosen to remain anonymous due to the sensitive nature of the content? I'm not intentionally trying to mislead anybody and I'll kindly ask that you retract that assertion. "Also from the article" meant the linked article, in the OP, which is what this thread is discussing. Anybody who has read the OP, yourself included, would readily identify that content from that source. Calling it a "cheap trick" is doubly insulting as you are implying that I have an agenda here, which is a ridiculous claim, given my lack of ties to any of the involved parties.

 

[Vikas]

OK; the original article's claims are all out there. Obviously, I was talking about the counter article and and put the link there. Your comment implied that the counter article had the assertion that you made. Since I misinterpreted your intention, I will retract that assertion. Getting back, I see absolutely no reason for the companies involved to make any comment. The correct way to handle would be to make no comment unless the entire thing has been blown way out of proportion. I am more inclined to believe the innocent explanation given by Panda aka BMC debugging chip.

 

[oil_film_movies]

From the bloomberg article cited at the beginning: "Two of Elemental's biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not." --- That there is funnee, I don't care who you are.....

 

[OVERKILL]

Originally Posted by SonofJoe

Of course we would NEVER do such a thing to the Chinese would we? It just wouldn't be cricket would it? LOL!

I don't think anybody could claim with a straight face that we aren't trying to gain, likely successfully, information of a sensitive nature from and about the Chinese through underhanded tactics. Where things differ in this case is that we aren't doing equipment manufacturing for the Chinese, whilst they are in a position to exploit that role with us. As PandaBear noted, most of the design work on a lot of this stuff is American, like Intel's CPU's for example, or AMD's. They are usually manufactured somewhere else, but these specifically, are complex components that would be more difficult to design-in a trojan horse that has the ability to capture and transmit sensitive information (which is the claimed functionality here), particularly if its a design deviation targeted specifically at a foreign market. The odds of it being found out, like with the recent Intel bug, is also more likely. On the other hand, a rather inconspicuous component placed on a motherboard that ties into something like the management controller can be installed at the production facility and could be easily customized based on the target market or customer for a bulk order. Fabrication and assembly, which is the case in the linked article, provides a unique opportunity for something of this nature. And of course given the depths of proliferation with the respect to the hardware in question, there is perhaps a level of depth attainable that would be much harder, if not impossible, for more traditional techniques.

 

[oil_film_movies]

Originally Posted by SonofJoe

Of course we would NEVER do such a thing to the Chinese would we? It just wouldn't be cricket would it? LOL!

Yes we would, and do. Yet, from my perspective, I'm more worried when autocratic and/or dictatorial governments around the world get this capability. ... Not to mention the infamous Chinese IP theft problem allowing them to get info we develop.

 

[OVERKILL]

Originally Posted by Vikas

OK; the original article's claims are all out there. Obviously, I was talking about the counter article and and put the link there. Your comment implied that the counter article had the assertion that you made. Since I misinterpreted your intention, I will retract that assertion.

Thank you.

Originally Posted by Vikas

Getting back, I see absolutely no reason for the companies involved to make any comment. The correct way to handle would be to make no comment unless the entire thing has been blown way out of proportion. I am more inclined to believe the innocent explanation given by Panda aka BMC debugging chip.

Sure, that's possible. It could also be that the solder points on the board for a debugging chip were utilized for the placement of this PLC component to behave as noted. There are a number of options here and unfortunately we don't have the statements from the people who provided the data on which the article is based to perhaps get more technical information, which would be extremely valuable as this discussion heads in that direction. Perhaps these companies have been instructed by the government to deny? There's that possibility as well. I agree with your premise, that "no comment" would be more appropriate than flatly denying it, but perhaps there's been some grooming to these public responses

 

[OVERKILL]

Originally Posted by oil_film_movies

From the bloomberg article cited at the beginning: "Two of Elemental's biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not." --- That there is funnee, I don't care who you are.....

Yeah, that was amusing.

 

[PandaBear]

Originally Posted by OVERKILL

The quote from Apple seems to indicate that it reached out to a computer on the internet in an inconspicuous manner, perhaps masquerading as other traffic, which is perhaps why that activity wasn't noticed immediately.

Yes, this is likely how they find it suspicious to begin with. We usually have enough security flaw in the debug / trace / testing mechanism that hacking only requires enabling them instead of designing something new to inject.

 

[OVERKILL]

Originally Posted by PandaBear

Originally Posted by OVERKILL

The quote from Apple seems to indicate that it reached out to a computer on the internet in an inconspicuous manner, perhaps masquerading as other traffic, which is perhaps why that activity wasn't noticed immediately.

Yes, this is likely how they find it suspicious to begin with. We usually have enough security flaw in the debug / trace / testing mechanism that hacking only requires enabling them instead of designing something new to inject.

Would be great to have some more details, preferably of the technical nature, that we could discuss where it is more obvious what is taking place. Unfortunately the Bloomberg article is designed to be real-world readable and thus that seems to be entirely omitted

 

[PandaBear]

Big companies like Amazon and Apple will always have suspicious people believing in something, no matter how small the probability is, that's just normal. Having a couple anonymous employees making statements and then get blown out of water in the media (i.e. this one) is the reason why they usually have a "no talking to media, PR will handle this" or you are fired policy. It is very normal. Usually you have enough bugs in the OS that you do not need a hardware approach to hack into a system, that's a lot easier and hard to find than an extra chip on the PCB.

 

[OVERKILL]

Originally Posted by PandaBear

Big companies like Amazon and Apple will always have suspicious people believing in something, no matter how small the probability is, that's just normal. Having a couple anonymous employees making statements and then get blown out of water in the media (i.e. this one) is the reason why they usually have a "no talking to media, PR will handle this" or you are fired policy. It is very normal.

For sure, but there appears to be some degree of confirmation from the government that these claims are legitimate and that it resulted in Apple removing all of the SuperMicro servers they purchased, which does support the idea that something was amiss. Also, it seems that the actual operation of the device was laid out, but we are not being given the specifics on what that was due to the dumbing down of the article content. I'd really like to see the quotes from the 17 people, as that would really allow us to wade through the speculation and know whether the headline and conclusion are in-line with the merit of the claims or whether a lot is being made about something rather benign in actuality.

 

[PandaBear]

Originally Posted by OVERKILL

Would be great to have some more details, preferably of the technical nature, that we could discuss where it is more obvious what is taking place. Unfortunately the Bloomberg article is designed to be real-world readable and thus that seems to be entirely omitted

When you design a chip you also need to provide way to test it. Chip making is not perfect and will always have yield issues. You need to make sure any logic you design has mechanism for testing. Usually, a chip will have a JTAG port that access locations between logics, and during testing you insert certain inputs and then extract the output, check it against a table for pre-determined results and see if they match. If your design is simple this will be it. For security concern this port may be turned off before releasing to customers. Sometimes not everything is turned off, because they ship products that may not be 100% ready, so they leave the access on and debug at the customer sites for the last 0.01% of the problems (and there's always something missing).

 

[OVERKILL]

Originally Posted by PandaBear

Usually you have enough bugs in the OS that you do not need a hardware approach to hack into a system, that's a lot easier and hard to find than an extra chip on the PCB.

This is true. Perhaps the idea was to gain access through systems that lack traditional connectivity? I mean, since we are in the realm of wild posit due to the lack of significant detail in the article and various government agencies are mentioned, what if the compromised server hardware was used as part of the operations systems on a sub? It would technically be able to gain outside access through maybe sat-link or something similar (I have zero knowledge as to how a sub control and its ancillary systems operate here, I'm just spitballing, so keep that in mind) or when the unit is docked While NPP's are islanded, there is also the possibility of DOE infiltration which could allow access to other assets. Further spitballing here of course.

 

[SonofJoe]

Originally Posted by oil_film_movies

Originally Posted by SonofJoe

Of course we would NEVER do such a thing to the Chinese would we? It just wouldn't be cricket would it? LOL!

Yes we would, and do. Yet, from my perspective, I'm more worried when autocratic and/or dictatorial governments around the world get this capability. ... Not to mention the infamous Chinese IP theft problem allowing them to get info we develop.

I love your tone of moral certitude! To you it's a simple case of the good guys in white hats versus the bad guys in black hats. The trouble is, to a lot of eyes, the hats that used to be so white are now so grubby, it's hard to tell what colour they are.

 

[OVERKILL]

Originally Posted by PandaBear

Originally Posted by OVERKILL

Would be great to have some more details, preferably of the technical nature, that we could discuss where it is more obvious what is taking place. Unfortunately the Bloomberg article is designed to be real-world readable and thus that seems to be entirely omitted

When you design a chip you also need to provide way to test it. Chip making is not perfect and will always have yield issues. You need to make sure any logic you design has mechanism for testing. Usually, a chip will have a JTAG port that access locations between logics, and during testing you insert certain inputs and then extract the output, check it against a table for pre-determined results and see if they match. If your design is simple this will be it. For security concern this port may be turned off before releasing to customers. Sometimes not everything is turned off, because they ship products that may not be 100% ready, so they leave the access on and debug at the customer sites for the last 0.01% of the problems (and there's always something missing).

Yeah, I've used JTAG ports to revive routers that have had their firmware corrupted for example, so I know what you are talking about. What do you think is the likelihood that a customized SOC could be developed that would utilize the JTAG interface on say the BBMC to leverage its functionality in the way the article implies?