18 ms·

Dell and HPE switches come with 'American Megatrands' stickers

345 points by geerlingguy 5y ago

Animats 5y ago
Time for someone to dump the code from the "American Megatrans" chip and compare it with one that says "American Megatrends".
Rarok 5y ago
It's just a sticker, but a sticker that has a license. In the world where we live I would expect that something like this has to be aproved by a legal team.
Am_I_Right 5y ago
Neither Dell nor HPE manufactures switches (the former never has, the latter hasn't for the past few years). So, these are all sourced from an OEM like Edgecore. And, someone at that OEM ordered a bunch of misspelled stickers. Easy mistake to make, if the latin alphabet is literally foreign to you. And if you think that sticker is bad? Wait until you see the actual firmware, oh boy... (I had some fun Edgecore LACP bugs take down an pretty sizable network. Things got slightly better once they moved to Linux-based firmware, but never to the point that their kit was, like, entirely reliable...)
- bluedino 5y ago
  Isn't the fear that they are imitation parts?
  - mxxx 5y ago
    Yes, but they’re not. It was just a typo.
  - Am_I_Right 5y ago
    I'm not entirely sure what the fear is. The AST2600 the sticker seems to have been pasted to is a pretty complicated IC with (and this is the important part) user-upgradeable flash to begin with. So, you want to do a supply-chain attack? Simply reflash the genuine modules. No need to spend more. On the other hand: you want to save a few bucks? Possibly do a knock-off chip, but you're definitely not going to bother with the firmware. Too expensive! This is definitely a case of "trying to save a few bucks". Both Dell and HPE are in a race to the bottom, and the sticker being indicative of anything significant beyond that is... unlikely...
  - NAR8789 5y ago
    I think so, but assuming that is the case... what's to stop a shady chipmaker from printing properly-spelled "American Megatrends" stickers? More generally... are there any actual protections offered by genuine stickers? The article makes this out to be a major supply chain security issue, and that only makes sense if branding stickers are actually reliable for validation purposes. But that seems... nonsensical? Wouldn't stickers be very easy to forge? But, I don't work in supply chains. Anyone with better expertise in this area able to chime in? I will admit I skimmed the article, because it is long and overly-detailed for my level of interest, and because it lacks summary sections.
    - jcrawfordor 5y ago
      It's not at all that a properly spelled sticker gives assurance that it's not counterfeit... it's just that a misspelled sticker is such an obvious sign of a potential counterfeit that it's basically the #1 thing that any counterfeit/suspect items program teaches people to look for. Most people working on counterfeits don't speak English so it's very easy for these kinds of mistakes to slip through, and on the other hand they're rarely made by the genuine manufacturer which usually has a process to check for this kind of thing even if the engineering work is done in a non-English speaking country (most of all that the logos usually come from off-the-shelf art files from the marketing department, so no one's even typing the name to make a mistake). Almost any corporate or institutional counterfeit or supply chain security program will explicitly teach you: if anything is misspelled or shows other obvious mistakes, hold the part as a suspected counterfeit. It's a pretty good quality indication. So of course manufacturers do genuinely make spelling mistakes sometimes, but this context makes it a pretty embarrassing and serious thing to do. It's like your bank misspelling their name in an account notification: sure, in some extremely theoretical sense it doesn't mean anything, but in practice they're giving you exactly the signal that everyone tells you to check for to identify phishing, and it raises questions about their processes that they let it slip through.
- ksec 5y ago
  Well, for many stickers, they have to be ordered from original vendor / AMI. I guess this is not the case here. Turns out it is coming from AMI, but AMI Taiwan. >AMI Taiwan needed to get license stickers for the local market. Instead of using the “American Megatrends” MegaRAC PM sticker template, it decided to make its own that had the misspelling.
  - hef19898 5y ago
    So really nothing to see here, other than an AMI subsidiary got a bunch of stickers with a typo and distributed hose to suppliers.
- walrus01 5y ago
  edgecore is just a marketing name, the actual company is accton they're generally a competitor of companies like compal, clevo, quanta. All well known in Taiwan if you're in the business of having 3rd parties manufacture your stuff.
  - alliao 5y ago
    I'm pretty sure accton's compal's communications subsidiary
    - walrus01 5y ago
      that is a good point, not something I'd had reason to think about since 2006 or so. Compal is quite a behemoth. https://en.wikipedia.org/wiki/Compal_Electronics https://en.wikipedia.org/wiki/Compal_Electronics US $26 billion revenues. Most people have never heard of it, only its consumer facing brands like Ignitenet.
  - merb 5y ago
    it's crazy how much tech companies taiwan has. i'm pretty sure that this has conflict potential with china (i.e. china with the eu/us).
    - walrus01 5y ago
      some 15-20 years ago most of the big taiwanese electronics manufacturing companies (top tier x86-64 motherboard makers would be a good example) moved a lot of their factory operations to mainland china, for lower cost labor. it's very interconnected now. there's a fascinating yearly trade show of taiwanese manufacturers: https://www.computextaipei.com.tw/en/index.html https://www.computextaipei.com.tw/en/index.html
      - c_o_n_v_e_x 5y ago
        I frequently work with Taiwanese manufacturers in the industrial PC space. Depending on what the client needs, I can specify whether the equipment is made in China (Suzhou) or Taiwan. Having the equipment made in Taiwan costs at least 10 to 15% more.
dubcanada 5y ago
General question, but who cares? Would you rather then reprint all of these stickers just to solve your OCD about proper spelling?
- xondono 5y ago
  If I’m spending thousands of dollars on these devices, I expect them to not screw up something like that. It would not be the first time someone sends a device for RMA to be told it’s out of warranty because it has been tampered, when in reality it was a manufacturing mistake.
- jacquesm 5y ago
  Labels like that are quite frequently a sign that you're dealing with a counterfeit product or something that has been places where it shouldn't.
- rasz 5y ago
  Misspelled stickers is how you recognize Chinese clones/counterfeits. Want some Amtech flux made in Colifomia? https://sudonull.com/post/100244-Amtech-fluxes-hoax-on-a-global-scale https://sudonull.com/post/100244-Amtech-fluxes-hoax-on-a-glo... https://ultrakeet.com.au/write-ups/fluxInfo https://ultrakeet.com.au/write-ups/fluxInfo
  - cbm-vic-20 5y ago
    Made with ♥ in Colifornia
    - walrus01 5y ago
      Designed by Abble in Califarnia
- newsclues 5y ago
  Yes.
- PeterisP 5y ago
  It's a sign that there wasn't any QA whatsoever - if they have missed something as obvious as a typo on a label, how can you assume that all the technical parameters of the components are up to spec?
ksec 5y ago
> (HPE did not care enough to investigate) That is a useful signal to avoid HPE.
jdlshore 5y ago
The article takes a while to get to the point, so here it is: 1. Article author discovers "American MegaTrAnds" sticker on a chip in high-end switches. 2. Author fears supply chain tampering. 3. Author contacts American Megatrends (AMI). Hijinks ensue. AMI eventually confirms that it's a typo. 4. Therefore, there is no supply chain tampering. 5. But author is concerned about what this means for the state of supply chain verification.
- walrus01 5y ago
  > But author is concerned about what this means for the state of supply chain verification. This is a brown M&Ms problem: https://conversableeconomist.blogspot.com/2020/10/the-no-brown-m-story-contract-theory-at.html https://conversableeconomist.blogspot.com/2020/10/the-no-bro... If they missed something as obvious as this, who knows what other problems are going on in supply chain security or total lack of QA.
  - NAR8789 5y ago
    I'd argue it's slightly different-- - no brown m&m's specifically calls out no brown m&ms in a list of requirements, and uses it as a canary for reading comprehension. - a misspelling is an "obvious" problem, but I suspect not called out anywhere as a specific requirement. "No brown m&ms" catches when people aren't paying detailed attention to your (presumably reasonably scoped) requirements doc. Asking people to catch all "obvious" problems holds them accountable to an unbounded guessing game, and you're far more likely to catch people out, simply because of differences in where they choose to focus.
    - walrus01 5y ago
      The misspelling means two different things to two categories of people... Those who haven't worked in electronics manufacturing are highly likely to say "oh it's just an honest mistake". Those who have worked in electronics manufacturing will immediately see it as a possibly scary sign of counterfeit components making their way into the supply chain. Same as what happened with counterfeit capacitors in east asia. Much like the early 1980s Van Halen tour example linked above, it's a reason for hitting the big red "OKAY, STOP EVERYTHING" button and re-check of all of the other components and supply chain going into the product.
bigmattystyles 5y ago
I had multiple thoughts - 1. If one is going to counterfeit the chip, counterfeiting the sticker wouldn't give you any pause - so it's probably just an error, however counterfeiters wouldn't have the review process that would catch typos. 2. Maybe it is a sneaky warranty workaround like a bank calls itself Banq on paperwork - https://en.wikipedia.org/wiki/Banq_(term) https://en.wikipedia.org/wiki/Banq_(term) So in the end, there's no real way for the end-user to know what's what and I agree with the author that the typo deserves an answer.
- toast0 5y ago
  Oddities in labels are a typical sign of a counterfeit. Yes, it's possible to counterfeit labels with good quality, but often it doesn't happen. If someone manages to do a good job of replicating the labels, hopefully they did a good job of replicating everything else. On the other hand, if QA didn't notice the label is misspelled, what other problems did they miss (some of which may be obvious to them if they look, but not obvious to me even)
ohazi 5y ago
> This was not caught by Dell, or even the STH team at first, it was a YouTube commenter. If that is how we as an industry are catching the easy plain-to-see stuff, that should scare everyone about what may be hard to see. And this is why the people shouting that "the direct cause of the log4j bug is our collective failure to give open source maintainers bags of money" are wrong. Don't get me wrong, you should give people bags of money if you want them to drop everything and go fix a critical bug after it's been discovered. But our collective ability to discover bugs is abysmal.
- fragmede 5y ago
  Which definition of our? Seems like it was found just fine. It didn't get the attention it needed until after it was publicly disclosed, however. The distinction is that even with a magic 100% infallible bug finding service, if the fixes don't get the attention they need, even with this magic service is, the problem's still going to be around. (While this perfect infallible service doesn't actually exist, fuzzers do find bugs in a semi-automated fashion. Still, getting someone to pay attention to the reports is an uphill battle due to false positives.)
  - yjftsjthsd-h 5y ago
    It was found by a random third party; I think it's reasonable to say that "our" should include the people who are getting paid to ship the thing.
    - mschuster91 5y ago
      > It was found by a random third party IIRC it was found and first exploited by Minecraft trolls. Lesson learned: the hardest opponent for any IT security team are online-game cheaters and trolls.
  - andrewxdiamond 5y ago
    > Seems like it was found just fine. Sure, something like 8 years after it was put in place.
MangoCoffee 5y ago
i used to work for a Dell contracted repair/shipping center. Corporate screw up happen all the time. there is one incident, before winter shopping holidays, Dell ordered thousands of 27" all in one pc for retail sales from a China OEM. the problem is the all in one is not all in one. the back of all in one is missing a PC! the all in one is now a 27" monitor with a empty lump where the PC supposed to be. Dell screw up. Dell end up selling these all in one PC monitors to Corporation for cheap.
- walrus01 5y ago
  Wouldn't the LCD panel in the all-in-one (missing actual x86-64 PC) be something like an LVDS interface between the intended motherboard and the panel? this would mean you couldn't just sell them off as cheap weird monitors since they would have no external HDMI or displayport connectors or interface to plug a computer into. Persons intending to use it as a monitor would have to go scrounge ebay for something like an HDMI-to-LVDS interface board and wire it up themselves.
  - MangoCoffee 5y ago
    it was made with a lump in the back as a functional monitor. it was supposed to be a all in one and now it just an "all in one" monitor. sorry if i wasn't clear.
    - walrus01 5y ago
      you weren't unclear I was just curious how they resolved it, since that's a really weird scenario...
  - Lammy 5y ago
    > or displayport They might be eDP if they're new enough, since LVDS doesn't provide enough bandwidth for modern high-res/high-refresh/high-DPI panels: https://www.embedded.com/edp-a-better-embedded-display-ecosystem/ https://www.embedded.com/edp-a-better-embedded-display-ecosy...
deleted 5y ago
[deleted]
Lammy 5y ago
And yet people loudly denounced Bloomberg for "The Big Hack" story every time it came up on HN: https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies https://www.bloomberg.com/news/features/2018-10-04/the-big-h... I believed it then, and I still believe it now. An "evil" BMC (like the ASPEED AST2600 mentioned in this article) is game over security-wise. BMCs are capable of flashing the BIOS/UEFI, capable of inserting arbitrary disk images as virtual CD/etc drives, capable of arbitrary keyboard/mouse input equivalent to having a hardware keyboard/mouse attached, like a remote evil-maid. If you had to pick one single thing to "pwn" in a server it would be the BMC. There's no way it's just a typo.
- wmf 5y ago
  It's obvious that an evil BMC would be game over because the BMC is the most trusted component in the system (for normal servers that don't have Titan or whatever). But there's no evidence of compromised BMCs actually happening.
  - Lammy 5y ago
    Is there evidence of it not happening? Can I do a reproducible build of a verified AMI BMC firmware and compare it against what comes flashed on my motherboard? They have a "MegaRAC Open Edition" but it says it's only for "OCP compliant platforms": https://github.com/opencomputeproject/HWMgmt-MegaRAC-OpenEdition https://github.com/opencomputeproject/HWMgmt-MegaRAC-OpenEdi...
    - wmf 5y ago
      Honestly the industry is a decade away from meeting that standard. And given that AMI is the lowest bidder we shouldn't even expect them to meet current best practices.
- rcxdude 5y ago
  It's very believable, but that's basically all that the bloomberg story had going for it. None of the actual concrete details (of which there were few) made any sense.
all2 5y ago
> William Barath December 13, 2021 At 5:27 pm > John Etulain of Seattle Washington registered those 2 domain names, and it is being served HTTP using STH’s SSL cert. > Staffer of yours, Pat? Huh.
- geerlingguy 5y ago
  In the linked video, it's explained that the domains were registered by STH to prevent them from falling into some malicious actor's hands.
ComputerGuru 5y ago
My Dell Precision 5520 (née XPS 9560) battery came with obvious spelling and grammatical errors.. just like the cheap Chinese clones I bought to replace it after a few years did (because Dell doesn’t sell the battery online). The clone I received was garbage and not up to the advertised specs so I tossed it and called my Dell sales rep and managed to get him to order me a new battery (it’s apparently not considered a user-replaceable part although it really is) and lo and behold the new one came with the same spelling errors as the one I got from China. I have zero faith in their supply chain.
- pixl97 5y ago
  Years later I'm still a bit pissed at Dell. Ordered a few hundred laptops with a office 2016 license key. They key came on a printed card with each laptop. About 33% of the cards were printed in the US. Every one of them worked. The other 2/3rds were printed in China. Half of those were already registered under random email addresses (as each had a randomly generated email that appeared to be at yahoo). It was a massive fight with Dell sales to rectify the situation.
- hef19898 5y ago
  Same manufacturer selling out-of-spec batches initially produced for Dell on the grey market maybe? As long as Dell parts are within spec, who really cares? If anything, geting in-spec parts from Dell from, what looks to be, the same supplier while direct orders from the supplier are out of spec should increase confidence in Dell's supply chain.
urbandw311er 5y ago
Slightly OT but why is American Megatrends called American Megatrends? It has always seems a slightly absurd name.
- SOLAR_FIELDS 5y ago
  TLDR: Just bad naming from the founders because they wanted to keep AMI initials * American Megatrends Inc. (AMI) was founded in 1985 by Subramonian Shankar and Pat Sarma with funds from a previous consulting venture, Access Methods Inc. (also AMI). Access Methods was a company run by Pat Sarma and his partner. After Access Methods successfully launched the AMIBIOS, there were legal issues among the owners of the company, resulting in Sarma buying out his partners. Access Methods still owned the rights to the AMIBIOS. Sarma had already started a company called Quintessential Consultants Inc. (QCI), and later set up an equal partnership with Shankar. By this time the AMIBIOS had become established and there was a need to keep the initials AMI. The partners renamed QCI as American Megatrends Inc., with the same initials as Access Methods Inc.; the renamed company then purchased AMIBIOS from Access Methods. Shankar became the president and Sarma the executive vice-president of this company. This partnership continued until 2001, when LSI Logic purchased the RAID Division of American Megatrends; American Megatrends then purchased all shares of the company owned by Sarma, making Shankar the majority owner. *
  - laurent92 5y ago
    It’s better than Atlassian Pty Ltd’s story, renamed from Atlassian Software Systems.
- deleted 5y ago
  [deleted]
- Lammy 5y ago
  I wonder if that's what inspired "Epic MegaGames, Inc."
  - selimthegrim 5y ago
    Or better yet this ear worm: https://mobile.twitter.com/atomicthumbs/status/1384721023786913801 https://mobile.twitter.com/atomicthumbs/status/1384721023786...
alliao 5y ago
I recently needed to contact Microsoft support, and looking at the support engineer's email address I literally felt like I was being targeted by fraudsters, easily the most suspicious company name I can think of. Shanghai Wicresoft Co.,Ltd
- jiggawatts 5y ago
  I deal with them regularly. They’re friendly and “helpful”, but they’re deliberately kept arms length from the Azure dev teams. This is a recipe for uselessness. If they can’t force the devs to do things properly, it won’t get done.
  - alliao 5y ago
    yeah, i find the whole azure support lacking. We have this server with 20+ disks, some in storage pool and now I have no way mapping them out to see which logical drive maps to which vm disk. the support after 3 months of back and forth just gave up and closed it citing known issue without official documentation...
    - jiggawatts 5y ago
      That's actually not that hard. The LUN numbers map through 1:1 to the guest OS. So the external LUN #1 is the internal LUN #1, etc... Then you can use the Windows OS storage pool commands to figure out the internal mapping. However, you do have a point -- the disk WWN or GUID or something properly unique ought be available via the Portal or the ARM REST API! That's far easier to automate robustly.
      - alliao 5y ago
        I thought that's how too, but maybe I got my mental model messed up. Will give it a proper visit, the last time I tried to downgrade some ssd to hdd resulted in windows boot loop. fun times, had to take a snapshot then fiddle it's registry in another vm so it'd use the last known working registry to get it back up.
rand49an 5y ago
Weird. A few weeks ago I turned up to a site that wasn't working after a power cut. They had two S5200's configured in a stack as core switches. Both had lost their firmware and we could only access the BIOS on them. All attempts to install/upgrade the firmware was a bust. Both had to be RMA'd and two days later we installed the replacements and 1 of those had the same issue.
jfrunyon 5y ago
... can we talk about the fact that it has a copyright year of 2003??? Sure, I totally believe there aren't a billion known vulnerabilities in that OOB management chip's code... the damn thing is old enough to vote!
- wmf 5y ago
  There are different philosophies about software copyright dates. A lot of people put the year of the initial release.
- kristopolous 5y ago
  Not necessarily. Simple things that don't have much context to interact with and are well defined can become stable. Many chips have been stable for decades without any serious modification with today's version being effectively identical to those of 25 years ago. It's not defacto secure, but age is not a reliable indicator without more information
  - jfrunyon 5y ago
    LOL. No, virtually nothing made 18 years ago is secure. In fact virtually nothing made a day ago is secure, but when something has been sitting around stagnating for 18 years that means the world has had ~6575x as many opportunities to find a vulnerability.
    - kristopolous 5y ago
      What on Earth are you talking about? There's countless things. Any random chip, let's take a max232, from 1987. It hasn't changed since 1987. They are widely used. There's countless simple chips there that have no "security issues" such as fram and pram chips that just store data and that's it. Many programmers are only accustomed to seeing the modern JavaScript systems with 5GB of dependencies to print hello world and sure, fine; a virtual machine on a browser, on a userland, on a kernel, there's lots of stuff there to go wrong. That's not what this necessarily is. Systems can be simple. You use 1970s era zilog-80 with code that hasn't changed in 30 years every day when you do things like wait at a stoplight, use municipal water when you turn on your faucet, put food in your microwave, turn on your light switch ... You probably interact with and certainly depend on more 1970s and 80s era hardware and software that's been faithfully reordered and reproduced for decades then you do any modern system. Motorola 6800 (not a typo) MOS 6502, and Intel 8080 clones are in every car, jet airplane, bus and train made in the past 30 years or so and they are still being used; oftentimes with no real modifications, for decades. They are used exactly because there's no surprises and everything is known and accounted for. Newness, the thing you advocate for is the vulnerability. Society entrusts things like interbank transfers (trillions a day) and international carrier switching on these systems. This strategy works so well that these systems are utterly invisible.
cassac 5y ago
Well at least it wasn’t American MAGAtrends?
- cassac 5y ago
  I dunno, as a typo, I think no matter which side you stand that’d be a funny typo. I know I laughed just thinking it.
Terretta 5y ago
TL;DR, quoting mid-bottom of the last page: > Final Words > All told, there is probably nothing to see here.
KingMachiavelli 5y ago
I wonder if this was a misguided attempt at A/B testing.
Eric_WVGG 5y ago
Okay so dumb misunderstanding whatever, am I alone in being ONE HUNDRED PERCENT ready to trade ill-gotten lucre in exchange for bootleg MACROSOFT WANDOWS branded streetwear
vzaliva 5y ago
This blog post is a lot of letters for 1-letter typo.
qwertox 5y ago
For those with old, tired eyes, there's a video from STH which is embedded in the article and talks about this issue: https://www.youtube.com/watch?v=dUVmDcv782k https://www.youtube.com/watch?v=dUVmDcv782k
edub 5y ago
I once ordered a large number of Compaq computers all identical for a credit union in Houston and found an esoteric combination of a specific hard drive firmware with a specific network card firmware that caused the machine to do very strange things when using an expensive piece of software (Hummingbird) to add a protocol to allow Windows 3.1 to be able to communicate with an IBM AS/400 to replace dumb terminals with terminal emulation on Windows. My memory may not be correct about everything I just said, but close enough. The point is that not every identical machine came with an identical hard drive or network card, but all considered comparable enough for Compaq to ship them to fulfill the order. I was incredibly impressed at the speed that Compaq escalated the issue from a first line support person to having a couple engineers drive out to the credit union to see it with their own eyes to determine a fix (I had no pull or special access, and we're only talking 150 computers). It was convenient that this happened in Houston so the engineers were in close proximity. This article makes me skeptical that HP would handle it the same way today that Compaq did at the time based on HPE's response to STH. I hate the buying process with both Dell and HP, but they are the brands I buy from when it is a server or workstation, because I love the build quality for both brands on their enterprise servers and high end workstations. I've never owned a consumer device from either brand, but I would not have the same expectations on a consumer device from them.
blunte 5y ago
Regardless of the reason for the misspelled stickers, they are so prominent and visible that any (western?) employee to saw the boards should have noticed. At the very least, there should be some QC human or even automated visual software which would identify oddities. In fact, what's the point of putting a sticker on if nobody is going to look at it?
- hef19898 5y ago
  Marketing? Without seeing the internal quality check requirements, and how any deviations are to be treated and were treated in that particular case, it is kind of pointless in reading to much into it.

345 points by geerlingguy 5y ago

Time for someone to dump the code from the "American Megatrans" chip and compare it with one that says "American Megatrends".

Rarok 5y ago

It's just a sticker, but a sticker that has a license. In the world where we live I would expect that something like this has to be aproved by a legal team.

Am_I_Right 5y ago

Neither Dell nor HPE manufactures switches (the former never has, the latter hasn't for the past few years). So, these are all sourced from an OEM like Edgecore. And, someone at that OEM ordered a bunch of misspelled stickers. Easy mistake to make, if the latin alphabet is literally foreign to you. And if you think that sticker is bad? Wait until you see the actual firmware, oh boy... (I had some fun Edgecore LACP bugs take down an pretty sizable network. Things got slightly better once they moved to Linux-based firmware, but never to the point that their kit was, like, entirely reliable...)

bluedino 5y ago
Isn't the fear that they are imitation parts?
- mxxx 5y ago
  Yes, but they’re not. It was just a typo.
- Am_I_Right 5y ago
  I'm not entirely sure what the fear is. The AST2600 the sticker seems to have been pasted to is a pretty complicated IC with (and this is the important part) user-upgradeable flash to begin with. So, you want to do a supply-chain attack? Simply reflash the genuine modules. No need to spend more. On the other hand: you want to save a few bucks? Possibly do a knock-off chip, but you're definitely not going to bother with the firmware. Too expensive! This is definitely a case of "trying to save a few bucks". Both Dell and HPE are in a race to the bottom, and the sticker being indicative of anything significant beyond that is... unlikely...
- NAR8789 5y ago
  I think so, but assuming that is the case... what's to stop a shady chipmaker from printing properly-spelled "American Megatrends" stickers? More generally... are there any actual protections offered by genuine stickers? The article makes this out to be a major supply chain security issue, and that only makes sense if branding stickers are actually reliable for validation purposes. But that seems... nonsensical? Wouldn't stickers be very easy to forge? But, I don't work in supply chains. Anyone with better expertise in this area able to chime in? I will admit I skimmed the article, because it is long and overly-detailed for my level of interest, and because it lacks summary sections.
  - jcrawfordor 5y ago
    It's not at all that a properly spelled sticker gives assurance that it's not counterfeit... it's just that a misspelled sticker is such an obvious sign of a potential counterfeit that it's basically the #1 thing that any counterfeit/suspect items program teaches people to look for. Most people working on counterfeits don't speak English so it's very easy for these kinds of mistakes to slip through, and on the other hand they're rarely made by the genuine manufacturer which usually has a process to check for this kind of thing even if the engineering work is done in a non-English speaking country (most of all that the logos usually come from off-the-shelf art files from the marketing department, so no one's even typing the name to make a mistake). Almost any corporate or institutional counterfeit or supply chain security program will explicitly teach you: if anything is misspelled or shows other obvious mistakes, hold the part as a suspected counterfeit. It's a pretty good quality indication. So of course manufacturers do genuinely make spelling mistakes sometimes, but this context makes it a pretty embarrassing and serious thing to do. It's like your bank misspelling their name in an account notification: sure, in some extremely theoretical sense it doesn't mean anything, but in practice they're giving you exactly the signal that everyone tells you to check for to identify phishing, and it raises questions about their processes that they let it slip through.
ksec 5y ago
Well, for many stickers, they have to be ordered from original vendor / AMI. I guess this is not the case here. Turns out it is coming from AMI, but AMI Taiwan. >AMI Taiwan needed to get license stickers for the local market. Instead of using the “American Megatrends” MegaRAC PM sticker template, it decided to make its own that had the misspelling.
- hef19898 5y ago
  So really nothing to see here, other than an AMI subsidiary got a bunch of stickers with a typo and distributed hose to suppliers.
walrus01 5y ago
edgecore is just a marketing name, the actual company is accton they're generally a competitor of companies like compal, clevo, quanta. All well known in Taiwan if you're in the business of having 3rd parties manufacture your stuff.
- alliao 5y ago
  I'm pretty sure accton's compal's communications subsidiary
  - walrus01 5y ago
    that is a good point, not something I'd had reason to think about since 2006 or so. Compal is quite a behemoth. https://en.wikipedia.org/wiki/Compal_Electronics https://en.wikipedia.org/wiki/Compal_Electronics US $26 billion revenues. Most people have never heard of it, only its consumer facing brands like Ignitenet.
- merb 5y ago
  it's crazy how much tech companies taiwan has. i'm pretty sure that this has conflict potential with china (i.e. china with the eu/us).
  - walrus01 5y ago
    some 15-20 years ago most of the big taiwanese electronics manufacturing companies (top tier x86-64 motherboard makers would be a good example) moved a lot of their factory operations to mainland china, for lower cost labor. it's very interconnected now. there's a fascinating yearly trade show of taiwanese manufacturers: https://www.computextaipei.com.tw/en/index.html https://www.computextaipei.com.tw/en/index.html
    - c_o_n_v_e_x 5y ago
      I frequently work with Taiwanese manufacturers in the industrial PC space. Depending on what the client needs, I can specify whether the equipment is made in China (Suzhou) or Taiwan. Having the equipment made in Taiwan costs at least 10 to 15% more.

dubcanada 5y ago

General question, but who cares? Would you rather then reprint all of these stickers just to solve your OCD about proper spelling?

xondono 5y ago
If I’m spending thousands of dollars on these devices, I expect them to not screw up something like that. It would not be the first time someone sends a device for RMA to be told it’s out of warranty because it has been tampered, when in reality it was a manufacturing mistake.
jacquesm 5y ago
Labels like that are quite frequently a sign that you're dealing with a counterfeit product or something that has been places where it shouldn't.
rasz 5y ago
Misspelled stickers is how you recognize Chinese clones/counterfeits. Want some Amtech flux made in Colifomia? https://sudonull.com/post/100244-Amtech-fluxes-hoax-on-a-global-scale https://sudonull.com/post/100244-Amtech-fluxes-hoax-on-a-glo... https://ultrakeet.com.au/write-ups/fluxInfo https://ultrakeet.com.au/write-ups/fluxInfo
- cbm-vic-20 5y ago
  Made with ♥ in Colifornia
  - walrus01 5y ago
    Designed by Abble in Califarnia
newsclues 5y ago
Yes.
PeterisP 5y ago
It's a sign that there wasn't any QA whatsoever - if they have missed something as obvious as a typo on a label, how can you assume that all the technical parameters of the components are up to spec?

ksec 5y ago

> (HPE did not care enough to investigate) That is a useful signal to avoid HPE.

jdlshore 5y ago

The article takes a while to get to the point, so here it is: 1. Article author discovers "American MegaTrAnds" sticker on a chip in high-end switches. 2. Author fears supply chain tampering. 3. Author contacts American Megatrends (AMI). Hijinks ensue. AMI eventually confirms that it's a typo. 4. Therefore, there is no supply chain tampering. 5. But author is concerned about what this means for the state of supply chain verification.

walrus01 5y ago
> But author is concerned about what this means for the state of supply chain verification. This is a brown M&Ms problem: https://conversableeconomist.blogspot.com/2020/10/the-no-brown-m-story-contract-theory-at.html https://conversableeconomist.blogspot.com/2020/10/the-no-bro... If they missed something as obvious as this, who knows what other problems are going on in supply chain security or total lack of QA.
- NAR8789 5y ago
  I'd argue it's slightly different-- - no brown m&m's specifically calls out no brown m&ms in a list of requirements, and uses it as a canary for reading comprehension. - a misspelling is an "obvious" problem, but I suspect not called out anywhere as a specific requirement. "No brown m&ms" catches when people aren't paying detailed attention to your (presumably reasonably scoped) requirements doc. Asking people to catch all "obvious" problems holds them accountable to an unbounded guessing game, and you're far more likely to catch people out, simply because of differences in where they choose to focus.
  - walrus01 5y ago
    The misspelling means two different things to two categories of people... Those who haven't worked in electronics manufacturing are highly likely to say "oh it's just an honest mistake". Those who have worked in electronics manufacturing will immediately see it as a possibly scary sign of counterfeit components making their way into the supply chain. Same as what happened with counterfeit capacitors in east asia. Much like the early 1980s Van Halen tour example linked above, it's a reason for hitting the big red "OKAY, STOP EVERYTHING" button and re-check of all of the other components and supply chain going into the product.

bigmattystyles 5y ago

I had multiple thoughts - 1. If one is going to counterfeit the chip, counterfeiting the sticker wouldn't give you any pause - so it's probably just an error, however counterfeiters wouldn't have the review process that would catch typos. 2. Maybe it is a sneaky warranty workaround like a bank calls itself Banq on paperwork - https://en.wikipedia.org/wiki/Banq_(term) https://en.wikipedia.org/wiki/Banq_(term) So in the end, there's no real way for the end-user to know what's what and I agree with the author that the typo deserves an answer.

toast0 5y ago
Oddities in labels are a typical sign of a counterfeit. Yes, it's possible to counterfeit labels with good quality, but often it doesn't happen. If someone manages to do a good job of replicating the labels, hopefully they did a good job of replicating everything else. On the other hand, if QA didn't notice the label is misspelled, what other problems did they miss (some of which may be obvious to them if they look, but not obvious to me even)

ohazi 5y ago

> This was not caught by Dell, or even the STH team at first, it was a YouTube commenter. If that is how we as an industry are catching the easy plain-to-see stuff, that should scare everyone about what may be hard to see. And this is why the people shouting that "the direct cause of the log4j bug is our collective failure to give open source maintainers bags of money" are wrong. Don't get me wrong, you should give people bags of money if you want them to drop everything and go fix a critical bug after it's been discovered. But our collective ability to discover bugs is abysmal.

fragmede 5y ago
Which definition of our? Seems like it was found just fine. It didn't get the attention it needed until after it was publicly disclosed, however. The distinction is that even with a magic 100% infallible bug finding service, if the fixes don't get the attention they need, even with this magic service is, the problem's still going to be around. (While this perfect infallible service doesn't actually exist, fuzzers do find bugs in a semi-automated fashion. Still, getting someone to pay attention to the reports is an uphill battle due to false positives.)
- yjftsjthsd-h 5y ago
  It was found by a random third party; I think it's reasonable to say that "our" should include the people who are getting paid to ship the thing.
  - mschuster91 5y ago
    > It was found by a random third party IIRC it was found and first exploited by Minecraft trolls. Lesson learned: the hardest opponent for any IT security team are online-game cheaters and trolls.
- andrewxdiamond 5y ago
  > Seems like it was found just fine. Sure, something like 8 years after it was put in place.

MangoCoffee 5y ago

i used to work for a Dell contracted repair/shipping center. Corporate screw up happen all the time. there is one incident, before winter shopping holidays, Dell ordered thousands of 27" all in one pc for retail sales from a China OEM. the problem is the all in one is not all in one. the back of all in one is missing a PC! the all in one is now a 27" monitor with a empty lump where the PC supposed to be. Dell screw up. Dell end up selling these all in one PC monitors to Corporation for cheap.

walrus01 5y ago
Wouldn't the LCD panel in the all-in-one (missing actual x86-64 PC) be something like an LVDS interface between the intended motherboard and the panel? this would mean you couldn't just sell them off as cheap weird monitors since they would have no external HDMI or displayport connectors or interface to plug a computer into. Persons intending to use it as a monitor would have to go scrounge ebay for something like an HDMI-to-LVDS interface board and wire it up themselves.
- MangoCoffee 5y ago
  it was made with a lump in the back as a functional monitor. it was supposed to be a all in one and now it just an "all in one" monitor. sorry if i wasn't clear.
  - walrus01 5y ago
    you weren't unclear I was just curious how they resolved it, since that's a really weird scenario...
- Lammy 5y ago
  > or displayport They might be eDP if they're new enough, since LVDS doesn't provide enough bandwidth for modern high-res/high-refresh/high-DPI panels: https://www.embedded.com/edp-a-better-embedded-display-ecosystem/ https://www.embedded.com/edp-a-better-embedded-display-ecosy...

deleted 5y ago

[deleted]

Lammy 5y ago

And yet people loudly denounced Bloomberg for "The Big Hack" story every time it came up on HN: https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies https://www.bloomberg.com/news/features/2018-10-04/the-big-h... I believed it then, and I still believe it now. An "evil" BMC (like the ASPEED AST2600 mentioned in this article) is game over security-wise. BMCs are capable of flashing the BIOS/UEFI, capable of inserting arbitrary disk images as virtual CD/etc drives, capable of arbitrary keyboard/mouse input equivalent to having a hardware keyboard/mouse attached, like a remote evil-maid. If you had to pick one single thing to "pwn" in a server it would be the BMC. There's no way it's just a typo.

wmf 5y ago
It's obvious that an evil BMC would be game over because the BMC is the most trusted component in the system (for normal servers that don't have Titan or whatever). But there's no evidence of compromised BMCs actually happening.
- Lammy 5y ago
  Is there evidence of it not happening? Can I do a reproducible build of a verified AMI BMC firmware and compare it against what comes flashed on my motherboard? They have a "MegaRAC Open Edition" but it says it's only for "OCP compliant platforms": https://github.com/opencomputeproject/HWMgmt-MegaRAC-OpenEdition https://github.com/opencomputeproject/HWMgmt-MegaRAC-OpenEdi...
  - wmf 5y ago
    Honestly the industry is a decade away from meeting that standard. And given that AMI is the lowest bidder we shouldn't even expect them to meet current best practices.
rcxdude 5y ago
It's very believable, but that's basically all that the bloomberg story had going for it. None of the actual concrete details (of which there were few) made any sense.

all2 5y ago

> William Barath December 13, 2021 At 5:27 pm > John Etulain of Seattle Washington registered those 2 domain names, and it is being served HTTP using STH’s SSL cert. > Staffer of yours, Pat? Huh.

geerlingguy 5y ago
In the linked video, it's explained that the domains were registered by STH to prevent them from falling into some malicious actor's hands.

ComputerGuru 5y ago

My Dell Precision 5520 (née XPS 9560) battery came with obvious spelling and grammatical errors.. just like the cheap Chinese clones I bought to replace it after a few years did (because Dell doesn’t sell the battery online). The clone I received was garbage and not up to the advertised specs so I tossed it and called my Dell sales rep and managed to get him to order me a new battery (it’s apparently not considered a user-replaceable part although it really is) and lo and behold the new one came with the same spelling errors as the one I got from China. I have zero faith in their supply chain.

pixl97 5y ago
Years later I'm still a bit pissed at Dell. Ordered a few hundred laptops with a office 2016 license key. They key came on a printed card with each laptop. About 33% of the cards were printed in the US. Every one of them worked. The other 2/3rds were printed in China. Half of those were already registered under random email addresses (as each had a randomly generated email that appeared to be at yahoo). It was a massive fight with Dell sales to rectify the situation.
hef19898 5y ago
Same manufacturer selling out-of-spec batches initially produced for Dell on the grey market maybe? As long as Dell parts are within spec, who really cares? If anything, geting in-spec parts from Dell from, what looks to be, the same supplier while direct orders from the supplier are out of spec should increase confidence in Dell's supply chain.

urbandw311er 5y ago

Slightly OT but why is American Megatrends called American Megatrends? It has always seems a slightly absurd name.

SOLAR_FIELDS 5y ago
TLDR: Just bad naming from the founders because they wanted to keep AMI initials * American Megatrends Inc. (AMI) was founded in 1985 by Subramonian Shankar and Pat Sarma with funds from a previous consulting venture, Access Methods Inc. (also AMI). Access Methods was a company run by Pat Sarma and his partner. After Access Methods successfully launched the AMIBIOS, there were legal issues among the owners of the company, resulting in Sarma buying out his partners. Access Methods still owned the rights to the AMIBIOS. Sarma had already started a company called Quintessential Consultants Inc. (QCI), and later set up an equal partnership with Shankar. By this time the AMIBIOS had become established and there was a need to keep the initials AMI. The partners renamed QCI as American Megatrends Inc., with the same initials as Access Methods Inc.; the renamed company then purchased AMIBIOS from Access Methods. Shankar became the president and Sarma the executive vice-president of this company. This partnership continued until 2001, when LSI Logic purchased the RAID Division of American Megatrends; American Megatrends then purchased all shares of the company owned by Sarma, making Shankar the majority owner. *
- laurent92 5y ago
  It’s better than Atlassian Pty Ltd’s story, renamed from Atlassian Software Systems.
deleted 5y ago
[deleted]
Lammy 5y ago
I wonder if that's what inspired "Epic MegaGames, Inc."
- selimthegrim 5y ago
  Or better yet this ear worm: https://mobile.twitter.com/atomicthumbs/status/1384721023786913801 https://mobile.twitter.com/atomicthumbs/status/1384721023786...

alliao 5y ago

I recently needed to contact Microsoft support, and looking at the support engineer's email address I literally felt like I was being targeted by fraudsters, easily the most suspicious company name I can think of. Shanghai Wicresoft Co.,Ltd

jiggawatts 5y ago
I deal with them regularly. They’re friendly and “helpful”, but they’re deliberately kept arms length from the Azure dev teams. This is a recipe for uselessness. If they can’t force the devs to do things properly, it won’t get done.
- alliao 5y ago
  yeah, i find the whole azure support lacking. We have this server with 20+ disks, some in storage pool and now I have no way mapping them out to see which logical drive maps to which vm disk. the support after 3 months of back and forth just gave up and closed it citing known issue without official documentation...
  - jiggawatts 5y ago
    That's actually not that hard. The LUN numbers map through 1:1 to the guest OS. So the external LUN #1 is the internal LUN #1, etc... Then you can use the Windows OS storage pool commands to figure out the internal mapping. However, you do have a point -- the disk WWN or GUID or something properly unique ought be available via the Portal or the ARM REST API! That's far easier to automate robustly.
    - alliao 5y ago
      I thought that's how too, but maybe I got my mental model messed up. Will give it a proper visit, the last time I tried to downgrade some ssd to hdd resulted in windows boot loop. fun times, had to take a snapshot then fiddle it's registry in another vm so it'd use the last known working registry to get it back up.

rand49an 5y ago

Weird. A few weeks ago I turned up to a site that wasn't working after a power cut. They had two S5200's configured in a stack as core switches. Both had lost their firmware and we could only access the BIOS on them. All attempts to install/upgrade the firmware was a bust. Both had to be RMA'd and two days later we installed the replacements and 1 of those had the same issue.

jfrunyon 5y ago

... can we talk about the fact that it has a copyright year of 2003??? Sure, I totally believe there aren't a billion known vulnerabilities in that OOB management chip's code... the damn thing is old enough to vote!

wmf 5y ago
There are different philosophies about software copyright dates. A lot of people put the year of the initial release.
kristopolous 5y ago
Not necessarily. Simple things that don't have much context to interact with and are well defined can become stable. Many chips have been stable for decades without any serious modification with today's version being effectively identical to those of 25 years ago. It's not defacto secure, but age is not a reliable indicator without more information
- jfrunyon 5y ago
  LOL. No, virtually nothing made 18 years ago is secure. In fact virtually nothing made a day ago is secure, but when something has been sitting around stagnating for 18 years that means the world has had ~6575x as many opportunities to find a vulnerability.
  - kristopolous 5y ago
    What on Earth are you talking about? There's countless things. Any random chip, let's take a max232, from 1987. It hasn't changed since 1987. They are widely used. There's countless simple chips there that have no "security issues" such as fram and pram chips that just store data and that's it. Many programmers are only accustomed to seeing the modern JavaScript systems with 5GB of dependencies to print hello world and sure, fine; a virtual machine on a browser, on a userland, on a kernel, there's lots of stuff there to go wrong. That's not what this necessarily is. Systems can be simple. You use 1970s era zilog-80 with code that hasn't changed in 30 years every day when you do things like wait at a stoplight, use municipal water when you turn on your faucet, put food in your microwave, turn on your light switch ... You probably interact with and certainly depend on more 1970s and 80s era hardware and software that's been faithfully reordered and reproduced for decades then you do any modern system. Motorola 6800 (not a typo) MOS 6502, and Intel 8080 clones are in every car, jet airplane, bus and train made in the past 30 years or so and they are still being used; oftentimes with no real modifications, for decades. They are used exactly because there's no surprises and everything is known and accounted for. Newness, the thing you advocate for is the vulnerability. Society entrusts things like interbank transfers (trillions a day) and international carrier switching on these systems. This strategy works so well that these systems are utterly invisible.

cassac 5y ago

Well at least it wasn’t American MAGAtrends?

cassac 5y ago
I dunno, as a typo, I think no matter which side you stand that’d be a funny typo. I know I laughed just thinking it.

Terretta 5y ago

TL;DR, quoting mid-bottom of the last page: > Final Words > All told, there is probably nothing to see here.

KingMachiavelli 5y ago

I wonder if this was a misguided attempt at A/B testing.

Eric_WVGG 5y ago

Okay so dumb misunderstanding whatever, am I alone in being ONE HUNDRED PERCENT ready to trade ill-gotten lucre in exchange for bootleg MACROSOFT WANDOWS branded streetwear

vzaliva 5y ago

This blog post is a lot of letters for 1-letter typo.

qwertox 5y ago

For those with old, tired eyes, there's a video from STH which is embedded in the article and talks about this issue: https://www.youtube.com/watch?v=dUVmDcv782k https://www.youtube.com/watch?v=dUVmDcv782k

edub 5y ago

I once ordered a large number of Compaq computers all identical for a credit union in Houston and found an esoteric combination of a specific hard drive firmware with a specific network card firmware that caused the machine to do very strange things when using an expensive piece of software (Hummingbird) to add a protocol to allow Windows 3.1 to be able to communicate with an IBM AS/400 to replace dumb terminals with terminal emulation on Windows. My memory may not be correct about everything I just said, but close enough. The point is that not every identical machine came with an identical hard drive or network card, but all considered comparable enough for Compaq to ship them to fulfill the order. I was incredibly impressed at the speed that Compaq escalated the issue from a first line support person to having a couple engineers drive out to the credit union to see it with their own eyes to determine a fix (I had no pull or special access, and we're only talking 150 computers). It was convenient that this happened in Houston so the engineers were in close proximity. This article makes me skeptical that HP would handle it the same way today that Compaq did at the time based on HPE's response to STH. I hate the buying process with both Dell and HP, but they are the brands I buy from when it is a server or workstation, because I love the build quality for both brands on their enterprise servers and high end workstations. I've never owned a consumer device from either brand, but I would not have the same expectations on a consumer device from them.

blunte 5y ago

Regardless of the reason for the misspelled stickers, they are so prominent and visible that any (western?) employee to saw the boards should have noticed. At the very least, there should be some QC human or even automated visual software which would identify oddities. In fact, what's the point of putting a sticker on if nobody is going to look at it?

hef19898 5y ago
Marketing? Without seeing the internal quality check requirements, and how any deviations are to be treated and were treated in that particular case, it is kind of pointless in reading to much into it.