11 ms·

The article takes a while to get to the point, so here it is: 1. Article author discovers "American MegaTrAnds" sticker on a chip in high-end switches. 2. Aut

The article takes a while to get to the point, so here it is: 1. Article author discovers "American MegaTrAnds" sticker on a chip in high-end switches. 2. Author fears supply chain tampering. 3. Author contacts American Megatrends (AMI). Hijinks ensue. AMI eventually confirms that it's a typo. 4. Therefore, there is no supply chain tampering. 5. But author is concerned about what this means for the state of supply chain verification.

walrus01 5y ago
> But author is concerned about what this means for the state of supply chain verification. This is a brown M&Ms problem: https://conversableeconomist.blogspot.com/2020/10/the-no-brown-m-story-contract-theory-at.html https://conversableeconomist.blogspot.com/2020/10/the-no-bro... If they missed something as obvious as this, who knows what other problems are going on in supply chain security or total lack of QA.
- NAR8789 5y ago
  I'd argue it's slightly different-- - no brown m&m's specifically calls out no brown m&ms in a list of requirements, and uses it as a canary for reading comprehension. - a misspelling is an "obvious" problem, but I suspect not called out anywhere as a specific requirement. "No brown m&ms" catches when people aren't paying detailed attention to your (presumably reasonably scoped) requirements doc. Asking people to catch all "obvious" problems holds them accountable to an unbounded guessing game, and you're far more likely to catch people out, simply because of differences in where they choose to focus.
  - walrus01 5y ago
    The misspelling means two different things to two categories of people... Those who haven't worked in electronics manufacturing are highly likely to say "oh it's just an honest mistake". Those who have worked in electronics manufacturing will immediately see it as a possibly scary sign of counterfeit components making their way into the supply chain. Same as what happened with counterfeit capacitors in east asia. Much like the early 1980s Van Halen tour example linked above, it's a reason for hitting the big red "OKAY, STOP EVERYTHING" button and re-check of all of the other components and supply chain going into the product.
    - NAR8789 5y ago
      Ahhh, that's the critical piece of context I was missing. Thank you! Article didn't fully explain why the sticker matters so much, so that left me scratching my head. (my gut reaction was "well, wouldn't a genuine sticker still be easy to counterfeit?). But based on your explanation this is more of a smell that everyone in the electronics manufacturing space is culturally attuned to. So, the fact that it slipped by so many people does indicate a slippage of norms. If I draw a comparison to software to bring it closer to something familiar to me... would this be like inconsistent variable name formatting? CamelCase in some places and snake_case in others? To an outsider, arguably inconsequential, so insisting on consistency here might seem OCD to them, but to someone who's worked in the space it's actually a useful marker of general detail orientation.
      - walrus01 5y ago
        In software it would be like if you found evidence that one of your trusted developers had outsourced their own job to an unknown person in India, who was controlling their home office desktop PC by some sort of remote desktop tool and adding code/making commits on their behalf.
      - bopbeepboop 5y ago
        This would be like if one day you found “MicronSoft Word” installed on your computer, when you went to write a document. Or your compiler started saying it was “Jawa 9”. Maybe someone just typo’d the name in an update.
      - 3np 5y ago
        > would this be like inconsistent variable name formatting? CamelCase in some places and snake_case in others? I’d say it’s more like misspelling your own company name in the JPEG logo in the HTML welcome email sent out to new users, and no one notices for a year.
      - outworlder 5y ago
        > If I draw a comparison to software to bring it closer to something familiar to me... would this be like inconsistent variable name formatting This would be more like finding a container coming from dockahub.com and then ultimately finding out that it was a legitimate typo.
  - hef19898 5y ago
    Also focusing on unimportant, but obvious, issues takes resources away from controlling for not so obvious but critical issues. That and given the fact that checking English spelling isn't a trivial thing for Asians (as it would be the other way round) means this is a non-issue. Will QC be improved after that? Potentially. Will there be more scrutiny around that particular supplier? I would think so. Is it a general supply chain problem? No.
- formerly_proven 5y ago
  I'm not so sure. These BMCs are dumb SoCs like any other (just with specialized I/O), their firmware comes from an external SPI flash. I doubt there is anything AMI specific at all in these chips. Looks like it's basically a license sticker some worker is putting on these boards after they're all assembled and tested. I can see how these stickers are there, maybe for contractual/legal reasons, without being a critical part of the BMC board BOM (1x roll of "AMI loicense stickers").
  - xondono 5y ago
    These “dumb SoCs” are one of the biggest security holes in a lot of high end equipment. It would not be the first time someone finds exploitable firmware bugs and vulnerable BMCs through Shodan.
- stronglikedan 5y ago
  I work in an industry rife with typos, and can assure you that typos are missed all the time, and it doesn't indicate a greater QA problem or lack of QA. It's just a missed typo.
  - mistrial9 5y ago
    in other contexts this is referred to as "alarm fatigue" .. there is no assurance and masking it is not the move here
  - treesknees 5y ago
    Where I work, we treat grammar and spelling errors as a sev 2 bug (sev 1 being highest.) It will quickly erode trust in your product even if everything else is working fine. All of our customer-facing text (UI/CLI/API names, labels, error messages, etc.) go through QA and also a separate documentation team for consistency and spelling checks.
    - Syonyk 5y ago
      Lul your just overeacting! </s>
    - cesaref 5y ago
      Here in the UK, one of the national newspapers, The Guardian was so famous for typing mistakes that it was nicknamed 'The Grauniad' by Private Eye, which is a satirical magazine. It's actually so well know that http://www.grauniad.co.uk http://www.grauniad.co.uk redirects to the correct sight
  - JohnJamesRambo 5y ago
    Idk man this isn’t just a typo it is the company name. Would Nike misspell their name on a product?
    - robbedpeter 5y ago
      Nuke sneakers would be kinda cool actually. Maybe Nake - proud sponsors of this year's Olympics streakers!
- ok_dad 5y ago
  I used to work in incoming inspection department for a medical device manufacturer. If the switch/router company did a "first article inspection" (FAI) and missed this (or missed it on a later incoming inspection), I would agree, but if they found it and called AMI and got the answer it was a typo, then they would note that on the inspection sheet and move on.
  - HeyLaughingBoy 5y ago
    You weren't responsible for my comment https://news.ycombinator.com/item?id=29559534 https://news.ycombinator.com/item?id=29559534 above, were you ;-)
    - ok_dad 5y ago
      We were super anal about things there, but I always made sure scrap was actually scrap and not recoverable. Your description sounds like a stupid situation and they should have just reinspected those blades.
  - fckgw 5y ago
    The problem is, as pointed out in the example, is that the memo the writer got from Dell that came from AMI informing Dell about the typo contained an image that the own writer sent to AMI. Meaning AMI had no idea about this typo before they were contacted by the writer.
- aaron695 5y ago
  > This is a brown M&Ms problem No it's not. Brown M&Ms gives you one thing to check. This misspelling is part of 1000's of things that could be checked. A misspelling can indicate counterfeits, a serious issue on critical infrastructure, unlike the Brown M&M's which is just a joke check. Yes it's weird someone illegally counterfeiting will misspell, but it happens for complex reasons [1] [1] A recent example - "Fake Vaccination Cards Seized At O’Hare Were Spotted Because They Had Misspelled Words" - https://blockclubchicago.org/2021/09/07/fake-vaccination-cards-seized-at-ohare-because-they-had-misspelled-words/ https://blockclubchicago.org/2021/09/07/fake-vaccination-car...
- PaulHoule 5y ago
  It's more of a conspicuous display of incompetence. Some politicians make a point of displaying this so that donors know they aren't going to be somebody smart, ambitious and dangerous like Ralph Nader. One example was Karen Bass, who bent over backwards to praise Scientology once.
- wyldfire 5y ago
  Maybe they're aware of the supply chain interdiction and willing or motivated to let it continue. You'd think that the integrity of their product would be paramount concern. But how might their attitude change if a state decided to intervene? Or what if they have short-sighted leadership making the best decisions for them individually?
- PragmaticPulp 5y ago
  More likely that nobody noticed the typo until it was too late and they didn’t want to halt an entire production run just to replace a sticker that most people will never see. Stopping the production run and delaying shipments would have a palpable impact on shipment numbers. Most likely it was noticed, corrected for later batches, and they made the call to run this round anyway because it had zero effect on function. (Source: If you work in hardware long enough, you’ll have to make similar calls like this eventually)
  - hef19898 5y ago
    For us older folks remembering what "stamps" are, and that people used to collect those: The most valuable ones are those with printing errors, e.g. typos. If you produce enough stuff at scale mistakes happen. And I would assume the stickers are much less controlled and monitored then the chips / hardware itself. No idea how it works in electronics, in aerospace so you would do thorough testing on the affected units. And if they meet the spec, someone would make a call regarding relabeling. Considering how expensive relabeling tends to be, it isn't unreasonable to assume the parts just got a concession for the labels. I wouldn't read to much into it.
- tomxor 5y ago
  > If they missed something as obvious as this, who knows what other problems are going on in supply chain security or total lack of QA. I'm pretty sure "they" that design the stickers are very different from "they" that design the chip... Similarly chip verification has rather a different skill set and attention to detail than someone doing "sticker QA". Also I doubt anyone was doing sticker QA, I'd guess that this was probably an outsourced mistake (I used to work in printing when I was younger - the mistakes you wouldn't believe).
1970-01-01 5y ago
Yes. TL;DR: Chip does not meet MILSPEC or any other spec. People are OK with it. The chips really should be de-capped and researched.
ricardobeat 5y ago
They said there is no tampering. Someone should x-ray this chip vs a correctly labeled one to verify.
- wmf 5y ago
  AMI is a firmware company so you'd check the hash of the BMC flash contents.
  - xondono 5y ago
    Although it would be much (much) more elaborate, a counterfeit BMC could have hidden ROMs or bootloaders capable of opening backdoors into running machines.
    - wmf 5y ago
      Note that the BMC chip comes from a different company (ASPEED) who has not been implicated in this "stickergate".
      - bruce343434 5y ago
        perhaps off-topic, but where does the -gate suffix come from? It seems it can be replaced by "troubles". Gamer troubles. Sticker troubles. But what does -gate even mean?
        freeman478 5y ago
        I think it comes from https://en.wikipedia.org/wiki/Watergate_scandal https://en.wikipedia.org/wiki/Watergate_scandal
        handrous 5y ago
        Ah, good old Watergate-gate.
        CBLT 5y ago
foobiekr 5y ago
This is laughably sloppy. One of my favorite customers actually took a calorimeter to the LEDs that were in one of our deliveries of router chassis. They felt that the Amber wasn’t quite Amber enough and so they measured it. They were right while it was amber it wasn’t spec amber. So they shipped it all back. We were very embarrassed and supply chain was given a dressing down. They missed a parts quality issue. A typo like this is extremely indicative of a sloppy organization.
- detaro 5y ago
  What is the relevant spec for colors here? Is there some ISO or whatever for signal lights on equipment defining it?
  - fxtentacle 5y ago
    Parent comment probably meant "colorimeter" which will measure colors in either spectral distribution or XYZ color space. So you could say something like RAL color # 123 +- 5% in XYZ space. Or you can just specify: That color needs to be closer to RAL #123 than to any other RAL number, so then you'd also have an implicit definition of the valid color range.
    - detaro 5y ago
      What the "something like" is exactly and where it came from is kind of my question.
  - jwandborg 5y ago
    It might be regulatory concerns, maybe they hadn't licensed that particular wavelength of orange for local broadcasting purposes, or EMI concerns with regards to the the off-white spectrum, at least if I let my thoughts run without moderation.
    - foobiekr 5y ago
      No, this really was just ethernet interface front panel LEDs. The customer viewed this as indicative of poor quality control.
deleted 5y ago
[deleted]
howdydoo 5y ago
> supply chain tampering Naive question, I don't know much about this industry. But if someone from China or the NSA wanted to make counterfeit chips, why would they risk putting something different on the label? It seems like a weird place to draw the line. Would the NSA really say "oh no, we can't violate anyone's trademark!" and misspell the label and hope their entire operation isn't exposed?
- bellyfullofbac 5y ago
  Sometimes bandits are sloppy or have Dunning-Kruger, e.g. with pirated DVDs the sleeve art is just a hot mess of random words, but I guess if they were making fake chips they'd be a bit better at it. Or the guy they hired to do the Photoshop says he's real good but he's actually sloppy, and everyone else just sees random glyphs (imagine if you were having to copy an Arabic label, I assume you can't read Arabic).
- vgel 5y ago
  Counterfeit chips aren't just a state-level actor problem, companies do it for profit as well. I think the worry is just, if the person who was supposed to be checking for counterfeits missed this, how would they have any chance of catching a more sophisticated counterfeit?
- mike_d 5y ago
  What the original author isn't aware of is the fact genuine product stickers often will include minor differences (two transposed letters, a line of text being slightly off to one side, a copyright symbol with a misprint, etc) that are used to quickly visually verify the stickers. This works incredibly well because forgers try to get everything right, and often won't detect these subtle differences from the photos they are working with. The military does the same thing on badges for secure areas. They will have a misspelling of the base name or two shades of the same color on the front/back so guards can visually verify them as legitimate.
KennyBlanken 5y ago
What I find funny: the guy who runs STH frothing at the mouth about a typo. To paraphrase Nick Fury: "Pot, kettle." The STH guy can't spell to save his life and his grammar is terrible. His sentence structure and general writing skills are about what I would expect from a fourth-grade child. He also is an incredible drama-llama, making mountains out of molehills; I've seen him do this time and time again. He sees a misprinted sticker and sees supply chain attacks? Dude. This gear is assembled by people in third world countries making wages that amount to a few dollars a day or less. They don't speak English. They may not even read roman letters. I'd challenge him to do QA on any non-roman alphabet... I bet someone did notice the stickers, but getting them reprinted (assuming it was caught before assembly started) may have meant a delay. Even a minor delay can be a major, major problem since this stuff is scheduled practically down to the hour in the factories; ditto for shipping deadlines. Or if they were already on assembled boards (or worse, inside assembled equipment) the cost to replace the sticker would be astronomical, with exactly zero value to the vendor or their customers. It's cosmetic.
- lsaferite 5y ago
  Having worked for a hardware company before, they were VERY particular about their hardware quality. They would frequently get engineering samples and do full teardowns on them to evaluate build quality. Those engineers were intimately familiar with the hardware. And these were toys. I would absolutely expect that high-end computer equipment goes through frequent engineering sample review. If they aren't doing that, then how are they ensuring the quality of the company doing the manufacturing?
  - wokkel 5y ago
    By counting the number of returned items and service calls: that's way cheaper than upfront checking and in the short run will suffice. That's MBA logic for you.
- NobodyNada 5y ago
  > The STH guy can't spell to save his life and his grammar is terrible. His sentence structure and general writing skills are about what I would expect from a fourth-grade child. I mean, he acknowledges this: > I make a lot of typos, so I know how it goes. As someone who is dyslexic, this piece is going to have a few typos when it goes live, so I empathize with this. At the same time, I tend to spell my name correctly on official documents. In the electronics industry, a typo on a label, and especially in the brand name, is a very strong indicator that the product is counterfeit (AFAIK such mispellings are often deliberate to avoid using a trademark). For instance, fake Apple products are often "Designed by Apole in Colifornia". And while it's true that the products are most likely assembled by non-native English speakers, this sticker appears to include some licensing/serial number information and was probably designed by someone at corporate headquarters for the express purpose of certifying that the chip is genuine. And the author places the bulk of the responsibility on Dell and AMI for not catching the error, rather than the workers at Foxconn or wherever. Counterfeit parts, supply chain issues, and cut corners are common, so engineers in the US closely inspect a random sample of boards received from the factories to make sure everything was built according to spec, which is where this should have been caught.
kristopolous 5y ago
What big company produces a chip, puts their name on it but leaves off their logo and trademark and also uses the wrong typeface? There's something wrong here. Do we have a second instance?
Spooky23 5y ago
HPE specifically emphasizes supply chain security in this type of product. I would consider this a big deal.
- wmf 5y ago
  The new "secure" iLO 5 is full of vulns. https://www.blackhat.com/us-21/briefings/schedule/index.html#hpe-ilo-firmware-security---go-home-cryptoprocessor-youre-drunk-23398 https://www.blackhat.com/us-21/briefings/schedule/index.html...
vmception 5y ago
Thank you! Christ wish they would have gotten to the point right up front, my disbelief when I saw this blog had actual pages for this article, wow had not seen that in a while
deleted 5y ago
[deleted]
sabalaba 5y ago
I can tell you that manufacturing businesses are difficult to manage, have thousands of moving parts if not millions, those parts are more often than not human. It's like a choreographed dance with millions of people simultaneously, somebody is bound to trip over their laces.
gigatexal 5y ago
Taking the other side of this: what if this is really a man-in-the-middle attack of sorts on the supply chain? These kinds of typos we are in phishing emails why not here? And it could very well be that AMI is doing damage control by just saying “it’s a typo” — like they can’t spell their own name?

walrus01 5y ago

> But author is concerned about what this means for the state of supply chain verification. This is a brown M&Ms problem: https://conversableeconomist.blogspot.com/2020/10/the-no-brown-m-story-contract-theory-at.html https://conversableeconomist.blogspot.com/2020/10/the-no-bro... If they missed something as obvious as this, who knows what other problems are going on in supply chain security or total lack of QA.

NAR8789 5y ago
I'd argue it's slightly different-- - no brown m&m's specifically calls out no brown m&ms in a list of requirements, and uses it as a canary for reading comprehension. - a misspelling is an "obvious" problem, but I suspect not called out anywhere as a specific requirement. "No brown m&ms" catches when people aren't paying detailed attention to your (presumably reasonably scoped) requirements doc. Asking people to catch all "obvious" problems holds them accountable to an unbounded guessing game, and you're far more likely to catch people out, simply because of differences in where they choose to focus.
- walrus01 5y ago
  The misspelling means two different things to two categories of people... Those who haven't worked in electronics manufacturing are highly likely to say "oh it's just an honest mistake". Those who have worked in electronics manufacturing will immediately see it as a possibly scary sign of counterfeit components making their way into the supply chain. Same as what happened with counterfeit capacitors in east asia. Much like the early 1980s Van Halen tour example linked above, it's a reason for hitting the big red "OKAY, STOP EVERYTHING" button and re-check of all of the other components and supply chain going into the product.
  - NAR8789 5y ago
    Ahhh, that's the critical piece of context I was missing. Thank you! Article didn't fully explain why the sticker matters so much, so that left me scratching my head. (my gut reaction was "well, wouldn't a genuine sticker still be easy to counterfeit?). But based on your explanation this is more of a smell that everyone in the electronics manufacturing space is culturally attuned to. So, the fact that it slipped by so many people does indicate a slippage of norms. If I draw a comparison to software to bring it closer to something familiar to me... would this be like inconsistent variable name formatting? CamelCase in some places and snake_case in others? To an outsider, arguably inconsequential, so insisting on consistency here might seem OCD to them, but to someone who's worked in the space it's actually a useful marker of general detail orientation.
    - walrus01 5y ago
      In software it would be like if you found evidence that one of your trusted developers had outsourced their own job to an unknown person in India, who was controlling their home office desktop PC by some sort of remote desktop tool and adding code/making commits on their behalf.
    - bopbeepboop 5y ago
      This would be like if one day you found “MicronSoft Word” installed on your computer, when you went to write a document. Or your compiler started saying it was “Jawa 9”. Maybe someone just typo’d the name in an update.
    - 3np 5y ago
      > would this be like inconsistent variable name formatting? CamelCase in some places and snake_case in others? I’d say it’s more like misspelling your own company name in the JPEG logo in the HTML welcome email sent out to new users, and no one notices for a year.
    - outworlder 5y ago
      > If I draw a comparison to software to bring it closer to something familiar to me... would this be like inconsistent variable name formatting This would be more like finding a container coming from dockahub.com and then ultimately finding out that it was a legitimate typo.
- hef19898 5y ago
  Also focusing on unimportant, but obvious, issues takes resources away from controlling for not so obvious but critical issues. That and given the fact that checking English spelling isn't a trivial thing for Asians (as it would be the other way round) means this is a non-issue. Will QC be improved after that? Potentially. Will there be more scrutiny around that particular supplier? I would think so. Is it a general supply chain problem? No.
formerly_proven 5y ago
I'm not so sure. These BMCs are dumb SoCs like any other (just with specialized I/O), their firmware comes from an external SPI flash. I doubt there is anything AMI specific at all in these chips. Looks like it's basically a license sticker some worker is putting on these boards after they're all assembled and tested. I can see how these stickers are there, maybe for contractual/legal reasons, without being a critical part of the BMC board BOM (1x roll of "AMI loicense stickers").
- xondono 5y ago
  These “dumb SoCs” are one of the biggest security holes in a lot of high end equipment. It would not be the first time someone finds exploitable firmware bugs and vulnerable BMCs through Shodan.
stronglikedan 5y ago
I work in an industry rife with typos, and can assure you that typos are missed all the time, and it doesn't indicate a greater QA problem or lack of QA. It's just a missed typo.
- mistrial9 5y ago
  in other contexts this is referred to as "alarm fatigue" .. there is no assurance and masking it is not the move here
- treesknees 5y ago
  Where I work, we treat grammar and spelling errors as a sev 2 bug (sev 1 being highest.) It will quickly erode trust in your product even if everything else is working fine. All of our customer-facing text (UI/CLI/API names, labels, error messages, etc.) go through QA and also a separate documentation team for consistency and spelling checks.
  - Syonyk 5y ago
    Lul your just overeacting! </s>
  - cesaref 5y ago
    Here in the UK, one of the national newspapers, The Guardian was so famous for typing mistakes that it was nicknamed 'The Grauniad' by Private Eye, which is a satirical magazine. It's actually so well know that http://www.grauniad.co.uk http://www.grauniad.co.uk redirects to the correct sight
- JohnJamesRambo 5y ago
  Idk man this isn’t just a typo it is the company name. Would Nike misspell their name on a product?
  - robbedpeter 5y ago
    Nuke sneakers would be kinda cool actually. Maybe Nake - proud sponsors of this year's Olympics streakers!
ok_dad 5y ago
I used to work in incoming inspection department for a medical device manufacturer. If the switch/router company did a "first article inspection" (FAI) and missed this (or missed it on a later incoming inspection), I would agree, but if they found it and called AMI and got the answer it was a typo, then they would note that on the inspection sheet and move on.
- HeyLaughingBoy 5y ago
  You weren't responsible for my comment https://news.ycombinator.com/item?id=29559534 https://news.ycombinator.com/item?id=29559534 above, were you ;-)
  - ok_dad 5y ago
    We were super anal about things there, but I always made sure scrap was actually scrap and not recoverable. Your description sounds like a stupid situation and they should have just reinspected those blades.
- fckgw 5y ago
  The problem is, as pointed out in the example, is that the memo the writer got from Dell that came from AMI informing Dell about the typo contained an image that the own writer sent to AMI. Meaning AMI had no idea about this typo before they were contacted by the writer.
aaron695 5y ago
> This is a brown M&Ms problem No it's not. Brown M&Ms gives you one thing to check. This misspelling is part of 1000's of things that could be checked. A misspelling can indicate counterfeits, a serious issue on critical infrastructure, unlike the Brown M&M's which is just a joke check. Yes it's weird someone illegally counterfeiting will misspell, but it happens for complex reasons [1] [1] A recent example - "Fake Vaccination Cards Seized At O’Hare Were Spotted Because They Had Misspelled Words" - https://blockclubchicago.org/2021/09/07/fake-vaccination-cards-seized-at-ohare-because-they-had-misspelled-words/ https://blockclubchicago.org/2021/09/07/fake-vaccination-car...
PaulHoule 5y ago
It's more of a conspicuous display of incompetence. Some politicians make a point of displaying this so that donors know they aren't going to be somebody smart, ambitious and dangerous like Ralph Nader. One example was Karen Bass, who bent over backwards to praise Scientology once.
wyldfire 5y ago
Maybe they're aware of the supply chain interdiction and willing or motivated to let it continue. You'd think that the integrity of their product would be paramount concern. But how might their attitude change if a state decided to intervene? Or what if they have short-sighted leadership making the best decisions for them individually?
PragmaticPulp 5y ago
More likely that nobody noticed the typo until it was too late and they didn’t want to halt an entire production run just to replace a sticker that most people will never see. Stopping the production run and delaying shipments would have a palpable impact on shipment numbers. Most likely it was noticed, corrected for later batches, and they made the call to run this round anyway because it had zero effect on function. (Source: If you work in hardware long enough, you’ll have to make similar calls like this eventually)
- hef19898 5y ago
  For us older folks remembering what "stamps" are, and that people used to collect those: The most valuable ones are those with printing errors, e.g. typos. If you produce enough stuff at scale mistakes happen. And I would assume the stickers are much less controlled and monitored then the chips / hardware itself. No idea how it works in electronics, in aerospace so you would do thorough testing on the affected units. And if they meet the spec, someone would make a call regarding relabeling. Considering how expensive relabeling tends to be, it isn't unreasonable to assume the parts just got a concession for the labels. I wouldn't read to much into it.
tomxor 5y ago
> If they missed something as obvious as this, who knows what other problems are going on in supply chain security or total lack of QA. I'm pretty sure "they" that design the stickers are very different from "they" that design the chip... Similarly chip verification has rather a different skill set and attention to detail than someone doing "sticker QA". Also I doubt anyone was doing sticker QA, I'd guess that this was probably an outsourced mistake (I used to work in printing when I was younger - the mistakes you wouldn't believe).

1970-01-01 5y ago

Yes. TL;DR: Chip does not meet MILSPEC or any other spec. People are OK with it. The chips really should be de-capped and researched.

ricardobeat 5y ago

They said there is no tampering. Someone should x-ray this chip vs a correctly labeled one to verify.

wmf 5y ago
AMI is a firmware company so you'd check the hash of the BMC flash contents.
- xondono 5y ago
  Although it would be much (much) more elaborate, a counterfeit BMC could have hidden ROMs or bootloaders capable of opening backdoors into running machines.
  - wmf 5y ago
    Note that the BMC chip comes from a different company (ASPEED) who has not been implicated in this "stickergate".
    - bruce343434 5y ago
      perhaps off-topic, but where does the -gate suffix come from? It seems it can be replaced by "troubles". Gamer troubles. Sticker troubles. But what does -gate even mean?
      - freeman478 5y ago
        I think it comes from https://en.wikipedia.org/wiki/Watergate_scandal https://en.wikipedia.org/wiki/Watergate_scandal
        handrous 5y ago
        Ah, good old Watergate-gate.
      - CBLT 5y ago

foobiekr 5y ago

This is laughably sloppy. One of my favorite customers actually took a calorimeter to the LEDs that were in one of our deliveries of router chassis. They felt that the Amber wasn’t quite Amber enough and so they measured it. They were right while it was amber it wasn’t spec amber. So they shipped it all back. We were very embarrassed and supply chain was given a dressing down. They missed a parts quality issue. A typo like this is extremely indicative of a sloppy organization.

detaro 5y ago
What is the relevant spec for colors here? Is there some ISO or whatever for signal lights on equipment defining it?
- fxtentacle 5y ago
  Parent comment probably meant "colorimeter" which will measure colors in either spectral distribution or XYZ color space. So you could say something like RAL color # 123 +- 5% in XYZ space. Or you can just specify: That color needs to be closer to RAL #123 than to any other RAL number, so then you'd also have an implicit definition of the valid color range.
  - detaro 5y ago
    What the "something like" is exactly and where it came from is kind of my question.
- jwandborg 5y ago
  It might be regulatory concerns, maybe they hadn't licensed that particular wavelength of orange for local broadcasting purposes, or EMI concerns with regards to the the off-white spectrum, at least if I let my thoughts run without moderation.
  - foobiekr 5y ago
    No, this really was just ethernet interface front panel LEDs. The customer viewed this as indicative of poor quality control.

deleted 5y ago

[deleted]

howdydoo 5y ago

> supply chain tampering Naive question, I don't know much about this industry. But if someone from China or the NSA wanted to make counterfeit chips, why would they risk putting something different on the label? It seems like a weird place to draw the line. Would the NSA really say "oh no, we can't violate anyone's trademark!" and misspell the label and hope their entire operation isn't exposed?

bellyfullofbac 5y ago
Sometimes bandits are sloppy or have Dunning-Kruger, e.g. with pirated DVDs the sleeve art is just a hot mess of random words, but I guess if they were making fake chips they'd be a bit better at it. Or the guy they hired to do the Photoshop says he's real good but he's actually sloppy, and everyone else just sees random glyphs (imagine if you were having to copy an Arabic label, I assume you can't read Arabic).
vgel 5y ago
Counterfeit chips aren't just a state-level actor problem, companies do it for profit as well. I think the worry is just, if the person who was supposed to be checking for counterfeits missed this, how would they have any chance of catching a more sophisticated counterfeit?
mike_d 5y ago
What the original author isn't aware of is the fact genuine product stickers often will include minor differences (two transposed letters, a line of text being slightly off to one side, a copyright symbol with a misprint, etc) that are used to quickly visually verify the stickers. This works incredibly well because forgers try to get everything right, and often won't detect these subtle differences from the photos they are working with. The military does the same thing on badges for secure areas. They will have a misspelling of the base name or two shades of the same color on the front/back so guards can visually verify them as legitimate.

KennyBlanken 5y ago

What I find funny: the guy who runs STH frothing at the mouth about a typo. To paraphrase Nick Fury: "Pot, kettle." The STH guy can't spell to save his life and his grammar is terrible. His sentence structure and general writing skills are about what I would expect from a fourth-grade child. He also is an incredible drama-llama, making mountains out of molehills; I've seen him do this time and time again. He sees a misprinted sticker and sees supply chain attacks? Dude. This gear is assembled by people in third world countries making wages that amount to a few dollars a day or less. They don't speak English. They may not even read roman letters. I'd challenge him to do QA on any non-roman alphabet... I bet someone did notice the stickers, but getting them reprinted (assuming it was caught before assembly started) may have meant a delay. Even a minor delay can be a major, major problem since this stuff is scheduled practically down to the hour in the factories; ditto for shipping deadlines. Or if they were already on assembled boards (or worse, inside assembled equipment) the cost to replace the sticker would be astronomical, with exactly zero value to the vendor or their customers. It's cosmetic.

lsaferite 5y ago
Having worked for a hardware company before, they were VERY particular about their hardware quality. They would frequently get engineering samples and do full teardowns on them to evaluate build quality. Those engineers were intimately familiar with the hardware. And these were toys. I would absolutely expect that high-end computer equipment goes through frequent engineering sample review. If they aren't doing that, then how are they ensuring the quality of the company doing the manufacturing?
- wokkel 5y ago
  By counting the number of returned items and service calls: that's way cheaper than upfront checking and in the short run will suffice. That's MBA logic for you.
NobodyNada 5y ago
> The STH guy can't spell to save his life and his grammar is terrible. His sentence structure and general writing skills are about what I would expect from a fourth-grade child. I mean, he acknowledges this: > I make a lot of typos, so I know how it goes. As someone who is dyslexic, this piece is going to have a few typos when it goes live, so I empathize with this. At the same time, I tend to spell my name correctly on official documents. In the electronics industry, a typo on a label, and especially in the brand name, is a very strong indicator that the product is counterfeit (AFAIK such mispellings are often deliberate to avoid using a trademark). For instance, fake Apple products are often "Designed by Apole in Colifornia". And while it's true that the products are most likely assembled by non-native English speakers, this sticker appears to include some licensing/serial number information and was probably designed by someone at corporate headquarters for the express purpose of certifying that the chip is genuine. And the author places the bulk of the responsibility on Dell and AMI for not catching the error, rather than the workers at Foxconn or wherever. Counterfeit parts, supply chain issues, and cut corners are common, so engineers in the US closely inspect a random sample of boards received from the factories to make sure everything was built according to spec, which is where this should have been caught.

kristopolous 5y ago

What big company produces a chip, puts their name on it but leaves off their logo and trademark and also uses the wrong typeface? There's something wrong here. Do we have a second instance?

Spooky23 5y ago

HPE specifically emphasizes supply chain security in this type of product. I would consider this a big deal.

wmf 5y ago
The new "secure" iLO 5 is full of vulns. https://www.blackhat.com/us-21/briefings/schedule/index.html#hpe-ilo-firmware-security---go-home-cryptoprocessor-youre-drunk-23398 https://www.blackhat.com/us-21/briefings/schedule/index.html...

vmception 5y ago

Thank you! Christ wish they would have gotten to the point right up front, my disbelief when I saw this blog had actual pages for this article, wow had not seen that in a while

deleted 5y ago

[deleted]

sabalaba 5y ago

I can tell you that manufacturing businesses are difficult to manage, have thousands of moving parts if not millions, those parts are more often than not human. It's like a choreographed dance with millions of people simultaneously, somebody is bound to trip over their laces.

gigatexal 5y ago

Taking the other side of this: what if this is really a man-in-the-middle attack of sorts on the supply chain? These kinds of typos we are in phishing emails why not here? And it could very well be that AMI is doing damage control by just saying “it’s a typo” — like they can’t spell their own name?