6 ms·

I'd argue it's slightly different-- - no brown m&m's specifically calls out no brown m&ms in a list of requirements, and uses it as a canary for reading compre

I'd argue it's slightly different-- - no brown m&m's specifically calls out no brown m&ms in a list of requirements, and uses it as a canary for reading comprehension. - a misspelling is an "obvious" problem, but I suspect not called out anywhere as a specific requirement. "No brown m&ms" catches when people aren't paying detailed attention to your (presumably reasonably scoped) requirements doc. Asking people to catch all "obvious" problems holds them accountable to an unbounded guessing game, and you're far more likely to catch people out, simply because of differences in where they choose to focus.

walrus01 5y ago
The misspelling means two different things to two categories of people... Those who haven't worked in electronics manufacturing are highly likely to say "oh it's just an honest mistake". Those who have worked in electronics manufacturing will immediately see it as a possibly scary sign of counterfeit components making their way into the supply chain. Same as what happened with counterfeit capacitors in east asia. Much like the early 1980s Van Halen tour example linked above, it's a reason for hitting the big red "OKAY, STOP EVERYTHING" button and re-check of all of the other components and supply chain going into the product.
- NAR8789 5y ago
  Ahhh, that's the critical piece of context I was missing. Thank you! Article didn't fully explain why the sticker matters so much, so that left me scratching my head. (my gut reaction was "well, wouldn't a genuine sticker still be easy to counterfeit?). But based on your explanation this is more of a smell that everyone in the electronics manufacturing space is culturally attuned to. So, the fact that it slipped by so many people does indicate a slippage of norms. If I draw a comparison to software to bring it closer to something familiar to me... would this be like inconsistent variable name formatting? CamelCase in some places and snake_case in others? To an outsider, arguably inconsequential, so insisting on consistency here might seem OCD to them, but to someone who's worked in the space it's actually a useful marker of general detail orientation.
  - walrus01 5y ago
    In software it would be like if you found evidence that one of your trusted developers had outsourced their own job to an unknown person in India, who was controlling their home office desktop PC by some sort of remote desktop tool and adding code/making commits on their behalf.
  - bopbeepboop 5y ago
    This would be like if one day you found “MicronSoft Word” installed on your computer, when you went to write a document. Or your compiler started saying it was “Jawa 9”. Maybe someone just typo’d the name in an update.
  - 3np 5y ago
    > would this be like inconsistent variable name formatting? CamelCase in some places and snake_case in others? I’d say it’s more like misspelling your own company name in the JPEG logo in the HTML welcome email sent out to new users, and no one notices for a year.
  - outworlder 5y ago
    > If I draw a comparison to software to bring it closer to something familiar to me... would this be like inconsistent variable name formatting This would be more like finding a container coming from dockahub.com and then ultimately finding out that it was a legitimate typo.
- ATsch 5y ago
  People who do not work in hardware seriously underestimate how often counterfeit components enter the supply chain. Even trustworthy distributors like digikey and mouser have had regular cases of their supplies becoming contaminated. It is just way too easy to do and rarely discovered as long as you mix them with enough legitimate components to avoid suspicion. Unless you're unlucky and they happen to reach someone with an affinity for chip photography, the worst case is they'll just think a few components were DOA or out of spec. It mostly affects low complexity components that are easy to clone so a BMC would be unlikely, but even that is not safe as sometimes used components de-soldered from other products make it back into the supply chain too.
  - xondono 5y ago
    I’m still haunted by nightmares of fake FTDI chips
    - ATsch 5y ago
      That's a good public example, here's a post for anyone that missed it: https://zeptobars.com/en/read/FTDI-FT232RL-real-vs-fake-supereal https://zeptobars.com/en/read/FTDI-FT232RL-real-vs-fake-supe...
    - kragen 5y ago
      The bigger supply-chain problem there wasn't the fake FTDI chips, which actually worked reasonably well; it was the totally genuine and authorized FTDI driver update which FTDI designed to brick your customers' hardware if they installed it, if you had been so unfortunate as to get fake FTDI chips.
      - ComputerGuru 5y ago
        Or worse: if they mistakenly thought you had a non-genuine FTDI component when you really didn’t.
        kragen 5y ago
- btown 5y ago
  More specifically, if there is a person in a QA role looking out for counterfeit components, and they miss such a glaringly obvious typo, they may not be paying attention to enough details. And if a language barrier is at fault, that's itself a flag that they may not have full comprehension of the specifications they are supposed to QA for. Perhaps not "stop the line" but "we need to do an audit of this component" - and that audit needs to actually be performed.
- KennyBlanken 5y ago
  Anyone familiar with electronics manufacturing knows that for most of the people involved in the production of your stuff, English is a second language and roman characters are not their primary alphabet. > it's a reason for hitting the big red "OKAY, STOP EVERYTHING" button and re-check of all of the other components and supply chain going into the product. The notion that someone would pause a line over this (even if we were not in the middle of unprecedented component/manufacturing/shipping disruptions) is beyond fucking absurd, much less that anyone would do so until a "re-check of the supply chain" is completed. Production schedules are tight as hell. You miss your deadline for getting the board assembled, they don't make it to the line or factory putting the boards into the chassis on time. That means they've started on another job and now you wait until they have free time on the line. That means you don't get your container to the port on time. That means you miss the space you had paid for on the ship. That means you miss your product launch date. Possibly by months; especially right now, shipping is severely constrained. That means your competitor takes your lunch money.
  - walrus01 5y ago
    If your goal is to crank out the largest quantity of cheapest-unit-priced products as quickly as possible, then yes, absolutely stick with what you just wrote above. Are you familiar with what happened with the counterfeit capacitor plague? https://www.google.com/search?client=firefox-b-d&q=capacitor+plague https://www.google.com/search?client=firefox-b-d&q=capacitor... Keep on cranking out that production line with your suspicious/manufacturer-source-unknown parts with improper labeling on them and end up in a situation with hundreds of millions of dollars of financial damages due to burst capacitors. Not every manufacturer has its absolute and highest goal set as massive quantity/cheap and shoddy QA/lowest price/highest volume possible. For something like a 100Gb ethernet switch, standards should be much higher than the PCBs of a bunch of $40 802.11ac wifi routers to put in a shiny box and sell at Best Buy.
  - HeyLaughingBoy 5y ago
    I worked at a place where I had ready access to a large lot of $1,000 "blade" PC's that I could use for experiments, tests, etc., as needed. The reason I had them available is that they were supposed to go into a machine that we sold, but since someone in Receiving missed a step in their inspection process, they all had to be marked as Non-Conforming and could not be used in the machine, even though there was absolutely nothing wrong with them. Since they were a special order, the vendor wasn't about to take them back. So it was either toss tens of thousands of dollars worth of hardware in the dumpster or try to find a use for them internally.
- HeyLaughingBoy 5y ago
  I agree. The amount of scrutiny that goes into specifying, procuring, and then validating something as seemingly simple as a label would be eye-opening to someone who hasn't lived it. Not to mention that I spent a solid 40 hours last year writing code to print what was apparently a very simple, but automated, label. If I saw a misspelling like that it would indeed be cause for hitting the Big Red Button.
  - II2II 5y ago
    As someone from outside the industry, I started reading the article with the mindset that this was about counterfeit parts and was interested in the outcome. Yet the more I read the more I was asking myself what the article was actually about. Eventually I just stopped reading, looked at the comments, and learned that it was about a typo. Quite frankly, it doesn't matter how much process there is to avoid this sort of thing. It is a big industry and we should expect an error as trivial as a typo to slip through eventually. The reader would have been much better served if the author was upfront about that, discussed why the error was significant, and offered some insight into how the industry normally avoids this type of error. Maybe they did get to that point eventually, but this reader will never know since this reader gave up when the article felt self-serving rather than informative.
- roywashere 5y ago
  A company I worked for ordered thousands of set-top boxes. When we received them they were labeled as containing 'Marcovision' copy protection on the back, instead of Macrovision. Turned out to be a stupid typo
- BizarroLand 5y ago
  Especially an independent black box SOC type of component that has wormed its way into all manner of systems. There's no proof that tampering has happened but there is proof that not only it might have happened but also that there is no way to tell if it did happen short of, idk, forensically auditing the network traffic of your $10,000 switch with the help of another $10,000 switch and someone skilled enough to pick apart the output over a long enough period to determine if tampering happened. I'm not paranoid to do something like that but if the US Government isn't paranoid enough to do something like that then something has gone horribly away in government cybersecurity.
hef19898 5y ago
Also focusing on unimportant, but obvious, issues takes resources away from controlling for not so obvious but critical issues. That and given the fact that checking English spelling isn't a trivial thing for Asians (as it would be the other way round) means this is a non-issue. Will QC be improved after that? Potentially. Will there be more scrutiny around that particular supplier? I would think so. Is it a general supply chain problem? No.

walrus01 5y ago

The misspelling means two different things to two categories of people... Those who haven't worked in electronics manufacturing are highly likely to say "oh it's just an honest mistake". Those who have worked in electronics manufacturing will immediately see it as a possibly scary sign of counterfeit components making their way into the supply chain. Same as what happened with counterfeit capacitors in east asia. Much like the early 1980s Van Halen tour example linked above, it's a reason for hitting the big red "OKAY, STOP EVERYTHING" button and re-check of all of the other components and supply chain going into the product.

NAR8789 5y ago
Ahhh, that's the critical piece of context I was missing. Thank you! Article didn't fully explain why the sticker matters so much, so that left me scratching my head. (my gut reaction was "well, wouldn't a genuine sticker still be easy to counterfeit?). But based on your explanation this is more of a smell that everyone in the electronics manufacturing space is culturally attuned to. So, the fact that it slipped by so many people does indicate a slippage of norms. If I draw a comparison to software to bring it closer to something familiar to me... would this be like inconsistent variable name formatting? CamelCase in some places and snake_case in others? To an outsider, arguably inconsequential, so insisting on consistency here might seem OCD to them, but to someone who's worked in the space it's actually a useful marker of general detail orientation.
- walrus01 5y ago
  In software it would be like if you found evidence that one of your trusted developers had outsourced their own job to an unknown person in India, who was controlling their home office desktop PC by some sort of remote desktop tool and adding code/making commits on their behalf.
- bopbeepboop 5y ago
  This would be like if one day you found “MicronSoft Word” installed on your computer, when you went to write a document. Or your compiler started saying it was “Jawa 9”. Maybe someone just typo’d the name in an update.
- 3np 5y ago
  > would this be like inconsistent variable name formatting? CamelCase in some places and snake_case in others? I’d say it’s more like misspelling your own company name in the JPEG logo in the HTML welcome email sent out to new users, and no one notices for a year.
- outworlder 5y ago
  > If I draw a comparison to software to bring it closer to something familiar to me... would this be like inconsistent variable name formatting This would be more like finding a container coming from dockahub.com and then ultimately finding out that it was a legitimate typo.
ATsch 5y ago
People who do not work in hardware seriously underestimate how often counterfeit components enter the supply chain. Even trustworthy distributors like digikey and mouser have had regular cases of their supplies becoming contaminated. It is just way too easy to do and rarely discovered as long as you mix them with enough legitimate components to avoid suspicion. Unless you're unlucky and they happen to reach someone with an affinity for chip photography, the worst case is they'll just think a few components were DOA or out of spec. It mostly affects low complexity components that are easy to clone so a BMC would be unlikely, but even that is not safe as sometimes used components de-soldered from other products make it back into the supply chain too.
- xondono 5y ago
  I’m still haunted by nightmares of fake FTDI chips
  - ATsch 5y ago
    That's a good public example, here's a post for anyone that missed it: https://zeptobars.com/en/read/FTDI-FT232RL-real-vs-fake-supereal https://zeptobars.com/en/read/FTDI-FT232RL-real-vs-fake-supe...
  - kragen 5y ago
    The bigger supply-chain problem there wasn't the fake FTDI chips, which actually worked reasonably well; it was the totally genuine and authorized FTDI driver update which FTDI designed to brick your customers' hardware if they installed it, if you had been so unfortunate as to get fake FTDI chips.
    - ComputerGuru 5y ago
      Or worse: if they mistakenly thought you had a non-genuine FTDI component when you really didn’t.
      - kragen 5y ago
btown 5y ago
More specifically, if there is a person in a QA role looking out for counterfeit components, and they miss such a glaringly obvious typo, they may not be paying attention to enough details. And if a language barrier is at fault, that's itself a flag that they may not have full comprehension of the specifications they are supposed to QA for. Perhaps not "stop the line" but "we need to do an audit of this component" - and that audit needs to actually be performed.
KennyBlanken 5y ago
Anyone familiar with electronics manufacturing knows that for most of the people involved in the production of your stuff, English is a second language and roman characters are not their primary alphabet. > it's a reason for hitting the big red "OKAY, STOP EVERYTHING" button and re-check of all of the other components and supply chain going into the product. The notion that someone would pause a line over this (even if we were not in the middle of unprecedented component/manufacturing/shipping disruptions) is beyond fucking absurd, much less that anyone would do so until a "re-check of the supply chain" is completed. Production schedules are tight as hell. You miss your deadline for getting the board assembled, they don't make it to the line or factory putting the boards into the chassis on time. That means they've started on another job and now you wait until they have free time on the line. That means you don't get your container to the port on time. That means you miss the space you had paid for on the ship. That means you miss your product launch date. Possibly by months; especially right now, shipping is severely constrained. That means your competitor takes your lunch money.
- walrus01 5y ago
  If your goal is to crank out the largest quantity of cheapest-unit-priced products as quickly as possible, then yes, absolutely stick with what you just wrote above. Are you familiar with what happened with the counterfeit capacitor plague? https://www.google.com/search?client=firefox-b-d&q=capacitor+plague https://www.google.com/search?client=firefox-b-d&q=capacitor... Keep on cranking out that production line with your suspicious/manufacturer-source-unknown parts with improper labeling on them and end up in a situation with hundreds of millions of dollars of financial damages due to burst capacitors. Not every manufacturer has its absolute and highest goal set as massive quantity/cheap and shoddy QA/lowest price/highest volume possible. For something like a 100Gb ethernet switch, standards should be much higher than the PCBs of a bunch of $40 802.11ac wifi routers to put in a shiny box and sell at Best Buy.
- HeyLaughingBoy 5y ago
  I worked at a place where I had ready access to a large lot of $1,000 "blade" PC's that I could use for experiments, tests, etc., as needed. The reason I had them available is that they were supposed to go into a machine that we sold, but since someone in Receiving missed a step in their inspection process, they all had to be marked as Non-Conforming and could not be used in the machine, even though there was absolutely nothing wrong with them. Since they were a special order, the vendor wasn't about to take them back. So it was either toss tens of thousands of dollars worth of hardware in the dumpster or try to find a use for them internally.
HeyLaughingBoy 5y ago
I agree. The amount of scrutiny that goes into specifying, procuring, and then validating something as seemingly simple as a label would be eye-opening to someone who hasn't lived it. Not to mention that I spent a solid 40 hours last year writing code to print what was apparently a very simple, but automated, label. If I saw a misspelling like that it would indeed be cause for hitting the Big Red Button.
- II2II 5y ago
  As someone from outside the industry, I started reading the article with the mindset that this was about counterfeit parts and was interested in the outcome. Yet the more I read the more I was asking myself what the article was actually about. Eventually I just stopped reading, looked at the comments, and learned that it was about a typo. Quite frankly, it doesn't matter how much process there is to avoid this sort of thing. It is a big industry and we should expect an error as trivial as a typo to slip through eventually. The reader would have been much better served if the author was upfront about that, discussed why the error was significant, and offered some insight into how the industry normally avoids this type of error. Maybe they did get to that point eventually, but this reader will never know since this reader gave up when the article felt self-serving rather than informative.
roywashere 5y ago
A company I worked for ordered thousands of set-top boxes. When we received them they were labeled as containing 'Marcovision' copy protection on the back, instead of Macrovision. Turned out to be a stupid typo
BizarroLand 5y ago
Especially an independent black box SOC type of component that has wormed its way into all manner of systems. There's no proof that tampering has happened but there is proof that not only it might have happened but also that there is no way to tell if it did happen short of, idk, forensically auditing the network traffic of your $10,000 switch with the help of another $10,000 switch and someone skilled enough to pick apart the output over a long enough period to determine if tampering happened. I'm not paranoid to do something like that but if the US Government isn't paranoid enough to do something like that then something has gone horribly away in government cybersecurity.

hef19898 5y ago

Also focusing on unimportant, but obvious, issues takes resources away from controlling for not so obvious but critical issues. That and given the fact that checking English spelling isn't a trivial thing for Asians (as it would be the other way round) means this is a non-issue. Will QC be improved after that? Potentially. Will there be more scrutiny around that particular supplier? I would think so. Is it a general supply chain problem? No.