12 ms·
Wait a minute... So your company has a Chinese equipment supplier, finds out that the supplier is tampering with your purchased equipment, and your solution is
by zelon88 8y ago
Wait a minute... So your company has a Chinese equipment supplier, finds out that the supplier is tampering with your purchased equipment, and your solution is to add criteria to the incoming inspection?
No wonder China keeps screwing with you guys. You aren't supposed to eat that cost! Write a PO with tons of fine print that says "We will disassembly units at random for compliance inspection. Non compliant products will be returned at the suppliers expense." And then add a clause that says ">3 non-compliance events in under # months will result in the entire PO (10 or 20 units) being returned and all contracts cancelled."
I cannot believe you are getting screwed by a company you choose to do business with and yet you eat cost to ensure they aren't screwing you. Just get a new supplier! Do on-site inspections at their facility. This is nuts.
- lmilcin 8y agoThere were other considerations like the fact we were actually buing it from large reputable company and what happened was that some employees were doing it with no involvement of the company. The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty (assuming the company was involved which they could not have been as they have been the ones damaged the most). If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs. The way to work is then verify everything and politely point it out. If you notice they will correct apparent mistake.
- SilasX 8y ago>There were other considerations like the fact we were actually buying it from large reputable company and what happened was that some employees were doing it with no involvement of the company. It's their responsibility to police for that. From the perspective of their business arrangement with you, it doesn't matter whether their left hand or right hand is evil; it's not your problem either way. Is it really hard to find alternatives? Are they really cheaper than non-Chinese sources when you account for these inspection costs?
- s73v3r_ 8y ago"There were other considerations like the fact we were actually buing it from large reputable company and what happened was that some employees were doing it with no involvement of the company." I fail to see how that has anything to do with it. The company is responsible for the product, full stop. If they can't stop their employees from tampering with the product, that's entirely their fault.
- ehsankia 8y ago> some employees were doing it with no involvement of the company Again, I don't see why that matters. I understand the latter part, but if this is one employee without the knowledge of the company, then pushing it on them forces them to add checks, not you.
- TheGRS 8y agoAlso it comes across a little naive. Oh they told us it was just an employee and would never happen again, OK!
- hinkley 8y agoThis still sounds nuts to me. Two thoughts: > they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs. This is how children behave. Still feels like you’re rewarding bad behavior. You’re enabling them. I think it’s more that you’ve valued the low per unit price over having a healthy contract. Your vendor is incentivized to make all of their money off of externalities and your company think it’s cheaper to outwit them than to demand QC on their end. Whose brand will be sullied if you miss some of these units and a customer finds the spyware? Not theirs. And second thought, shouldn’t serial numbers be coming off the line in ascending order? The kind of work they are doing would require taking parts off the line and putting them back later so odd lots of SNs are the ones you need to verify. You could also be mandating how many boards are allowed or that the SNs go on early in the build process.
- pickleman 8y agoYou're starting to realize that not all cultures are that great. >I think it’s more that you’ve valued the low per unit price over having a healthy contract. Well, when the choice is to either play ball or go out of business, it's not a tough decision.
- phendrenad2 8y agoAll employee actions are company actions. You partnered with a company that can’t control what it’s employees do? No internal audits to make sure their reputation wasn’t being tarnished by a few employees?! Your loss.
- TomVDB 8y agoI understand the indignation etc etc. And the suggestion to not use these kind of companies anymore. And that sounds really reasonable, until you realize that pretty much all contract manufacturers in the Far East will source cheaper or off-spec components than those on the BOM if they can get away with it. One of my friends supplied small widgets for a well known consumer electronics maker. He routinely gets widgets returned to him as defect for inspection, which then inevitably turn out to be clones of his widgets. The only way to make sure that your product rolls of the product line as expected, is to have people on-site with continuous inspection (and pray that they're not the cousin of somebody who's on the other side.) If you want the benefit of dirt cheap manufacturing, you need to have a system in place to deal with these practices.
- phendrenad2 8y agoHow much do you think you would have to pay them to make it worth their while to not use cheap components (so as not to risk losing you as a customer)? It’s amazing how people never consider using economics. Edit: But let’s not get away from the matter at hand. The issue wasn’t cheap components, it was full-on credit skimmers installed in yor hardware (IIRC). Should be easier to incentivize the conpany into halting those before they get shipped to you, no?
- TomVDB 8y agoAs somebody else noted: it doesn't have to be the company that does it, but a lower level employee who cuts a deal. So paying the company more wouldn't solve the issue completely. All of these things are extremely cost sensitive. Your suggestion that people don't consider using economics is simply wrong. If you can manufacture a million pieces for a few cent less per piece, and the only negative is having to paying a bunch of inspectors who are fixed cost, it's an easy choice.
- gcbw2 8y agothe indignation was not that. you are still missing the point. after the 1st device found, you should have contacted the manufacturer and said that you will start a department that has the capability of opening the device, inspecting and re-sealing in a way that it won't impact any guarantee the factory provides. If they denied this very sensible request, you had proof that it wasn't a isolated employee doing the hack.
- nray 8y agoNot really familliar with PCI DSS but it might be that the card-readers/terminals aren't PCI-compliant if opened? So not the manufacturer's issue but the customer's.
- 394549 8y ago> Not really familliar with PCI DSS but it might be that the card-readers/terminals aren't PCI-compliant if opened? So not the manufacturer's issue but the customer's. I think that's the case. The EEV Blog guy did a teardown of and old one once and pointed out the numerous tamper-detection features that would clear the device if opened. However, if I were the customer here, I'd tell the supplier that from that point forward they need to supply me free extra product with my orders, so I can do my own random destructive testing to look for implants. I order 100, they send me 105 for the price of 100.
- late2part 8y agoYou all recognize the irony, right?
- neuralzen 8y agoPCI DSS allows for "Mitigating Controls" if you need to deviate from specified requirements, provided it is well documented and is equal to or greater in security. Doing teardowns to review circumspect hardware, and applying one's own tamper protection deal (and with accompanying documentation and tracking/logged information) would very likely be sufficient to maintain complaince.
- PurpleBoxDragon 8y ago>The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty Sounds like the solution is not doing business with them and pushing for a ban on others doing business with them (since this largely has a socialized cost when things go wrong, such as individual people having their credit cards stolen).
- whydoineedthis 8y agoOMG, that is so so so so brilliant! I'm going to go tell my boss right now that the suppliers better suite up, otherwise we are going right to their more expensive, less experienced vendor and will delay our product launch for a year....and likely still suffer the same problem. Those Chinese vendors better shape up or we're going to really teach them a lesson by driving ourself out of business right quick!
- PurpleBoxDragon 8y agoJust imagine for a second if someone was advocating on HN for the use of a web framework that has knowingly allowed itself to be compromised to steal passwords because it was cheaper to use. How about a compromise? If any of your customers are a victim of a crime because you continued to use a shady but cheap vendor after seeing them trying to slip past tampered hardware, your company is held fiscally and criminally responsible.
- orochi235 8y ago
- deleted 8y ago[deleted]
- dirtfree22 8y agoI am reading it as "we don't any any option outside China and we have to live with this". I guess even after adding cost of doing additional of products you buy from China its less than what you would pay to buy it from other place. Thinking about it more, what you are doing make sense also because tampering can happen with any supplier, irrespective of factory location, given how aggressively states are pushing for spying on each other and their people.
- pokemongoaway 8y agoHere you have it... Moral relativism should get you paid less as an employee. Regardless, the modern craze of relativism should still require evidence.
- bluetwo 8y agoSounds like a libertarian oasis.
- neuralRiot 8y ago>If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs. They don't pass the savings to you so they know very well what they're doing, just playing fool to see if it slips.
- BurningFrog 8y agoThis is also such a good anonymous story that I'd give decent odds it's made up.
- hyperpape 8y agoHe's not anonymous, though.
- repolfx 8y agoI've heard very similar stories from people in the card industry before. I don't think it's made up, unfortunately. Card readers from China are systematically untrustworthy.
- retrogradeorbit 8y agoUsing the technically incorrect term "angular momentum" rather than the more correct term "moment of inertia" made me think the same thing. I would think a person tasked with building such a device would know their physics well enough to use the right term, but I may be wrong.
- turtlecloud 8y agoMaybe he is afraid of giving too much info away. By mentioning it, now the opposing side will be measuring angular momentum. So now he needs to make a new test. Ow that that is out of the bag
- polynomial 8y agoThe fact that they didn't just run the units through an xray machine would seem to support your theory.
- blazespin 8y agoProbably the cost per unit is too attractive
- az123zaz 8y agoAlso it forces their customer's customer to essentially subsidize production cost. Some amount of financial theft happens to random consumers who don't know where it came from and this stolen money is paid in bribes to employees as a subsidy to their low wage. As a result this immoral company externalizes aspects of their cost structure to greater society.
- liveoneggs 8y agohe's saying that his company knowingly allows a percentage of fraud, identity, and financial theft to occur against his customers' customers because there are enough layers of indirection where they (he and his bosses) probably can't be held personally responsible. Enough efforts (fancy tables with big receipts, etc) have been provided for coverage in case of a court case or media blitz but actually using a reputable supplier is clearly too expensive. I can't believe the parent comment is so brazen. It makes me physically sick.
- candiodari 8y agoI guess it's news to you that the financial sector and especially banks in fact do exactly that: https://www.creditcards.com/credit-card-news/credit-card-security-id-theft-fraud-statistics-1276.php https://www.creditcards.com/credit-card-news/credit-card-sec... But it doesn't stop with the cards. ATMs, Tax Fraud, ... almost all businesses within banks have to deal with fraud and there is some threshold that they just allow to happen.
- liveoneggs 8y agoknowing about it and willfully facilitating it are two different things
- mattkerle 8y agolike shoplifting. Most large chains know exactly how much "loss" they experience, and they just raise their prices to cover it. Theft and fraud is a part of life unfortunately...
- konschubert 8y agoMaybe you and me are missing something here. It seems crazy that somebody would go to these lengths without switching suppliers, there must be an underlying reason that is assumed to be understood by the informed reader. (But isn't understood by me and you it seems.)
- JoblessWonder 8y agoProfit. Changing suppliers is going to cost money.
- TeMPOraL 8y agoOr: you can expect this kind of shit from pretty much any supplier. Even if you go to a western company, they all source their components from China anyway.
- westiseast 8y agoA good read is 'Poorly Made in China'. The product is different, but the problems are the same. * They drop changes and problems at the last minute, so you're over a barrel with your customers. * Relationships take months, maybe years to build. Switching suppliers is a long and costly exercise. * Often suppliers themselves are in communication, so your attempt to build a new relationship is scuppered by your current (corrupted) supplier before you've got far. * The new supplier will 100% play the same games. * Suppliers/factories are often expert at these games - they're not idiots, their bread and butter is manipulating managers and big companies into corners and fleecing them along the way. * Large suppliers often have strong local/regional government connections. Once you start causing serious problems, they'll have no trouble causing massive, massive problems for your business. * Are you going to admit to your customers and bosses that your products were faulty and you knew? It's a poisoned chalice. Companies want low cost manufacturing, and sometimes China is the only place a product can be manufactured competitively. But this kind of behavior is standard for Chinese companies.
- ticviking 8y ago> * Are you going to admit to your customers and bosses that your products were faulty and you knew? If you can't answer that with a yes, maybe you don't have the backbone to work in anything critical. When you discover a fault in something, particularly a fault that might hurt someone, you have a moral obligation to speak up. To do otherwise is cowardice. Failure to speak up when we see shit is how stuff like the VW emissions gate happen, and also why security professionals can make a career out of ferreting out your mistakes and engaging in responsible disclosure. All this tells me is that Made in the USA is more valuable than I once believed.
- calimac 8y ago@zelon88 - You are a true business mind. Silicon Valley is full of pussies and you are the exception.
- blihp 8y agoThis problem is not specific to China, it's a problem of going with the lowest priced supplier. Your proposed recourse presumes that the supplier is making enough money on the deal to engage in a contract with teeth... many are not. Also, even if they moved everything to the U.S. they'd likely have different flavors of the same types of issues (i.e. did the NSA embed someone in the company to implant backdoors etc.) Fraud, sabotage and espionage aren't confined to specific countries. When greed drives you to move operations to the absolute cheapest places you can find on Earth, that's not going to be without its own set of risks and problems.
- imhoguy 8y agoHonestly, to anyone bashing GP, look into your pockets, laptops, watches, cars, TVs, routers, CCs, singing toys, bitcoin mining ASICs. All with sealed black-box chips. Sure we can x-ray a couple of randomly stripped chips, but each one? Now show me alternatives when most consumer grade electronic parts are fully or partially made in China.