5 ms·

"Intel Packet of Death" not Intel's problem

78 points by Maci 14y ago

Maci 14y ago
Follow up story on: http://news.ycombinator.com/item?id=5177815 http://news.ycombinator.com/item?id=5177815
brownbat 14y ago
Company from Taipei flashes some Intel equipment, then it appears to function correctly, but can be bricked remotely with a specially crafted incoming packet. Company has US branch that's a government contractor: http://government-contractor.bizdirlib.com/ceo/Synertron_Technology_Inc http://government-contractor.bizdirlib.com/ceo/Synertron_Tec... Charming.
- ersii 14y ago
  I think you're reading way more into this, than there is to it. Taiwan (Republic of China) is by the way, basically it's own country with it's own leadership and currency. I find it somewhat hard to put China (People's Republic of China) and Taiwan (Republic of China) together.
  - StavrosK 14y ago
    The difference is in the people.
  - alanctgardner2 14y ago
    Just to clarify, there's a difference between a Special Administrative Region like Hong Kong or Macau, and Taiwan. While Hong Kong is largely self-governing internally, it's still part of the PRC. Meanwhile, Taiwan (the ROC) was founded by people ousted during the revolution. It's like saying North and South Korea are 'basically' their own countries. Politically they aren't even friendly.
    - ersii 14y ago
      Indeed, I guess I was a little too fuzzy in how I phrased myself in hind sight. Thanks - a good addition in itself. I guess the only really suitable way of explaining the situation is "It's complicated.". It's a colourful situation and in no way is it neither black nor white.
  - brownbat 14y ago
    Hmm, thought by sticking to raw facts, I would avoid that... turns out I was wrong, now it just looks thinly veiled. Sincere apologies, poor judgment on my part.
  - ajross 14y ago
    Well, I suppose the argument would go that a Taiwanese engineer would, by dint of shared language and culture, be more susceptible to coercion and bribes than someone from europe, south asia, etc... And I agree that you don't want to be too paranoid about this stuff. But at the same time, if you were expecting and looking for an "illicit backdoor" in hardware, this is exactly the kind of thing you'd expect. Firmware in a place virtually no one knows about gets modified on a per-product basis to do nefarious things. And this is exactly how you'd expect such a modification to be discovered, by accidentally introducing a bug that distinguishes itself from the clean parent. I mean, I'm not screaming "spy" here, but if I were to have read this story in a techno-thriller novel I'd be writing a post applauding the author for her excellently researched and eminently plausible plot hook.
    - sliverstorm 14y ago
      Well, I suppose the argument would go that a Taiwanese engineer would, by dint of shared language and culture, be more susceptible to coercion and bribes than someone from europe, south asia, etc... You could try to make that argument, but I've always figured it would go kind of the other direction. Every Taiwanese national I know is a fierce supporter of Taiwan's independence of China, and China certainly does all it can to foster that every time it tries to annex the country.
  - samstave 14y ago
    I have commented about this in the past; when I was at lockheed, in 2006, we had security debriefings about chinese hacking attempts. There were trojans on the network that were sending little data packets back to china... but more interestingly: Lockheed employees were not allowed to connect their macines to any foreign network. Even of those which were suppliers. There was a supplier in Taiwan where employees would go and would transfer some files via sneakernet (USB keys) - the supplier had been hacked and the chinese were using the Taiwanese suppliers machines to attack the lockheed employees via the transfer of the USB sticks. The point is that don't underestimate Chinese hackers and the potntial vectors they are willing to exploit.
  - specialist 14y ago
    Trust, but verify. With everything that we know, it's silly to not be paranoid. I got pretty excited about election integrity for a while. The default position of the defenders of the status quo was "I can't believe you don't trust us. Prove there's something wrong. You 'experts' in computers, security, and elections are just a bunch of conspiracy freaks." My default position is "show me". That skepticism merely makes me an informed consumer.
- eps 14y ago
  If this were a DoS backdoor, it would've not been that much harder to make it less discoverable. Just use two magic bytes, or three. The chance of false positive are virtually zero and yet you'd still be able to use basic ICMP/ping to trigger it if needed.
  - mark-r 14y ago
    A backdoor that bricks the device, but only if it's the first packet received, isn't terribly useful. The more plausible explanation is that it's a bug that was introduced by the actual backdoor that still remains undiscovered.
noonespecial 14y ago
Kielhofner's response. http://blog.krisk.org/2013/02/packets-of-death-update.html http://blog.krisk.org/2013/02/packets-of-death-update.html I used to use "Lanner" gear for voip and these had embedded intel ethernets. I don't have any more of them to test, but I swear I've seen it on them as well. We suspected power supply problems because the link lights would just go dark every once in a blue moon and need a power cycle to set right, but then we were never be able to reproduce it.
- gonzo 14y ago
  We use Lanner gear for VoIP, and have never seen a problem.
  - noonespecial 14y ago
    I think we only saw it on FW-7550's at the beginning of the production run (the ones with the "snout" fans on the CPU and no case fan).
- jessaustin 14y ago
  I am impressed by his original troubleshooting, but this followup seems impractical. Of his three suggestions, only the third (Intel providing improved board testing tools) even seems like it could possibly prevent this sort of problem. Asking for hardware-enforced "sane" behavior is like asking, "why doesn't my computer know I don't want my program to deadlock, segfault, or loop indefinitely?" That is, if the controller could do that then it would solve the Halting Problem. Improved drivers, his second suggestion, are always a good thing, but drivers only get patched to handle broken hardware in response to the discovery of broken hardware. There is no way to anticipate each particular way a NIC could possibly be broken ahead of time. The market demands controllers with flexible and expandable functionality. Board manufacturers use the EEPROM to specify exactly what behavior is required. If a particular manufacturer underestimates the importance of correctness and doesn't perform the code review and testing necessary to prevent a PoD, that isn't Intel's fault.
  - viraptor 14y ago
fulafel 14y ago
Firmware images usually have checksums. Was this an Intel blob suffering from bitrot, or does Intel have some more or less error prone way to build your own FW images for NICs?
- noonespecial 14y ago
  I suspect NICs these days are tiny computers in their own right. As a motherboard manufacturer, you can probably program them to do all sorts of nifty, with the possible downside of strange things happening if you get it wrong.
- mrb 14y ago
  It is not the firmware that was corrupted. It is the EEPROM that was incorrectly programmed. The EEPROM is typically 4kB on the 82754. When it is reprogrammed either by the end-user (eg. via ethtool(1)) or by the manufacturer, the programming procedure recomputes a checksum on the first 128 bytes IIRC (when reprogramming via ethtool, the kernel driver e1000e is responisble for automatically updating the checksum. So all in all, no, the packet of death issue was not caused by bitrot.
GiHe 14y ago
Cross-posted at h-online.com: I have a plurality of systems with Intel motherboards which demonstrate the same kind of problems. The motherboards in question have two Intel ethernet controllers, one of which is an 82574L. The systems connect to two different networks. When the systems attach to one of the networks (but not the other) using the 82574L interface (but not the other), that interface dies after some unpredictable amount of time. I have tried posting comments to the Intel engineer's blog post (and PM-ing the engineer directly), but they do not appear. In fact, there seem to be no comments at Intel's site, despite the post having nearly 6000 views (at my time of writing). Something is not right here.
- kkielhofner 14y ago
  This. As I say in my updated post, this is a complex issue with clear combinatorial factors. More than likely it's not limited to one chip, one packet, or one EEPROM configuration. A quick reading of the web shows various unexplained issues with this family of Intel ethernet controllers randomly exhibiting the exact behavior I've described. Different controllers, different mobo OEMs, different EEPROM settings. Are all of these issues related to some kind of "packet of death"? Certainly not. However, are at least some of them? Almost certainly, even if they're not vulnerable to my (extremely specific) "packet of death". We still don't know exactly why this is happening (even in my extremely specific case).
  - GiHe 14y ago
    I have another interesting (and reproducible) manifestation on Supermicro motherboards with two 82574L controllers. In this case, it is again true that we only experience problems on the first (as ordered by ascending MAC address) of the two interfaces. That was the case, though I did not clearly state so above, on the Intel motherboard with one 82574L and one 82579LM.
- kkielhofner 14y ago
  I've been thinking about your situation and I'm reminded of my own words: "there is always a reason". Would you be able to e-mail me (CAPTCHA here): http://tinyurl.com/66srzt http://tinyurl.com/66srzt I'd like to discuss your issue further. Thanks!

78 points by Maci 14y ago

Maci 14y ago

Follow up story on: http://news.ycombinator.com/item?id=5177815 http://news.ycombinator.com/item?id=5177815

Company from Taipei flashes some Intel equipment, then it appears to function correctly, but can be bricked remotely with a specially crafted incoming packet. Company has US branch that's a government contractor: http://government-contractor.bizdirlib.com/ceo/Synertron_Technology_Inc http://government-contractor.bizdirlib.com/ceo/Synertron_Tec... Charming.

ersii 14y ago
I think you're reading way more into this, than there is to it. Taiwan (Republic of China) is by the way, basically it's own country with it's own leadership and currency. I find it somewhat hard to put China (People's Republic of China) and Taiwan (Republic of China) together.
- StavrosK 14y ago
  The difference is in the people.
- alanctgardner2 14y ago
  Just to clarify, there's a difference between a Special Administrative Region like Hong Kong or Macau, and Taiwan. While Hong Kong is largely self-governing internally, it's still part of the PRC. Meanwhile, Taiwan (the ROC) was founded by people ousted during the revolution. It's like saying North and South Korea are 'basically' their own countries. Politically they aren't even friendly.
  - ersii 14y ago
    Indeed, I guess I was a little too fuzzy in how I phrased myself in hind sight. Thanks - a good addition in itself. I guess the only really suitable way of explaining the situation is "It's complicated.". It's a colourful situation and in no way is it neither black nor white.
- brownbat 14y ago
  Hmm, thought by sticking to raw facts, I would avoid that... turns out I was wrong, now it just looks thinly veiled. Sincere apologies, poor judgment on my part.
- ajross 14y ago
  Well, I suppose the argument would go that a Taiwanese engineer would, by dint of shared language and culture, be more susceptible to coercion and bribes than someone from europe, south asia, etc... And I agree that you don't want to be too paranoid about this stuff. But at the same time, if you were expecting and looking for an "illicit backdoor" in hardware, this is exactly the kind of thing you'd expect. Firmware in a place virtually no one knows about gets modified on a per-product basis to do nefarious things. And this is exactly how you'd expect such a modification to be discovered, by accidentally introducing a bug that distinguishes itself from the clean parent. I mean, I'm not screaming "spy" here, but if I were to have read this story in a techno-thriller novel I'd be writing a post applauding the author for her excellently researched and eminently plausible plot hook.
  - sliverstorm 14y ago
    Well, I suppose the argument would go that a Taiwanese engineer would, by dint of shared language and culture, be more susceptible to coercion and bribes than someone from europe, south asia, etc... You could try to make that argument, but I've always figured it would go kind of the other direction. Every Taiwanese national I know is a fierce supporter of Taiwan's independence of China, and China certainly does all it can to foster that every time it tries to annex the country.
- samstave 14y ago
  I have commented about this in the past; when I was at lockheed, in 2006, we had security debriefings about chinese hacking attempts. There were trojans on the network that were sending little data packets back to china... but more interestingly: Lockheed employees were not allowed to connect their macines to any foreign network. Even of those which were suppliers. There was a supplier in Taiwan where employees would go and would transfer some files via sneakernet (USB keys) - the supplier had been hacked and the chinese were using the Taiwanese suppliers machines to attack the lockheed employees via the transfer of the USB sticks. The point is that don't underestimate Chinese hackers and the potntial vectors they are willing to exploit.
- specialist 14y ago
  Trust, but verify. With everything that we know, it's silly to not be paranoid. I got pretty excited about election integrity for a while. The default position of the defenders of the status quo was "I can't believe you don't trust us. Prove there's something wrong. You 'experts' in computers, security, and elections are just a bunch of conspiracy freaks." My default position is "show me". That skepticism merely makes me an informed consumer.
eps 14y ago
If this were a DoS backdoor, it would've not been that much harder to make it less discoverable. Just use two magic bytes, or three. The chance of false positive are virtually zero and yet you'd still be able to use basic ICMP/ping to trigger it if needed.
- mark-r 14y ago
  A backdoor that bricks the device, but only if it's the first packet received, isn't terribly useful. The more plausible explanation is that it's a bug that was introduced by the actual backdoor that still remains undiscovered.

noonespecial 14y ago

Kielhofner's response. http://blog.krisk.org/2013/02/packets-of-death-update.html http://blog.krisk.org/2013/02/packets-of-death-update.html I used to use "Lanner" gear for voip and these had embedded intel ethernets. I don't have any more of them to test, but I swear I've seen it on them as well. We suspected power supply problems because the link lights would just go dark every once in a blue moon and need a power cycle to set right, but then we were never be able to reproduce it.

gonzo 14y ago
We use Lanner gear for VoIP, and have never seen a problem.
- noonespecial 14y ago
  I think we only saw it on FW-7550's at the beginning of the production run (the ones with the "snout" fans on the CPU and no case fan).
jessaustin 14y ago
I am impressed by his original troubleshooting, but this followup seems impractical. Of his three suggestions, only the third (Intel providing improved board testing tools) even seems like it could possibly prevent this sort of problem. Asking for hardware-enforced "sane" behavior is like asking, "why doesn't my computer know I don't want my program to deadlock, segfault, or loop indefinitely?" That is, if the controller could do that then it would solve the Halting Problem. Improved drivers, his second suggestion, are always a good thing, but drivers only get patched to handle broken hardware in response to the discovery of broken hardware. There is no way to anticipate each particular way a NIC could possibly be broken ahead of time. The market demands controllers with flexible and expandable functionality. Board manufacturers use the EEPROM to specify exactly what behavior is required. If a particular manufacturer underestimates the importance of correctness and doesn't perform the code review and testing necessary to prevent a PoD, that isn't Intel's fault.
- viraptor 14y ago

fulafel 14y ago

Firmware images usually have checksums. Was this an Intel blob suffering from bitrot, or does Intel have some more or less error prone way to build your own FW images for NICs?

noonespecial 14y ago
I suspect NICs these days are tiny computers in their own right. As a motherboard manufacturer, you can probably program them to do all sorts of nifty, with the possible downside of strange things happening if you get it wrong.
mrb 14y ago
It is not the firmware that was corrupted. It is the EEPROM that was incorrectly programmed. The EEPROM is typically 4kB on the 82754. When it is reprogrammed either by the end-user (eg. via ethtool(1)) or by the manufacturer, the programming procedure recomputes a checksum on the first 128 bytes IIRC (when reprogramming via ethtool, the kernel driver e1000e is responisble for automatically updating the checksum. So all in all, no, the packet of death issue was not caused by bitrot.

GiHe 14y ago

Cross-posted at h-online.com: I have a plurality of systems with Intel motherboards which demonstrate the same kind of problems. The motherboards in question have two Intel ethernet controllers, one of which is an 82574L. The systems connect to two different networks. When the systems attach to one of the networks (but not the other) using the 82574L interface (but not the other), that interface dies after some unpredictable amount of time. I have tried posting comments to the Intel engineer's blog post (and PM-ing the engineer directly), but they do not appear. In fact, there seem to be no comments at Intel's site, despite the post having nearly 6000 views (at my time of writing). Something is not right here.

kkielhofner 14y ago
This. As I say in my updated post, this is a complex issue with clear combinatorial factors. More than likely it's not limited to one chip, one packet, or one EEPROM configuration. A quick reading of the web shows various unexplained issues with this family of Intel ethernet controllers randomly exhibiting the exact behavior I've described. Different controllers, different mobo OEMs, different EEPROM settings. Are all of these issues related to some kind of "packet of death"? Certainly not. However, are at least some of them? Almost certainly, even if they're not vulnerable to my (extremely specific) "packet of death". We still don't know exactly why this is happening (even in my extremely specific case).
- GiHe 14y ago
  I have another interesting (and reproducible) manifestation on Supermicro motherboards with two 82574L controllers. In this case, it is again true that we only experience problems on the first (as ordered by ascending MAC address) of the two interfaces. That was the case, though I did not clearly state so above, on the Intel motherboard with one 82574L and one 82579LM.
kkielhofner 14y ago
I've been thinking about your situation and I'm reminded of my own words: "there is always a reason". Would you be able to e-mail me (CAPTCHA here): http://tinyurl.com/66srzt http://tinyurl.com/66srzt I'd like to discuss your issue further. Thanks!