6 ms·

That is great HN content! Debugging deep down the rabbit hole, until you find a bug in the NIC EEPROM - and the disbelief many show when hearing a software mes

by guylhem 14y ago

That is great HN content! Debugging deep down the rabbit hole, until you find a bug in the NIC EEPROM - and the disbelief many show when hearing a software message can bring down a NIC. I for one would enjoy reading more content like this on HN that what qualifies as best as a friday-night hack

brazzy 14y ago
> the disbelief many show when hearing a software message can bring down a NIC. Shouldn't be a surprise to anyone. Firmware is just software, and it necessarily deals with raw bytes. Not really surprising that it can contain bugs that are triggered by certain byte patterns.
- moconnor 14y ago
  In this light, the surprising thing is that we don't hear about firmware bugs more often. Have manufacturers have been doing a great job of keeping things simple and exhaustively testing, driven by the fear of the losses an incident and recall would cause?
  - belorn 14y ago
    Rare bugs on released hardware are almost by nature ignored in favor of new products. Its also why fuzzing is such as great method in finding security bugs.
    - helmut_hed 14y ago
      No revenue accrues from hardware that's already been shipped :)
    - 7952 14y ago
      This is probably common in routers. It is easier for most people to buy a new one than upgrade the firmware that was 3 years old when the device was bought.
  - kisielk 14y ago
    In my experience many software developers are pretty ignorant about hardware issues since they rarely have to deal with hardware beyond their desktops or laptops. If you live in the IT world for a while and work with enterprise-grade equipment these kinds of problems actually crop up a lot. For any given piece of gear in your server room there is likely a massive list of firmware bugs for it and its subcomponents. Sometimes they are fixed, sometimes not. When you run in to one that affects and need to deploy a firmware fix to dozens or hundreds of servers, it's not a whole lot of fun.
- voidlogic 14y ago
  It would be interesting to see what the actual error in the NIC firmware source was. This invalidates my assumption that a shop like Intel probably uses formal verification in firmware development. Its also scary to consider how many very important (nuclear/damn control, etc) systems while themselves might be formally verified are dependent on services of lower level software (OS, drivers) and hardware (firmware) that are not...
  - fnordfnordfnord 14y ago
    Dam control, or damn control? I teach a control systems course, so I know about both kinds. PS You should be very frightened. The industrial control systems industry are slow to adopt new tech/new best practices/anything else new; and even more reticent to touch anything that "works." Tons of vulnerable equipment controlling dangerous things with default passwords (if any at all). I could go on for days.
    - kevinherron 14y ago
      Too true... (I work at a SCADA/HMI software company :)
    - voidlogic 14y ago
      Sorry typo :(
    - pnathan 14y ago
      Absolutely utterly true. (I work at a SCADA-ish vendor).
  - Joeri 14y ago
    Isn't formal verification the unicorn of software development? I know it's rumored to exist, and you will find it in academia, but I've never come across it 'in the wild'.
    - voidlogic 14y ago
      It is out there: http://en.wikipedia.org/wiki/Formal_verification#Industry_use http://en.wikipedia.org/wiki/Formal_verification#Industry_us... In the wild what is more common in my experience is to use a formal specification language like Z or Object-Z and then recreate the the program in a "normal" language using pair programmers and/or extensive code review to attempt to ensure it conforms to the specification. While not perfect, for critical things this approach is far more time consuming and less bug prone than typical development. I have also seen limited attempts at writing code generators that take Object-Z specifications as input. My understanding is that this is an area of ongoing research, but even a program that generates code for a subset of a Object-Z specification can save a lot of time. Example research: http://www.jot.fm/issues/issue_2008_11/article6.pdf http://www.jot.fm/issues/issue_2008_11/article6.pdf
- ajross 14y ago
  Indeed. But unlike "just software", firmware is never, ever fixable in source form except deep within the bowels of hardware manufacturers. Even when found, firmware bugs are generally ignored. I wonder if anyone has checked: has the fixed image been released to linux-firmware, or the Windows driver yet? Will it ever? Just browse through the ./drivers tree of the kernel source some time and look at all the "quirks" and "workarounds" there. Recognize that virtually all of those could have been fixed in firmware, but weren't because no one cared. It's just deeply depressing. This was a virtuoso debugging performance, but it didn't have to be that way if hardware companies were sane. But they aren't, and we all pay for it. (edit: I just checked for myself, the Linux e1000e driver apparently doesn't support runtime firmware update via the kernel API, so linux-firmware wouldn't be expected to have this. I don't know what the process is for an affected end user to get a copy. I suspect there is none.)
- curiousdannii 14y ago
  I know a dev on the OLPC project who told me they found bugs in the firmware of the SD cards they were using for their laptop 'harddrive'. I imagine it'd be a whole lot of fun fixing them.
sideproject 14y ago
That is some serious debugging, supported by even more serious persistence by the author. Total respect.
noselasd 14y ago
You may then enjoy these: * http://www.youtube.com/watch?v=euMHlV6MNqs http://www.youtube.com/watch?v=euMHlV6MNqs * http://www.youtube.com/watch?v=8Q8EFwKVKdA http://www.youtube.com/watch?v=8Q8EFwKVKdA
- guylhem 14y ago
  (watched this and submitted it immediately to HN) HIGHLY recommended to anyone who enjoyed the article. It's a remote PHY injection at the hardware level - basically hacking the chip directly thanks to "special" content, i.e. putting a packet header within the packet.

by guylhem 14y ago

brazzy 14y ago

> the disbelief many show when hearing a software message can bring down a NIC. Shouldn't be a surprise to anyone. Firmware is just software, and it necessarily deals with raw bytes. Not really surprising that it can contain bugs that are triggered by certain byte patterns.

moconnor 14y ago
In this light, the surprising thing is that we don't hear about firmware bugs more often. Have manufacturers have been doing a great job of keeping things simple and exhaustively testing, driven by the fear of the losses an incident and recall would cause?
- belorn 14y ago
  Rare bugs on released hardware are almost by nature ignored in favor of new products. Its also why fuzzing is such as great method in finding security bugs.
  - helmut_hed 14y ago
    No revenue accrues from hardware that's already been shipped :)
  - 7952 14y ago
    This is probably common in routers. It is easier for most people to buy a new one than upgrade the firmware that was 3 years old when the device was bought.
- kisielk 14y ago
  In my experience many software developers are pretty ignorant about hardware issues since they rarely have to deal with hardware beyond their desktops or laptops. If you live in the IT world for a while and work with enterprise-grade equipment these kinds of problems actually crop up a lot. For any given piece of gear in your server room there is likely a massive list of firmware bugs for it and its subcomponents. Sometimes they are fixed, sometimes not. When you run in to one that affects and need to deploy a firmware fix to dozens or hundreds of servers, it's not a whole lot of fun.
voidlogic 14y ago
It would be interesting to see what the actual error in the NIC firmware source was. This invalidates my assumption that a shop like Intel probably uses formal verification in firmware development. Its also scary to consider how many very important (nuclear/damn control, etc) systems while themselves might be formally verified are dependent on services of lower level software (OS, drivers) and hardware (firmware) that are not...
- fnordfnordfnord 14y ago
  Dam control, or damn control? I teach a control systems course, so I know about both kinds. PS You should be very frightened. The industrial control systems industry are slow to adopt new tech/new best practices/anything else new; and even more reticent to touch anything that "works." Tons of vulnerable equipment controlling dangerous things with default passwords (if any at all). I could go on for days.
  - kevinherron 14y ago
    Too true... (I work at a SCADA/HMI software company :)
  - voidlogic 14y ago
    Sorry typo :(
  - pnathan 14y ago
    Absolutely utterly true. (I work at a SCADA-ish vendor).
- Joeri 14y ago
  Isn't formal verification the unicorn of software development? I know it's rumored to exist, and you will find it in academia, but I've never come across it 'in the wild'.
  - voidlogic 14y ago
    It is out there: http://en.wikipedia.org/wiki/Formal_verification#Industry_use http://en.wikipedia.org/wiki/Formal_verification#Industry_us... In the wild what is more common in my experience is to use a formal specification language like Z or Object-Z and then recreate the the program in a "normal" language using pair programmers and/or extensive code review to attempt to ensure it conforms to the specification. While not perfect, for critical things this approach is far more time consuming and less bug prone than typical development. I have also seen limited attempts at writing code generators that take Object-Z specifications as input. My understanding is that this is an area of ongoing research, but even a program that generates code for a subset of a Object-Z specification can save a lot of time. Example research: http://www.jot.fm/issues/issue_2008_11/article6.pdf http://www.jot.fm/issues/issue_2008_11/article6.pdf
ajross 14y ago
Indeed. But unlike "just software", firmware is never, ever fixable in source form except deep within the bowels of hardware manufacturers. Even when found, firmware bugs are generally ignored. I wonder if anyone has checked: has the fixed image been released to linux-firmware, or the Windows driver yet? Will it ever? Just browse through the ./drivers tree of the kernel source some time and look at all the "quirks" and "workarounds" there. Recognize that virtually all of those could have been fixed in firmware, but weren't because no one cared. It's just deeply depressing. This was a virtuoso debugging performance, but it didn't have to be that way if hardware companies were sane. But they aren't, and we all pay for it. (edit: I just checked for myself, the Linux e1000e driver apparently doesn't support runtime firmware update via the kernel API, so linux-firmware wouldn't be expected to have this. I don't know what the process is for an affected end user to get a copy. I suspect there is none.)
curiousdannii 14y ago
I know a dev on the OLPC project who told me they found bugs in the firmware of the SD cards they were using for their laptop 'harddrive'. I imagine it'd be a whole lot of fun fixing them.

sideproject 14y ago

That is some serious debugging, supported by even more serious persistence by the author. Total respect.

noselasd 14y ago

You may then enjoy these: * http://www.youtube.com/watch?v=euMHlV6MNqs http://www.youtube.com/watch?v=euMHlV6MNqs * http://www.youtube.com/watch?v=8Q8EFwKVKdA http://www.youtube.com/watch?v=8Q8EFwKVKdA

guylhem 14y ago
(watched this and submitted it immediately to HN) HIGHLY recommended to anyone who enjoyed the article. It's a remote PHY injection at the hardware level - basically hacking the chip directly thanks to "special" content, i.e. putting a packet header within the packet.