6 ms·

They don't care, until a kernel panic causes disk corruption in just the right way to cause a cross-security-boundary information leak that also ends up breakin

They don't care, until a kernel panic causes disk corruption in just the right way to cause a cross-security-boundary information leak that also ends up breaking an entire global pipeline, or a rare buffer overread in a ubiquitous piece of code causes once in a petabyte mismatches in cross-checked processes with compliance implications. Source: xoogler, developed a reputation for black magic debugging, debugged both of those things. Correctness still matters, no matter how big you are or how much you think auto-restarts help.

jasmas 5y ago
Nah they still don't care
- nisegami 5y ago
  Never stay at one place long enough for these things to come up /s
- bluGill 5y ago
  That depends on how noticed it gets. My company makes embedded systems that in some failure cases can kill - we have a history of losing large lawsuits over the base 70 years (before computers of course). As such we have a powerful safety review system in place because million dollar lawsuits make CEOs care - it isn't perfect but there are a lot of things that we don't do that our customers would love.
discardable_dan 5y ago
Yes, they really don't care unless it results in a site going down. An industrial giant is happy to throw infinite machines at a problem until it is resolved. The issue isn't in their behavior, it's in Linux's implementation being so poor that they have to. But what are the alternatives? Should Google/Facebook/Amazon throw infinite developer time resolving kernel issues? Should they walk away from Linux?
- mhoad 5y ago
  I mean one of them at least went ahead and wrote (technically is in the process of writing) an entire operating system from scratch in response. I say this as someone who uses and has used linux as a daily driver for years now but honestly I don’t feel good about the future of the platform in the medium term at all. If anyone has some time to throw at understanding why this is probably the single most sober and rational set of arguments I’ve found of the topic yet https://m.youtube.com/watch?v=WtJ9T_IJOPE https://m.youtube.com/watch?v=WtJ9T_IJOPE
  - samhw 5y ago
    Sorry if this is a bit blunt, but I'm mystified by the description of that video as "sober and rational". I'm not a Linux fanboy at all -- I happily use my Mac and don't even own a Linux computer besides my Raspberry Pi -- and in fact I was looking forward to digging into that talk, but it's just a guy rambling extraordinarily slowly about a bunch of half-points for an hour. The parts that I managed to endure were these: "Some Linux figures are being cancelled, like Richard Stallman." Silence the pianos and with muffled drum... oh wait, actually, systemd will probably keep running and crons will keep firing. "Linux companies undergoing changes." Where's Heraclitus when you need him? "Linux complexity increasing rapidly." He counts 2m lines of code added this year. But it's mostly drivers, and anyway this number has been trending downwards for a decade from what I can see (https://www.pingdom.com/wp-content/uploads/2012/04/linux-kernel.0021.jpg https://www.pingdom.com/wp-content/uploads/2012/04/linux-ker...). "All Linux events are cancelled." Hmm, far be it from me to say, but I faintly recall that there was something besides Linux which might account for that. "Google building Linux replacement." Google not building Linux replacement, Google building operating system. "No operating system stays dominant forever." Linux is indeed not Ozymandias, sitting alone at the top of all creation, outside of space and time ... but I'm not sure how that means it sucks. I wanted to like this, I honestly did, but the arguments are so dreadful that it feels like a deleted scene from one of Plato's dialogues. And Lord almighty he could have got to the point - such as it was - in much less than an hour.
    - effie 5y ago
diveanon 5y ago
Why should they care? Sounds like that's your job.
nefitty 5y ago
Fuck. I envy the mental models you can probably evoke on command. If you have a second to engage, I have found that I learn best in sandbox-types of environments, ie repls, notebooks, etc. What method do you find yourself going back to when you need to expand your understanding of a big production system like at Google? It might be my own ignorance, but I'm not sure how to move into big-systems engineering with low-stakes realism.
- marcan_42 5y ago
  Most of the time this kind of debugging is either retroactive (look through logs and just try to piece things together from all available information), or you can manage to get a test case out of the system. In both cases, it's a matter of digging through the stack in some direction. So it's a lot of code searching to work out how different pieces fit together, and looking through the bread crumbs left in logs. This isn't unique to big production systems in any way; while the numbers may have a few extra zeroes and the pieces of code may be more disparate and numerous, it's exactly the same process I go through today when debugging a kernel issue (these days I work on Linux support for reverse engineered Apple Silicon platforms). You start at the point where the problem happens, then add debugging statements and dig through the call stack, gaining an understanding of the system as you go, until you find the root cause. The good thing about Google, at least in the field I worked in (production SRE), is that usually there was enough logging already, in different forms and at different levels, that you could work things out without resorting to patching code to add more. But digging through the stack like this is certainly a skill, and it applies regardless of the scale of the system. This skill is the crux of a fairly classic Google interview question: "You type www.google.com into your web browser and hit enter, what happens?" The goal there isn't to test your knowledge of any specific part of the puzzle, but rather to see if you have an understanding of how the puzzle fits together at some level, and hopefully enough of an interest in some part of it to go into significant detail. If you're a networking person, you might start talking about DNS requests, the TCP 3-way handshake, TLS handshakes, HTTP headers... If you're a web person, you might start describing how HTML is progressively loaded and resources are fetched in the background, how CSS lookups happen, how the DOM works... If you're a systems person, you might start talking about input events from the window manager, system calls to initiate network operations, caching to local filesystem... And if you're a smartass like me, you might start "At first, nothing happens; the microcontroller inside your keyboard is likely sleeping waiting for an interrupt that will have it advance to the next column in the scan matrix. Eventually, once the timer expires and the interrupt fires enough times to reach the column corresponding to the Enter key, it will bring that column to a logic high level, and the voltage will travel through the Enter key switch into the corresponding row line. The microcontroller will sample this line a short time later, and detect that a key has been pressed. It will then build a USB HID keyboard report packet containing the scan code, which will be placed in a buffer. The USB device hardware will then wait until it receives an INTERRUPT IN packet from the USB host, addressed to the HID interface endpoint responsible for delivering input reports..." (if I really wanted to frustrate the interviewer, I could start by describing a generic clock oscillator circuit and how the timer countdown logic might be implemented in terms of logic gates and transistors... but that might be going a bit too far, and we probably wouldn't get to the USB packet before the interview was over :-) ) All of those are valid answers, and the idea is that hopefully you can dive through some subset of the stack. So if you're interested in this kind of thing, that's a good place to start - pick something you're at least vaguely familiar with, and start digging through the stack to see how much of an understanding you can get of the different bits and pieces. Doing this with an open source OS is invaluable, since all the code is right there at your fingertips - there is nothing you can't figure out given enough time and patience. If you find interactive environments with a REPL more intuitive, you might use a debugger, set a breakpoint at the top level of the code, and single-step your way down the stack. If you're more of a reading code and printf() debugging person like me, you might do it that way instead. If you're interesting in digging downward towards the metal, perhaps try working with microcontrollers (e.g. Arduino, but make sure you dig through the libraries, don't stop at the interface - it's not that hard to learn how to program 8-bit microcontrollers in assembly!), and if you want to look at larger distributed systems, perhaps take a look at something like Ceph or some kind of Kubernetes deployment (both of which can be tested in a single machine with containers or VMs).
pi7h3n 5y ago
What kind of black magic debugging?
- deleted 5y ago
  [deleted]
- marcan_42 5y ago
  HN always seems to like this one when it comes up :-) https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ This one isn't bad either: https://marcan.st/2015/10/rescuing-a-broken-ntfs-filesystem/ https://marcan.st/2015/10/rescuing-a-broken-ntfs-filesystem/
thomasjudge 5y ago
I know you probably can't share details, but would be curious to hear a little more color around these two situations.
- marcan_42 5y ago
  Honestly, it's not that sensitive (at least without service details; plus this was over 7 years ago). I think the lawyers won't send me a nastygram for this much detail: The kernel panic one started when I got pinged by a downstream team that we somehow delivered garbage data (valid-looking but entirely the wrong schema) to them, and that broke everything. Tracing things back up the chain, I found out that the source machine had had a kernel panic. Turns out that ended up with file metadata flushed to disk, but not file data. The unwritten sectors happened to contain data from a different process that was in the right file format, but had the wrong payloads, and since the file format was garbage-tolerant and self-synchronizing, the processing from that point on cleaned it up into data that looked completely valid... but was just from entirely the wrong place. All that happened automatically upon reboot from the panic. I was able to quite conclusively prove that this is what had happened by observing the data offsets logged to debug logs, and noting that the funkiness happened on filesystem block boundaries, and also aligned time-wise with the kernel panic, and a process that matched the schema of the data that ended up delivered had been running simultaneously on the same machine. This was all from just one instance of the fault, forensically debugged; we never reproduced it. The conclusion was basically "working as intended"... that is, this is apparently a thing that can happen given the filesystem mount mode in use (which was chosen for performance), so I think in the end they introduced some sanity checks on the data shape as a stop-gap. I think this was all soon deprecated and replaced with a system that didn't use local disk at all anyway, so there wasn't much point in introducing a big refactor to eliminate this rare failure mode at that point, but it's interesting that it did have security implications (it was just rare and not directly triggerable). It was very satisfying being able to trace down exactly what had happened (and the team lead sent this one off to me specifically due to my reputation for working this kind of stuff out... :) ). The other one had to do with Google's NIHed gzip implementation, because of course they have one. We had found data mismatches between output computed over identical input in two different data centers (for redundancy/cross-checking), but it always went away if we tried again. This happened once a petabyte or so, give or take (I'll let you guess how often that was in terms of days, at Google scale). Digging through the error logs, I found that the mismatch was an offset mismatch, not a data mismatch. These files had gzip-compressed data blocks. Investigating the data was tricky because we weren't allowed to copy parts of it to workstations due to privacy reasons, but doing some contortions to work in the cloud I found out that the logical data was equivalent, but the compressed data was one or two bytes larger in one of the files, hence shifting the offset of the next block. Eventually I managed to grab one of the problem-causing uncompressed data blocks, build a test case for the gzip compressor with exactly the same flags used in production, and I had the idea of running it under valgrind... and that's when it screamed. Turns out the gzip compressor used a ring buffer of history to perform searches for compression, and for performance reasons the lookups weren't always wrapped, but rather there was a chunk of the ring buffer duplicated at the end. This was calculated to never exceed how much the pointer could fall off the end of the buffer. But, on top of that, the reads used unaligned 32-bit reads, again for performance reasons. And so you could end up with the pointer less than 4 bytes from the end of the buffer padding area, reading 1 to 3 bytes of random RAM after that. The way the compressor worked, if the data was bad it wouldn't break the compression, but it could compress a bit less. And so, the chances of both the read falling off the end of the buffer and happening to be one that would've improved compression were tiny, and that's how we ended up with the rarity. We never saw crashes due to the OOB read, but I think I heard some other team had been having actual segfaults (I guess the page after the buffer was unmapped for them) and this fix resolved them. I think we had the problem going on sporadically for a few months before I got sick of it (and I'd managed to rule out the obvious "it's a RAM bit flip / bad CPU / something" things you tend blame rare random failures on at that kind of scale) and spent a couple evenings figuring it out.
nijave 5y ago
Honestly I think this just rolls down from management. "Oh you can spend $x * 80 hours debugging or $x * 2 hours putting a patch to constantly restart" It's hard to quantify the impact of having a house of cards system with components constantly restarting to side steps bugs until the whole thing lights on fire in a cascading failure later (oh the service scheduler was down for 30 mins and things are restarting so much a massive amount of capacity went offline during that time)
- vlovich123 5y ago
  There's a cost efficiency tradeoff that's always important to keep in mind. As an example, I was working with a very experienced kernel developer on a (at the time) bleeding edge mobile phone. We were seeing random memory corruption causing kernel panics. The engineer spent weeks trying to find the root cause but eventually gave up and put in a patch. You can't lose the forest for the trees - you still have product to ship & other priorities. It's rare for any bug to be so valuable to fix that it eats up all the other oxygen in the room.

jasmas 5y ago

Nah they still don't care

nisegami 5y ago
Never stay at one place long enough for these things to come up /s
bluGill 5y ago
That depends on how noticed it gets. My company makes embedded systems that in some failure cases can kill - we have a history of losing large lawsuits over the base 70 years (before computers of course). As such we have a powerful safety review system in place because million dollar lawsuits make CEOs care - it isn't perfect but there are a lot of things that we don't do that our customers would love.

discardable_dan 5y ago

Yes, they really don't care unless it results in a site going down. An industrial giant is happy to throw infinite machines at a problem until it is resolved. The issue isn't in their behavior, it's in Linux's implementation being so poor that they have to. But what are the alternatives? Should Google/Facebook/Amazon throw infinite developer time resolving kernel issues? Should they walk away from Linux?

mhoad 5y ago
I mean one of them at least went ahead and wrote (technically is in the process of writing) an entire operating system from scratch in response. I say this as someone who uses and has used linux as a daily driver for years now but honestly I don’t feel good about the future of the platform in the medium term at all. If anyone has some time to throw at understanding why this is probably the single most sober and rational set of arguments I’ve found of the topic yet https://m.youtube.com/watch?v=WtJ9T_IJOPE https://m.youtube.com/watch?v=WtJ9T_IJOPE
- samhw 5y ago
  Sorry if this is a bit blunt, but I'm mystified by the description of that video as "sober and rational". I'm not a Linux fanboy at all -- I happily use my Mac and don't even own a Linux computer besides my Raspberry Pi -- and in fact I was looking forward to digging into that talk, but it's just a guy rambling extraordinarily slowly about a bunch of half-points for an hour. The parts that I managed to endure were these: "Some Linux figures are being cancelled, like Richard Stallman." Silence the pianos and with muffled drum... oh wait, actually, systemd will probably keep running and crons will keep firing. "Linux companies undergoing changes." Where's Heraclitus when you need him? "Linux complexity increasing rapidly." He counts 2m lines of code added this year. But it's mostly drivers, and anyway this number has been trending downwards for a decade from what I can see (https://www.pingdom.com/wp-content/uploads/2012/04/linux-kernel.0021.jpg https://www.pingdom.com/wp-content/uploads/2012/04/linux-ker...). "All Linux events are cancelled." Hmm, far be it from me to say, but I faintly recall that there was something besides Linux which might account for that. "Google building Linux replacement." Google not building Linux replacement, Google building operating system. "No operating system stays dominant forever." Linux is indeed not Ozymandias, sitting alone at the top of all creation, outside of space and time ... but I'm not sure how that means it sucks. I wanted to like this, I honestly did, but the arguments are so dreadful that it feels like a deleted scene from one of Plato's dialogues. And Lord almighty he could have got to the point - such as it was - in much less than an hour.
  - effie 5y ago

diveanon 5y ago

Why should they care? Sounds like that's your job.

nefitty 5y ago

Fuck. I envy the mental models you can probably evoke on command. If you have a second to engage, I have found that I learn best in sandbox-types of environments, ie repls, notebooks, etc. What method do you find yourself going back to when you need to expand your understanding of a big production system like at Google? It might be my own ignorance, but I'm not sure how to move into big-systems engineering with low-stakes realism.

marcan_42 5y ago
Most of the time this kind of debugging is either retroactive (look through logs and just try to piece things together from all available information), or you can manage to get a test case out of the system. In both cases, it's a matter of digging through the stack in some direction. So it's a lot of code searching to work out how different pieces fit together, and looking through the bread crumbs left in logs. This isn't unique to big production systems in any way; while the numbers may have a few extra zeroes and the pieces of code may be more disparate and numerous, it's exactly the same process I go through today when debugging a kernel issue (these days I work on Linux support for reverse engineered Apple Silicon platforms). You start at the point where the problem happens, then add debugging statements and dig through the call stack, gaining an understanding of the system as you go, until you find the root cause. The good thing about Google, at least in the field I worked in (production SRE), is that usually there was enough logging already, in different forms and at different levels, that you could work things out without resorting to patching code to add more. But digging through the stack like this is certainly a skill, and it applies regardless of the scale of the system. This skill is the crux of a fairly classic Google interview question: "You type www.google.com into your web browser and hit enter, what happens?" The goal there isn't to test your knowledge of any specific part of the puzzle, but rather to see if you have an understanding of how the puzzle fits together at some level, and hopefully enough of an interest in some part of it to go into significant detail. If you're a networking person, you might start talking about DNS requests, the TCP 3-way handshake, TLS handshakes, HTTP headers... If you're a web person, you might start describing how HTML is progressively loaded and resources are fetched in the background, how CSS lookups happen, how the DOM works... If you're a systems person, you might start talking about input events from the window manager, system calls to initiate network operations, caching to local filesystem... And if you're a smartass like me, you might start "At first, nothing happens; the microcontroller inside your keyboard is likely sleeping waiting for an interrupt that will have it advance to the next column in the scan matrix. Eventually, once the timer expires and the interrupt fires enough times to reach the column corresponding to the Enter key, it will bring that column to a logic high level, and the voltage will travel through the Enter key switch into the corresponding row line. The microcontroller will sample this line a short time later, and detect that a key has been pressed. It will then build a USB HID keyboard report packet containing the scan code, which will be placed in a buffer. The USB device hardware will then wait until it receives an INTERRUPT IN packet from the USB host, addressed to the HID interface endpoint responsible for delivering input reports..." (if I really wanted to frustrate the interviewer, I could start by describing a generic clock oscillator circuit and how the timer countdown logic might be implemented in terms of logic gates and transistors... but that might be going a bit too far, and we probably wouldn't get to the USB packet before the interview was over :-) ) All of those are valid answers, and the idea is that hopefully you can dive through some subset of the stack. So if you're interested in this kind of thing, that's a good place to start - pick something you're at least vaguely familiar with, and start digging through the stack to see how much of an understanding you can get of the different bits and pieces. Doing this with an open source OS is invaluable, since all the code is right there at your fingertips - there is nothing you can't figure out given enough time and patience. If you find interactive environments with a REPL more intuitive, you might use a debugger, set a breakpoint at the top level of the code, and single-step your way down the stack. If you're more of a reading code and printf() debugging person like me, you might do it that way instead. If you're interesting in digging downward towards the metal, perhaps try working with microcontrollers (e.g. Arduino, but make sure you dig through the libraries, don't stop at the interface - it's not that hard to learn how to program 8-bit microcontrollers in assembly!), and if you want to look at larger distributed systems, perhaps take a look at something like Ceph or some kind of Kubernetes deployment (both of which can be tested in a single machine with containers or VMs).

pi7h3n 5y ago

What kind of black magic debugging?

deleted 5y ago
[deleted]
marcan_42 5y ago
HN always seems to like this one when it comes up :-) https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ This one isn't bad either: https://marcan.st/2015/10/rescuing-a-broken-ntfs-filesystem/ https://marcan.st/2015/10/rescuing-a-broken-ntfs-filesystem/

thomasjudge 5y ago

I know you probably can't share details, but would be curious to hear a little more color around these two situations.

marcan_42 5y ago
Honestly, it's not that sensitive (at least without service details; plus this was over 7 years ago). I think the lawyers won't send me a nastygram for this much detail: The kernel panic one started when I got pinged by a downstream team that we somehow delivered garbage data (valid-looking but entirely the wrong schema) to them, and that broke everything. Tracing things back up the chain, I found out that the source machine had had a kernel panic. Turns out that ended up with file metadata flushed to disk, but not file data. The unwritten sectors happened to contain data from a different process that was in the right file format, but had the wrong payloads, and since the file format was garbage-tolerant and self-synchronizing, the processing from that point on cleaned it up into data that looked completely valid... but was just from entirely the wrong place. All that happened automatically upon reboot from the panic. I was able to quite conclusively prove that this is what had happened by observing the data offsets logged to debug logs, and noting that the funkiness happened on filesystem block boundaries, and also aligned time-wise with the kernel panic, and a process that matched the schema of the data that ended up delivered had been running simultaneously on the same machine. This was all from just one instance of the fault, forensically debugged; we never reproduced it. The conclusion was basically "working as intended"... that is, this is apparently a thing that can happen given the filesystem mount mode in use (which was chosen for performance), so I think in the end they introduced some sanity checks on the data shape as a stop-gap. I think this was all soon deprecated and replaced with a system that didn't use local disk at all anyway, so there wasn't much point in introducing a big refactor to eliminate this rare failure mode at that point, but it's interesting that it did have security implications (it was just rare and not directly triggerable). It was very satisfying being able to trace down exactly what had happened (and the team lead sent this one off to me specifically due to my reputation for working this kind of stuff out... :) ). The other one had to do with Google's NIHed gzip implementation, because of course they have one. We had found data mismatches between output computed over identical input in two different data centers (for redundancy/cross-checking), but it always went away if we tried again. This happened once a petabyte or so, give or take (I'll let you guess how often that was in terms of days, at Google scale). Digging through the error logs, I found that the mismatch was an offset mismatch, not a data mismatch. These files had gzip-compressed data blocks. Investigating the data was tricky because we weren't allowed to copy parts of it to workstations due to privacy reasons, but doing some contortions to work in the cloud I found out that the logical data was equivalent, but the compressed data was one or two bytes larger in one of the files, hence shifting the offset of the next block. Eventually I managed to grab one of the problem-causing uncompressed data blocks, build a test case for the gzip compressor with exactly the same flags used in production, and I had the idea of running it under valgrind... and that's when it screamed. Turns out the gzip compressor used a ring buffer of history to perform searches for compression, and for performance reasons the lookups weren't always wrapped, but rather there was a chunk of the ring buffer duplicated at the end. This was calculated to never exceed how much the pointer could fall off the end of the buffer. But, on top of that, the reads used unaligned 32-bit reads, again for performance reasons. And so you could end up with the pointer less than 4 bytes from the end of the buffer padding area, reading 1 to 3 bytes of random RAM after that. The way the compressor worked, if the data was bad it wouldn't break the compression, but it could compress a bit less. And so, the chances of both the read falling off the end of the buffer and happening to be one that would've improved compression were tiny, and that's how we ended up with the rarity. We never saw crashes due to the OOB read, but I think I heard some other team had been having actual segfaults (I guess the page after the buffer was unmapped for them) and this fix resolved them. I think we had the problem going on sporadically for a few months before I got sick of it (and I'd managed to rule out the obvious "it's a RAM bit flip / bad CPU / something" things you tend blame rare random failures on at that kind of scale) and spent a couple evenings figuring it out.

nijave 5y ago

Honestly I think this just rolls down from management. "Oh you can spend $x * 80 hours debugging or $x * 2 hours putting a patch to constantly restart" It's hard to quantify the impact of having a house of cards system with components constantly restarting to side steps bugs until the whole thing lights on fire in a cascading failure later (oh the service scheduler was down for 30 mins and things are restarting so much a massive amount of capacity went offline during that time)

vlovich123 5y ago
There's a cost efficiency tradeoff that's always important to keep in mind. As an example, I was working with a very experienced kernel developer on a (at the time) bleeding edge mobile phone. We were seeing random memory corruption causing kernel panics. The engineer spent weeks trying to find the root cause but eventually gave up and put in a patch. You can't lose the forest for the trees - you still have product to ship & other priorities. It's rare for any bug to be so valuable to fix that it eats up all the other oxygen in the room.