14 ms·

A cache invalidation bug in Linux memory management

312 points by known 8y ago

jacquesm 8y ago
"This fits in with the kernel's policy of attempting to keep the system running as much as possible by default; if an accidental use-after-free bug occurs here for some reason, the kernel can probably heuristically mitigate its effects and keep the process working." That's a pretty bad design decision. I'd much rather have a kernel panic than a kernel that continues to run with known bad datastructures. Like that the bugs will never really shake out, silent failures like that are super dangerous because essentially the system is running in an undefined state past that point.
- quotemstr 8y ago
  I agree. Efforts to continue past unexpected states frequently just make the situation worse and end up causing more user pain than a deterministic restart. This kind of recovery also masks real bugs and leads to their persistence in the codebase. (Heuristic error recovery also makes debugging more difficult.) A fail-fast approach both preserves user invariants and provides an impetus to fix bugs fast.
  - xyproto 8y ago
    To reboot when this happen could be a kernel configuration option. This way users could choose.
- NightMKoder 8y ago
  IMO - it's actually a really hard call. Error isolation is hard but we all want it. One of the somewhat less controversial versions is failing a single user request (in node or your web framework) with a 5xx on an exception. You're essentially betting that the exception is a logic bug and not somewhere that will affect other users. That's probably true, unless, you know, your database driver is responsible. We bet that's usually not the case - and we're mostly right. Honestly I'm ok with trying to keep the system limping with a huge but - you must, at the point you first detect an error, dump...everything you know. You can try to limp along because you're trying to be a good host, but debugging after that state is, as you said, not trivial. If you separate the two concerns (post-mortem debugging & uptime) there's a nice medium to be found. Ideally kernel panics aren't the only source of observability. You can have a daemon running that files a bug report to your favorite error tracker (sentry, etc) and (attempts to) gracefully reboot the system. That would be pretty sweet.
  - jacquesm 8y ago
    Erlang is the only environment that I know about that gets this right.
- tonysdg 8y ago
  It's terrible from a correctness perspective, sure. But from a business perspective, that could mean $50 million of revenue not-lost to a flaw you (as an owner/operator) can't do much about. Sure, it could lead to an exploit like this, but I'd wager a cost-benefit analysis for almost any organization (except maybe CIA/NSA types) would support this design choice. That's not to say it's a good design choice, but it's certainly a defensible one IMO. You can have the most secure OS in the world, but if no one wants to use it, all you've got is a very secure waste of hard drive space.
  - jacquesm 8y ago
    If a kernel panic can cost you $50 million you have other problems. Really, in an organization where downtime due to a server rebooting would be that expensive you'd hope they would be able to deal with that gracefully and that they would be deploying the rough equivalent of chaosmonkey to ensure that their stuff is protected against such errors. After all, a harddrive or a CPU could die just the same.
    - sangnoir 8y ago
      Your implicit assumption is that the server will only reboot once. I think if the Linux kernel is upfront about its security posture and your threat-model is incompatible, it is OK to go with OpenBSD.
      - jacquesm 8y ago
        The more frequent the reboots the quicker the problem will get diagnosed! Especially if the system is still stable enough to write a log entry to disk.
    - ploxiln 8y ago
      This is a principle borne of experience. There have been a number of cases where a new fatal-error check killed a bunch of systems that apparently were working for a long time before. Usually it is preferred to WARN_ONCE() but not BUG_ON(). Otherwise, big corporate users, or a wide random assortment of desktop users, end up going back to an older kernel to make their systems work. The linux kernel tries above all to never have regressions, even if these regressions are just catching probably very bad bugs that were just not caught before. This happens, often enough! Think of this from the lense of natural selection. There are many many subtle tricky low-level bugs that can result in memory corruption - many drivers, many features, many optimizations, many "cleanups", a surprisingly high rate of code changes. Many of them are caught when they cause problematic corruption. Others, just due to luck, very rarely cause any visible problem, - for months or years. We do want to know about them. We do not want to suicide the whole system, it may be one of those rare bugs that made it this far because it's usually (surprisingly) not fatal on its own.
- palotasb 8y ago
  IIUC the heuristic here is how much of the system to take down (i.e., back out of fast path [this case] < kill thread < kill process < kernel panic) and this seems correct (but could still be a symptom of more corruption). Anyway, there has been discussion about this issue in general previously on the Linux kernel mailing list, and Linus said [1] that the correct procedure is to first introduce by-default reporting and an opt-in kill switch, then make the kill switch the default, then remove the non-kill option. This is supposed to weed out bugs eventually without disrupting users too much, but I can imagine that it enables some expolits. There was an HN discussion about it too. [2] [1] https://lkml.org/lkml/2017/11/21/356 https://lkml.org/lkml/2017/11/21/356 [2] https://news.ycombinator.com/item?id=15754988 https://news.ycombinator.com/item?id=15754988
- amaccuish 8y ago
  > I'd much rather have a kernel panic than a kernel that continues to run with known bad datastructures I mean, that is literally what an Oops is.
- pcwalton 8y ago
  It's extremely useful for developing drivers. I've developed drivers on kernels that are hardwired to panic at the slightest problem and it's a real nuisance. Whether you should leave it on in production is debatable, but I like at least having the option of making exceptions raised in drivers nonfatal.
  - jacquesm 8y ago
    And the reason for that is that these drivers are linked into the monolith. If they were standalone processes you'd love the ability to do post-mortem debugging on them, or maybe even the luxury of being able to run them directly under the debugger. Whenever I had to write device drivers I wrote them under QnX first and only then ported them to other OS's, that saved so much time, at least I knew the hardware interface would be up and running, and the data structures would all work as intended. After that all I then had to do was glue it to whatever calling convention the various *NIX flavors had. That trick served me well, for motion capture devices as well as for high speed serial cards and some more exotic devices.
    - im_down_w_otp 8y ago
      This is fairly analogous to our approach too, just substituting QNX w/ seL4.
tom_ 8y ago
4 billion was a lot when I first got into programming, but it just isn't that much any more. Always start out by making your sequence numbers 64 bits: 2^64 nanoseconds = ~585 years. (I have in the past stolen a few bits for other uses, but I'm less certain of the wisdom of that these days.) 32 bits is suitable for things that can only happen once a frame - (2^32/60) seconds = 2.26 years - but not much more than that. But there's a flip side to computers being able to wrap a uint32_t so quickly: if you've got 32 bits'-worth of combinations, then you can might actually be able to do an exhaustive test. 2^32 microseconds = 71 minutes. Work in C++, run on all threads, and you probably won't even need to wait that long. Good example: https://randomascii.wordpress.com/2014/01/27/theres-only-four-billion-floatsso-test-them-all/ https://randomascii.wordpress.com/2014/01/27/theres-only-fou... - runs in 90 seconds. I did something similar a few years ago, prototyping a 16x16 multiply routine for an 8-bit CPU. It took about 5 minutes running on both threads on my 2.5GHz dual core laptop, and found some interesting cases I'd got completely wrong.
- Kenji 8y ago
  Yes, that's a good point. I unit test all small functions that take no more than 32 bit in input on literally all inputs.
  - deleted 8y ago
    [deleted]
- kardos 8y ago
  Exhaustive search is untenable for the 64-bit case. Any strategy here? Testing 2^40 bits might be tenable; any way to select which ones would have the best ROI?
  - mehrdadn 8y ago
    It's hard to give a general strategy without knowing the problem structure, but I guess you could say one general strategy is to exploit the problem structure in a way that lets you test multiple values in one shot.
  - 8y ago
afwaller 8y ago
There are only two hard problems in computer science. Cache invalidation, and naming things. And off by one errors.
- drharby 8y ago
  What you did there, I see
- progval 8y ago
  And off by 2^32 errors
  - taneq 8y ago
    Don't think I've ever heard of off-by-0 errors before. Doesn't sound that bad...
- jstimpfle 8y ago
  And asynchronous callbacks.
  - quickthrower2 8y ago
    Why is that a hard problem? (Once someone answers this, I promise to read and process the answer, but I'll go and do something else in the meantime)
    - ibotty 8y ago
      The joke is, that this answer came late.
- Scarblac 8y ago
  That's the difference between computer science and software engineering. Software engineering has only a single hard problem: people.
quotemstr 8y ago
I've been taking a pretty close look at Linux's mm code recently, and I've been wondering why it needs to be as complex as it is. There's a brain-exploding amount of complexity, most of it open-coded, that seems to my eye to be unnecessary, or at least amenable to abstraction. For example: 1) Why do we have entirely different management structures for anonymous memory (anon_vma and friends) and shared memory (inodes) when they really end up doing the same job? (Anonymous memory can be shared, so both paths end up needing to do the same kind of thing.) ISTM we can just use shared objects in all cases, tweaking the semantics as needed for shared memory. Anonymous memory is just swap-backed shared memory, after all. It should use the same logic. (Yes, you need to work without a swapfile. No, that doesn't change the conceptual model.) 2) Do we really need open-coded page table walking? Why duplicate essentially the same logic four times? Sure, you occasionally want to do slightly different things at each page table level, but you can still unify most of the logic. (If you really want, you can encode the per-level differences in code generated via C++ template.) 3) Do micro-optimizations of the sort mentioned in the article really help? If these sequence numbers had been 64 bits long, they wouldn't have overflowed. If the VMA tree had been implemented as a splay tree (like on NT) instead of an RB tree with a per-thread cache, the per-thread cache might not have been needed. (Splay trees automatically cache recently-accessed nodes.) How much do these little tricks actually help? Is their cumulative effect positive? 4) Why is so much of the vm internal logic spread throughout the kernel? Both NT and the BSDs have a well-defined API that separates the MM subsystem from the rest of ring 0, but on Linux, ISTM that lots of code needs to care directly about low-level mm structures and locks. Why should code very far from the mm core (like the i915 driver) take mmap_sem directly? It feels like more abstraction would be possible here. I get arguments based around ruthless pursuit of performance, but with function calls taking nanoseconds and page faults taking orders of magnitude more time, is anyone saving anything significant?
- jacquesm 8y ago
  If you strip out all performance optimizations from modern computers they will be a lot more secure but they will run slow as molasses. Starting with the hardware caches and ending with optimizations like these we are not talking 'minor gains' but likely orders of magnitude difference between optimized and non-optimized. And all of those have security trade-offs. It's a tough call, the mantra used to be 'get it right, then make it fast', but in practice whoever ships the fastest stuff will win the race, even if it is incorrect. As long as it is correct long enough to run the benchmarks I guess. This is yet another reason why I love microkernels (real ones), they are a lot easier to reason about and to get right to the point that you can really rely on them not to suddenly exhibit some weird behavior due to the complexity of their execution context.
steelframe 8y ago
Note this is by Jann Horn, the same Google security researcher who discovered Spectre/Meltdown. Serious props to this guy, and hopefully he stays in a whitehat role where he can find and deal with these issues through responsible disclosure.
- ronnier 8y ago
  Wow he’s only 22 https://www.bloomberg.com/news/articles/2018-01-17/how-a-22-year-old-discovered-the-worst-chip-flaws-in-history https://www.bloomberg.com/news/articles/2018-01-17/how-a-22-...
  - merb 8y ago
    I mean with 20 (while still in colleague), cure53 got already curious about Horn, which is quite crazy. (cure53 is one of german's best penetration/security company, they even analayze your software as a service, they actually pentested curl: https://cure53.de/pentest-report_curl.pdf https://cure53.de/pentest-report_curl.pdf, prometheus: https://cure53.de/pentest-report_prometheus.pdf https://cure53.de/pentest-report_prometheus.pdf and probably other high valuable software)
  - romed 8y ago
    Perhaps people new to the codebase (or the industry, or to life in general) are less likely to view existing constructs as obviously correct, and therefore more likely to point out flaws.
    - quotemstr 8y ago
      That's one reason big tech companies should allow easy inter-team transfers.
      - romed 8y ago
        Also why you should give all the critical projects to interns.
    - 8y ago
alain94040 8y ago
The bug was fixed by changing the sequence numbers to 64 bits, thereby making an overflow infeasible, and removing the overflow handling logic. So the bug is still there then? It will just take longer to hit. Until someone else decides to change the encoding of that field, for some reason, and not initialize it at 0. It instead starts with a random value (for safety reasons). Good luck figuring out that random crash that happens once in a blue moon.
- jacquesm 8y ago
  > So the bug is still there then? It will just take longer to hit. It will take so long that the hardware will have died many times over before the bug triggers, assuming society as we know it is still around.
  - alain94040 8y ago
    Like I said in my original comment, not if a future kernel change randomized that value to make it unpredictable to attacks. Then the bug will be back.
    - deleted 8y ago
      [deleted]
    - winstonewert 8y ago
      The bug doesn't trigger when the counter resets to zero, the bug triggers when the counter reuses a previous value. So: A) There is no reason whatsoever to randomize the counter B) Even if they did, it wouldn't be a problem because it would still take an absurdly long time to loop around to the starting positions.
kolderman 8y ago
Sounds like a classic case of "not thinking things through properly at the beginning". I wonder how freeBSD engineered this...Linux appears to grow organically at times.
- markjdb 8y ago
  You could say that about most bugs... FreeBSD doesn't have a per-thread cache, but does similarly use a generation number to allow operations to detect stale state upon reacquiring the vm map lock. I don't see why it isn't prone to the same kinds of bugs.
  - kolderman 8y ago
    I entirely disagree. You have have the most well thought out, architected, designed system in the world with: if (i=0) {
    - markjdb 8y ago
      You could argue that a system written in a language that permits such errors is not well-thought out. The terms you're using are not well-defined, so it's easy to disagree endlessly without arriving at a useful insight.
zydeco 8y ago
I can see why they haven't come up with a catchy name for it yet.
- db48x 8y ago
  It should obviously be The Mummy's Curse, because of all the wrappings.
misiti3780 8y ago
Serious question - where would you start if you wanted to learn about how to do find/diagnose this types of issues and maybe one day work for project zero. Everytime I see these posts it blows my mind, maybe because I have never done any kernel development, but I tend to get the feeling these guys/girls must be the best of the best.
hi41 8y ago
This is such fabulous work! Sometimes I look at such work with a tinge of jealousy. How does one get good so good at programming. I try but I cannot get to be this good. I especially find it hard to understand someone else's code.

312 points by known 8y ago

"This fits in with the kernel's policy of attempting to keep the system running as much as possible by default; if an accidental use-after-free bug occurs here for some reason, the kernel can probably heuristically mitigate its effects and keep the process working." That's a pretty bad design decision. I'd much rather have a kernel panic than a kernel that continues to run with known bad datastructures. Like that the bugs will never really shake out, silent failures like that are super dangerous because essentially the system is running in an undefined state past that point.

quotemstr 8y ago
I agree. Efforts to continue past unexpected states frequently just make the situation worse and end up causing more user pain than a deterministic restart. This kind of recovery also masks real bugs and leads to their persistence in the codebase. (Heuristic error recovery also makes debugging more difficult.) A fail-fast approach both preserves user invariants and provides an impetus to fix bugs fast.
- xyproto 8y ago
  To reboot when this happen could be a kernel configuration option. This way users could choose.
NightMKoder 8y ago
IMO - it's actually a really hard call. Error isolation is hard but we all want it. One of the somewhat less controversial versions is failing a single user request (in node or your web framework) with a 5xx on an exception. You're essentially betting that the exception is a logic bug and not somewhere that will affect other users. That's probably true, unless, you know, your database driver is responsible. We bet that's usually not the case - and we're mostly right. Honestly I'm ok with trying to keep the system limping with a huge but - you must, at the point you first detect an error, dump...everything you know. You can try to limp along because you're trying to be a good host, but debugging after that state is, as you said, not trivial. If you separate the two concerns (post-mortem debugging & uptime) there's a nice medium to be found. Ideally kernel panics aren't the only source of observability. You can have a daemon running that files a bug report to your favorite error tracker (sentry, etc) and (attempts to) gracefully reboot the system. That would be pretty sweet.
- jacquesm 8y ago
  Erlang is the only environment that I know about that gets this right.
tonysdg 8y ago
It's terrible from a correctness perspective, sure. But from a business perspective, that could mean $50 million of revenue not-lost to a flaw you (as an owner/operator) can't do much about. Sure, it could lead to an exploit like this, but I'd wager a cost-benefit analysis for almost any organization (except maybe CIA/NSA types) would support this design choice. That's not to say it's a good design choice, but it's certainly a defensible one IMO. You can have the most secure OS in the world, but if no one wants to use it, all you've got is a very secure waste of hard drive space.
- jacquesm 8y ago
  If a kernel panic can cost you $50 million you have other problems. Really, in an organization where downtime due to a server rebooting would be that expensive you'd hope they would be able to deal with that gracefully and that they would be deploying the rough equivalent of chaosmonkey to ensure that their stuff is protected against such errors. After all, a harddrive or a CPU could die just the same.
  - sangnoir 8y ago
    Your implicit assumption is that the server will only reboot once. I think if the Linux kernel is upfront about its security posture and your threat-model is incompatible, it is OK to go with OpenBSD.
    - jacquesm 8y ago
      The more frequent the reboots the quicker the problem will get diagnosed! Especially if the system is still stable enough to write a log entry to disk.
  - ploxiln 8y ago
    This is a principle borne of experience. There have been a number of cases where a new fatal-error check killed a bunch of systems that apparently were working for a long time before. Usually it is preferred to WARN_ONCE() but not BUG_ON(). Otherwise, big corporate users, or a wide random assortment of desktop users, end up going back to an older kernel to make their systems work. The linux kernel tries above all to never have regressions, even if these regressions are just catching probably very bad bugs that were just not caught before. This happens, often enough! Think of this from the lense of natural selection. There are many many subtle tricky low-level bugs that can result in memory corruption - many drivers, many features, many optimizations, many "cleanups", a surprisingly high rate of code changes. Many of them are caught when they cause problematic corruption. Others, just due to luck, very rarely cause any visible problem, - for months or years. We do want to know about them. We do not want to suicide the whole system, it may be one of those rare bugs that made it this far because it's usually (surprisingly) not fatal on its own.
palotasb 8y ago
IIUC the heuristic here is how much of the system to take down (i.e., back out of fast path [this case] < kill thread < kill process < kernel panic) and this seems correct (but could still be a symptom of more corruption). Anyway, there has been discussion about this issue in general previously on the Linux kernel mailing list, and Linus said [1] that the correct procedure is to first introduce by-default reporting and an opt-in kill switch, then make the kill switch the default, then remove the non-kill option. This is supposed to weed out bugs eventually without disrupting users too much, but I can imagine that it enables some expolits. There was an HN discussion about it too. [2] [1] https://lkml.org/lkml/2017/11/21/356 https://lkml.org/lkml/2017/11/21/356 [2] https://news.ycombinator.com/item?id=15754988 https://news.ycombinator.com/item?id=15754988
amaccuish 8y ago
> I'd much rather have a kernel panic than a kernel that continues to run with known bad datastructures I mean, that is literally what an Oops is.
pcwalton 8y ago
It's extremely useful for developing drivers. I've developed drivers on kernels that are hardwired to panic at the slightest problem and it's a real nuisance. Whether you should leave it on in production is debatable, but I like at least having the option of making exceptions raised in drivers nonfatal.
- jacquesm 8y ago
  And the reason for that is that these drivers are linked into the monolith. If they were standalone processes you'd love the ability to do post-mortem debugging on them, or maybe even the luxury of being able to run them directly under the debugger. Whenever I had to write device drivers I wrote them under QnX first and only then ported them to other OS's, that saved so much time, at least I knew the hardware interface would be up and running, and the data structures would all work as intended. After that all I then had to do was glue it to whatever calling convention the various *NIX flavors had. That trick served me well, for motion capture devices as well as for high speed serial cards and some more exotic devices.
  - im_down_w_otp 8y ago
    This is fairly analogous to our approach too, just substituting QNX w/ seL4.

tom_ 8y ago

4 billion was a lot when I first got into programming, but it just isn't that much any more. Always start out by making your sequence numbers 64 bits: 2^64 nanoseconds = ~585 years. (I have in the past stolen a few bits for other uses, but I'm less certain of the wisdom of that these days.) 32 bits is suitable for things that can only happen once a frame - (2^32/60) seconds = 2.26 years - but not much more than that. But there's a flip side to computers being able to wrap a uint32_t so quickly: if you've got 32 bits'-worth of combinations, then you can might actually be able to do an exhaustive test. 2^32 microseconds = 71 minutes. Work in C++, run on all threads, and you probably won't even need to wait that long. Good example: https://randomascii.wordpress.com/2014/01/27/theres-only-four-billion-floatsso-test-them-all/ https://randomascii.wordpress.com/2014/01/27/theres-only-fou... - runs in 90 seconds. I did something similar a few years ago, prototyping a 16x16 multiply routine for an 8-bit CPU. It took about 5 minutes running on both threads on my 2.5GHz dual core laptop, and found some interesting cases I'd got completely wrong.

Kenji 8y ago
Yes, that's a good point. I unit test all small functions that take no more than 32 bit in input on literally all inputs.
- deleted 8y ago
  [deleted]
kardos 8y ago
Exhaustive search is untenable for the 64-bit case. Any strategy here? Testing 2^40 bits might be tenable; any way to select which ones would have the best ROI?
- mehrdadn 8y ago
  It's hard to give a general strategy without knowing the problem structure, but I guess you could say one general strategy is to exploit the problem structure in a way that lets you test multiple values in one shot.
- 8y ago

afwaller 8y ago

There are only two hard problems in computer science. Cache invalidation, and naming things. And off by one errors.

drharby 8y ago
What you did there, I see
progval 8y ago
And off by 2^32 errors
- taneq 8y ago
  Don't think I've ever heard of off-by-0 errors before. Doesn't sound that bad...
jstimpfle 8y ago
And asynchronous callbacks.
- quickthrower2 8y ago
  Why is that a hard problem? (Once someone answers this, I promise to read and process the answer, but I'll go and do something else in the meantime)
  - ibotty 8y ago
    The joke is, that this answer came late.
Scarblac 8y ago
That's the difference between computer science and software engineering. Software engineering has only a single hard problem: people.

quotemstr 8y ago

I've been taking a pretty close look at Linux's mm code recently, and I've been wondering why it needs to be as complex as it is. There's a brain-exploding amount of complexity, most of it open-coded, that seems to my eye to be unnecessary, or at least amenable to abstraction. For example: 1) Why do we have entirely different management structures for anonymous memory (anon_vma and friends) and shared memory (inodes) when they really end up doing the same job? (Anonymous memory can be shared, so both paths end up needing to do the same kind of thing.) ISTM we can just use shared objects in all cases, tweaking the semantics as needed for shared memory. Anonymous memory is just swap-backed shared memory, after all. It should use the same logic. (Yes, you need to work without a swapfile. No, that doesn't change the conceptual model.) 2) Do we really need open-coded page table walking? Why duplicate essentially the same logic four times? Sure, you occasionally want to do slightly different things at each page table level, but you can still unify most of the logic. (If you really want, you can encode the per-level differences in code generated via C++ template.) 3) Do micro-optimizations of the sort mentioned in the article really help? If these sequence numbers had been 64 bits long, they wouldn't have overflowed. If the VMA tree had been implemented as a splay tree (like on NT) instead of an RB tree with a per-thread cache, the per-thread cache might not have been needed. (Splay trees automatically cache recently-accessed nodes.) How much do these little tricks actually help? Is their cumulative effect positive? 4) Why is so much of the vm internal logic spread throughout the kernel? Both NT and the BSDs have a well-defined API that separates the MM subsystem from the rest of ring 0, but on Linux, ISTM that lots of code needs to care directly about low-level mm structures and locks. Why should code very far from the mm core (like the i915 driver) take mmap_sem directly? It feels like more abstraction would be possible here. I get arguments based around ruthless pursuit of performance, but with function calls taking nanoseconds and page faults taking orders of magnitude more time, is anyone saving anything significant?

jacquesm 8y ago
If you strip out all performance optimizations from modern computers they will be a lot more secure but they will run slow as molasses. Starting with the hardware caches and ending with optimizations like these we are not talking 'minor gains' but likely orders of magnitude difference between optimized and non-optimized. And all of those have security trade-offs. It's a tough call, the mantra used to be 'get it right, then make it fast', but in practice whoever ships the fastest stuff will win the race, even if it is incorrect. As long as it is correct long enough to run the benchmarks I guess. This is yet another reason why I love microkernels (real ones), they are a lot easier to reason about and to get right to the point that you can really rely on them not to suddenly exhibit some weird behavior due to the complexity of their execution context.

steelframe 8y ago

Note this is by Jann Horn, the same Google security researcher who discovered Spectre/Meltdown. Serious props to this guy, and hopefully he stays in a whitehat role where he can find and deal with these issues through responsible disclosure.

ronnier 8y ago
Wow he’s only 22 https://www.bloomberg.com/news/articles/2018-01-17/how-a-22-year-old-discovered-the-worst-chip-flaws-in-history https://www.bloomberg.com/news/articles/2018-01-17/how-a-22-...
- merb 8y ago
  I mean with 20 (while still in colleague), cure53 got already curious about Horn, which is quite crazy. (cure53 is one of german's best penetration/security company, they even analayze your software as a service, they actually pentested curl: https://cure53.de/pentest-report_curl.pdf https://cure53.de/pentest-report_curl.pdf, prometheus: https://cure53.de/pentest-report_prometheus.pdf https://cure53.de/pentest-report_prometheus.pdf and probably other high valuable software)
- romed 8y ago
  Perhaps people new to the codebase (or the industry, or to life in general) are less likely to view existing constructs as obviously correct, and therefore more likely to point out flaws.
  - quotemstr 8y ago
    That's one reason big tech companies should allow easy inter-team transfers.
    - romed 8y ago
      Also why you should give all the critical projects to interns.
  - 8y ago

alain94040 8y ago

The bug was fixed by changing the sequence numbers to 64 bits, thereby making an overflow infeasible, and removing the overflow handling logic. So the bug is still there then? It will just take longer to hit. Until someone else decides to change the encoding of that field, for some reason, and not initialize it at 0. It instead starts with a random value (for safety reasons). Good luck figuring out that random crash that happens once in a blue moon.

jacquesm 8y ago
> So the bug is still there then? It will just take longer to hit. It will take so long that the hardware will have died many times over before the bug triggers, assuming society as we know it is still around.
- alain94040 8y ago
  Like I said in my original comment, not if a future kernel change randomized that value to make it unpredictable to attacks. Then the bug will be back.
  - deleted 8y ago
    [deleted]
  - winstonewert 8y ago
    The bug doesn't trigger when the counter resets to zero, the bug triggers when the counter reuses a previous value. So: A) There is no reason whatsoever to randomize the counter B) Even if they did, it wouldn't be a problem because it would still take an absurdly long time to loop around to the starting positions.

kolderman 8y ago

Sounds like a classic case of "not thinking things through properly at the beginning". I wonder how freeBSD engineered this...Linux appears to grow organically at times.

markjdb 8y ago
You could say that about most bugs... FreeBSD doesn't have a per-thread cache, but does similarly use a generation number to allow operations to detect stale state upon reacquiring the vm map lock. I don't see why it isn't prone to the same kinds of bugs.
- kolderman 8y ago
  I entirely disagree. You have have the most well thought out, architected, designed system in the world with: if (i=0) {
  - markjdb 8y ago
    You could argue that a system written in a language that permits such errors is not well-thought out. The terms you're using are not well-defined, so it's easy to disagree endlessly without arriving at a useful insight.

zydeco 8y ago

I can see why they haven't come up with a catchy name for it yet.

db48x 8y ago
It should obviously be The Mummy's Curse, because of all the wrappings.

misiti3780 8y ago

Serious question - where would you start if you wanted to learn about how to do find/diagnose this types of issues and maybe one day work for project zero. Everytime I see these posts it blows my mind, maybe because I have never done any kernel development, but I tend to get the feeling these guys/girls must be the best of the best.

hi41 8y ago

This is such fabulous work! Sometimes I look at such work with a tinge of jealousy. How does one get good so good at programming. I try but I cannot get to be this good. I especially find it hard to understand someone else's code.