10 ms·

The moment I read 'it is a content update that causes the BSOD, deleting it solves the problem', I was immediately willing to bet a hundred quid (for the non-Br

The moment I read 'it is a content update that causes the BSOD, deleting it solves the problem', I was immediately willing to bet a hundred quid (for the non-British, that's £100) that it was a combination of said bad binary data and a poorly-written parser that didn't error out correctly upon reading invalid data (in this case, read an array of pointers, didn't verify that all of them were both non-null and pointed to valid data/code). In the past ten years or so of having done somewhat serious computing and zero cybersecurity whatsoever, I have my mind concluded, feel free to disagree. Approximately 100% of CVEs, crashes, bugs, slowdowns, and pain points of computing have to do with various forms of deserialising binary data back into machine-readable data structures. All because a) human programmers forget to account for edge cases, and b) imperative programming languages allow us to do so. This includes everything from: decompression algorithms; font outline readers; image, video, and audio parsers; video game data parsers; XML and HTML parsers; the various certificate/signature/key parsers in OpenSSL (and derivatives); and now, this CrowdStrike content parser in its EDR program. That wager stands, by the way, and I'm happy to up the ante by £50 to account for my second theory.

bostik 2y ago
> Approximately 100% of CVEs, crashes, bugs, [...], deserialising binary data I'd make that 98%. Outside of rounding errors in the margins, the remaining two percent is made up of logic bugs, configuration errors, bad defaults, and outright insecure design choices. Disclosure: infosec for more than three decades.
- epanchin 2y ago
  They forgot to account for those edge cases
  - delta_p_delta_x 2y ago
    Heh, touché.
- delta_p_delta_x 2y ago
  I feel vindicated but also a bit surprised that my gut feeling was this accurate.
  - bostik 2y ago
    Not really a surprise, to be honest. "Deserialisation" encapsulates most forms of injection attacks. OWASP top-10 was dominated by those for a very long time. They have only recently been overtaken by authorization failures.
smackeyacky 2y ago
Hmmm. Most common problems these days are certificate related I would have thought. Binary data transfers are pretty rare in an age of base64 json bloat
- madaxe_again 2y ago
  There are plenty of binary serialisation protocols out there, many proprietary - maybe you’ll stuff that base64’d in a json container for transit, but you’re still dealing with a binary decoder.
Sakos 2y ago
I can't decide what's more damning. The fact that there was effectively no error/failure handling or this: > Note "channel updates ...bypassed client's staging controls and was rolled out to everyone regardless" > A few IT folks who had set the CS policy to ignore latest version confirmed this was, ya, bypassed, as this was "content" update (vs. a version update) If your content updates can break clients, they should not be able to bypass staging controls or policies.
- vladvasiliu 2y ago
  The way I understand it, the policy the users can configure are about "agent versions". I don't think there's a setting for "content versions" you can toggle.
  - sateesh 2y ago
    Maybe there isn't a switch that says "content version",but from end user perspective it is a new version. Whether it was a content change, or just a fix for typo in documentation (say) the change being pushed is different than what currently exists.And for the end user the configuration implies that they have a chance to decide whether to accept any new change being pushed or not.
- SoftTalker 2y ago
  > If your content updates can break clients This is going to be what most customers did not realize. I'm sure Crowdstrike assured them that content updates were completely safe "it's not a change to the software" etc. Well they know differently now.
miohtama 2y ago
I was immediately willing to bet a hundred quid this was C/C++ code :)
- formerly_proven 2y ago
  Not that interesting a bet considering we know it's a Windows driver.
fire_lake 2y ago
Yes indeed. If you are doing this kind of job, reach for a parser generator framework and fuzz your program. Also go read Parse Don’t Validate https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
- teeheelol 2y ago
  Yep. Looking at how this whole thing is pasted together, there's probably a regex engine in one of those sys files somewhere that was doing the "parsing"...
- lolinder 2y ago
  > reach for a parser generator framework and fuzz your program I agree to the second but disagree on the first. Parser generator frameworks produce a lot of code that is hard to read and understand and they don't necessarily do a better job of error handling than you would. A hand-written recursive descent parser will usually be more legible, will clearly line up with the grammar that you're supposed to be parsing, and will be easier to add better error handling to. Once you're aware of the risks of a bad parser you're halfway there. Write a parser with proper parsing theory in mind and in a language that forces you to handle all cases. Then fuzz the program, turn bad inputs that turn up into permanent regression tests, and write your own tests with your knowledge of the inner workings of your parser in mind. This isn't like rolling your own crypto because the alternative isn't a battle-tested open source library, it's a framework that generates a brand new library that only you will use and maintain. If you're going to end up with a bespoke library anyway, you ought to understand it well.
  - fire_lake 2y ago
    You place your trust in: 1. The battle testing of the generator (e.g. lex yacc is widely used) 2. The readability of the grammar file, which enables good review You don’t (often) read the generated code, only test it. Much like you wouldn’t read the binary generated by a compiler.
mtlmtlmtlmtl 2y ago
There's at least five different things that went wrong simultaneously. 1. Poorly written code in the kernel module crashed the whole OS, and kept trying to parse the corrupted files, causing a boot loop. Instead of handling the error gracefully and deleting/marking the files as corrupt. 2. Either the corrupted files slipped through internal testing, or there is no internal testing. 3. Individual settings for when to apply such updates were apparently ignored. It's unclear whether this was a glitch or standard practice. Either way I consider it a bug(it's just a matter of whether it's a software bug or a bug in their procedures). 4. This was pushed out everywhere simultaneously instead of staggered to limit any potential damage. 5. Whatever caused the corruption in the first place, which is anyone's guess.
- simonh 2y ago
  There is a story out that the problem was introduced in a post processing step after testing. That makes more sense than that there was no testing. If true it means they thought they’d tested the update, but actually hadn’t.
- rwmj 2y ago
  Zero effort to fuzz test the parser too. I mean, we know how to harden parsers against bugs and attacks, and any semi-competent fuzzer would have caught such a trivial bug.
  - chrisjj 2y ago
    The triggering file was all zeros. It is not possible that only this pattern caused the crash, and fuzzing omitted to try this unfuzzy pattern?
    - monsieurbanana 2y ago
      In my limited experience, I thought any serious fuzzing program does test for all "standard" patters like only null bytes, empty strings, etc...
    - gliptic 2y ago
bradley13 2y ago
No bet. There are two failures here. (1) Failing to check the data for validity, and (2) Failing to handle an error gracefully. Both of these are undergraduate-level techniques. Heck, they are covered in most first-semester programming courses. Either of these failures is inexcusable in a professional product, much less one that is running with kernel-level privileges. Bet: CrowdStrike has outsourced much of its development work.
- ahoka 2y ago
  What do you mean by outsourced?
  - Rinzler89 2y ago
    He probably means work was sent offshore to offices with cheaper labor that's less skilled or less vested into delivering quality work. Though there's no proof of that yet, people just like to throw the blame on offshoring whoever $BIG_CORP fucks up, as if all programmers in the US are John Carmack and they can never cause catastrophic fuckups with their code or processes.
    - jojobas 2y ago
      Not everyone in the US might be Carmack, but it's ridiculously nearsighted to assert that cultural differences don't play into people desire and ability to Do It Right.
      - Rinzler89 2y ago
        It's not cultural differences that make the difference in output quality, it's pay and quality standards of the output set by the team/management, which is also mostly a function of pay, since underpaid and unhappy developers tend not to care at all beyond doing the bare minimum to not getting fired (#notmyjob, laying flat movement, etc). You think everyone writing code in the US would give two shits about the quality of their output if they see the CEO pocketing another private jet while they can barley make big-city rent? Hell, even well paid devs at top companies in the US can be careless and lazy if their company doesn't care about quality. Have you seen some of the vulnerabilities and bugs that make it into the Android source code and on Pixel devices? And guess what, that code was written by well paid developers in the US, hired at Google leetcode standards, yet would give far-east sweatshops a run for their money in terms of carelessness. It's what you get when you have a high barrier of entry but a low barrier of output quality where devs just care about "rest and vest".
variadix 2y ago
More or less. Binary parsers are the easiest place to find exploits because of how hard it is to do correctly. Bounds checks, overflow checks, pointer checks, etc. Especially when the data format is complicated.
- hnthrowaway0328 2y ago
  Is there any reading about this topic? By saying binary parsing I guess you meant code that parses say PNG or WAD files?
praptak 2y ago
> imperative languages allow us to do so This problem has a promising solution, WUFFS, "a memory-safe programming language (and a standard library written in that language) for Wrangling Untrusted File Formats Safely." HN discussion: https://news.ycombinator.com/item?id=40378433 https://news.ycombinator.com/item?id=40378433 HN discussion of Wuffs implementation of PNG parser: https://news.ycombinator.com/item?id=26714831 https://news.ycombinator.com/item?id=26714831
noobermin 2y ago
So, I also have near zero cybersecurity expertise (I took an online intro course on cryptography due to curiousity) and no expertise in writing kernel modules actually, but why if ever would you parse an array of pointers...in a file...instead of any other way of serializing data that doesn't include hardcoded array offsets in an on-disk file... Ignore this failure which was catastrophic, this was a bad design asking to be exploited by criminals.
- Jare 2y ago
  Performance, I assume. Right now it may look like the wrong tradeoff, but every day in between incidents like this we're instead complaining that software is slow. Of course it doesn't have to be either/or; you can have fast + secure, but it costs a lot more to design, develop, maintain and validate. What you can't have is a "why don't they just" simple and obvious solution that makes it cheap without making it either less secure, less performant, or both. Given all the other mishaps in this story, it is very well possible that the software is insecure (we know that), slow and also still very expensive. There's a limit to how high you can push the triangle, but there's not bottom to how bad it can get.
- deaddodo 2y ago
  I'm curious, how else would you store direct memory offsets? No matter how you store/transmit them, eventually you're going to need those same offsets. The problem wasn't storing raw memory offsets, it was not having some way to validate the data at runtime.
  - noobermin 2y ago
    You don't store direct memory offsets if possible. You have a system of handles and store the handle keys instead ofndirect pointer addresses. Yes you still need to validate the keys to avoid logic errors, but you can avoid the memory errors.
    - deaddodo 2y ago
      In this case, the direct memory addresses are literally needed. The addresses aren't being generated internal to the program, so there are no "handles". They are referencing external data by design. That's like saying "you shouldn't use a hard-coded volatile pointer to reference a hardware device". No, you literally need to do that sometimes; especially in embedded software.
lol768 2y ago
> I'm happy to up the ante by £50 to account for my second theory What's that, three pints in a pub inside the M25? :P Completely agree with this sentiment though, we've known that handling of binary data in memory unsafe languages has been risky for yonks. At the very least, fuzzing should've been employed here to try and detect these sorts of issues. More fundamentally though, where was their QA? These "channel files" just went out of the door without any idea as to their validity? Was there no continuous integration check to just .. ensure they parsed with the same parser as was deployed to the endpoints? And why were the channel files not deployed gradually?
- TeMPOraL 2y ago
  FWIW, before someone brings up JSON, GP's bet only makes sense when "binary" includes parsing text as well. In fact, most notorious software bugs are related to misuse of textual formats like SQL or JS.
1992spacemovie 2y ago
Interesting observation. As a non-developer, what can one do to enhance coverage for these types of scenerios? Fuzz testing?
- rwmj 2y ago
  Fuzz testing absolutely should be used whenever you parse anything.
  - SoftTalker 2y ago
    Yeah, even if you are only parsing "safe" inputs such as ones you created yourself. Other bugs and sometimes even truly random events can corrupt data.
throw0101d 2y ago
> Approximately 100% of CVEs, crashes, bugs, slowdowns, and pain points of computing have to do with various forms of deserialising binary data back into machine-readable data structures. For the record, the top 25 common weaknesses for 2023 are listed at: * https://cwe.mitre.org/top25/archive/2023/2023_top25_list.html https://cwe.mitre.org/top25/archive/2023/2023_top25_list.htm... Deserialization of Untrusted Data (CWE-502) was number fifteen. Number one was Out-of-bounds Write (CWE-787), Use After Free (CWE-416) was number four. CWEs that have been in every list since they started doing this (2019): * https://cwe.mitre.org/top25/archive/2023/2023_stubborn_weaknesses.html https://cwe.mitre.org/top25/archive/2023/2023_stubborn_weakn...
- lioeters 2y ago
  # Top Stubborn Software Weaknesses (2019-2023) Out-of-bounds Write Improper Neutralization of Input During Web Page Generation (‘Cross-site Scripting’) Improper Neutralization of Special Elements used in an SQL Command (‘SQL Injection’) Use After Free Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') Improper Input Validation Out-of-bounds Read Improper Limitation of a Pathname to a Restricted Directory (‘Path Traversal’) Cross-Site Request Forgery (CSRF) NULL Pointer Dereference Improper Authentication Integer Overflow or Wraparound Deserialization of Untrusted Data Improper Restriction of Operations within Bounds of a Memory Buffer Use of Hard-coded Credentials
  - TeMPOraL 2y ago
    Yup. Almost all of them are various flavor of fucking up a parser or misusing it (in particular, all the injection cases are typically caused by writing stupid code that glues strings together instead of proper parsing).
    - lolinder 2y ago
      That's not parsing, that's the inverse of parsing. It's taking untrusted data and injecting it into a string that will later be parsed into code without treating the data as untrusted and adapting accordingly. It's compiling, of a sort. Parsing is the reverse—taking an untrusted string (or binary string) that is meant to be code and converting it into a data structure. Both are the result of taking untrusted data and assuming it'll look like what you expect, but both are not parsing issues.
eru 2y ago
> Approximately 100% of CVEs, crashes, bugs, slowdowns, and pain points of computing have to do with various forms of deserialising binary data back into machine-readable data structures. All because a) human programmers forget to account for edge cases, and b) imperative programming languages allow us to do so. I wouldn't blame imperative programming. Eg Rust is imperative, and pretty good at telling you off when you forgot a case in your switch. By contrast the variant of Scheme I used twenty years ago was functional, but didn't have checks for covering all cases. (And Haskell's ghc didn't have that checked turned on by default a few years ago. Not sure if they changed that.)
seymore_12 2y ago
>Approximately 100% of CVEs, crashes, bugs, slowdowns, and pain points of computing have to do with various forms of deserialising binary data back into machine-readable data structures. All because a) human programmers forget to account for edge cases, and b) imperative programming languages allow us to do so. This. One year ago UK air traffic control collapsed due to inability to properly parse "faulty" flight plan: https://news.ycombinator.com/item?id=37461695 https://news.ycombinator.com/item?id=37461695
cedws 2y ago
I’d say that it is a bug by definition if your program ungracefully crashes when it’s passed malformed data at runtime.
stefan_ 2y ago
People are target fixating too much. Sure, this parser crashed and caused the system to go down. But in an alternative universe they push a definition file that rejects every openat() or connect() syscall. Your system is now equally as dead, except it probably won't even have the grace to restart. The whole concept of "we fuck with the system in kernel based on data downloaded from the internet" is just not very sound and safe.
- hello_moto 2y ago
  It's not and that's the sad state of AV in Windows
xxs 2y ago
>(for the non-British, that's £100) next time you'd be adding /s to your posts
back_to_basics 2y ago
"human programmers forget to account for edge cases" Which is precisely the rationale which led to Standard Operating Procedures and Best Practices (much like any other Sector of business has developed). I submit to you, respectfully, that a corporation shall never rise to a $75 Billion Market Cap without a bullet-proof adherence to such, and thus, this "event" should be properly characterized and viewed as a very suspicious anomaly, at the least https://news.ycombinator.com/item?id=41023539 https://news.ycombinator.com/item?id=41023539 fleshes out the proper context.
divan 2y ago
Related talk: 28c3: The Science of Insecurity (2011) https://www.youtube.com/watch?v=3kEfedtQVOY https://www.youtube.com/watch?v=3kEfedtQVOY
- nonrandomstring 2y ago
  Excellent talk. So long ago and what since?
smsm42 2y ago
> combination of said bad binary data and a poorly-written parser that didn't error out correctly upon reading invalid data By now, if you write any parser that deals with any outside data and don't fuzz the heck out of it, you are willfully negligent. Fuzzers are pretty easy to use, automatic and would likely catch any such problem pretty soon. So, did they fuzz and got very very unlucky or do they just like to live dangerously?
Log_out_ 2y ago
Soon, update revert watchdogs

bostik 2y ago

> Approximately 100% of CVEs, crashes, bugs, [...], deserialising binary data I'd make that 98%. Outside of rounding errors in the margins, the remaining two percent is made up of logic bugs, configuration errors, bad defaults, and outright insecure design choices. Disclosure: infosec for more than three decades.

epanchin 2y ago
They forgot to account for those edge cases
- delta_p_delta_x 2y ago
  Heh, touché.
delta_p_delta_x 2y ago
I feel vindicated but also a bit surprised that my gut feeling was this accurate.
- bostik 2y ago
  Not really a surprise, to be honest. "Deserialisation" encapsulates most forms of injection attacks. OWASP top-10 was dominated by those for a very long time. They have only recently been overtaken by authorization failures.

smackeyacky 2y ago

Hmmm. Most common problems these days are certificate related I would have thought. Binary data transfers are pretty rare in an age of base64 json bloat

madaxe_again 2y ago
There are plenty of binary serialisation protocols out there, many proprietary - maybe you’ll stuff that base64’d in a json container for transit, but you’re still dealing with a binary decoder.

Sakos 2y ago

I can't decide what's more damning. The fact that there was effectively no error/failure handling or this: > Note "channel updates ...bypassed client's staging controls and was rolled out to everyone regardless" > A few IT folks who had set the CS policy to ignore latest version confirmed this was, ya, bypassed, as this was "content" update (vs. a version update) If your content updates can break clients, they should not be able to bypass staging controls or policies.

vladvasiliu 2y ago
The way I understand it, the policy the users can configure are about "agent versions". I don't think there's a setting for "content versions" you can toggle.
- sateesh 2y ago
  Maybe there isn't a switch that says "content version",but from end user perspective it is a new version. Whether it was a content change, or just a fix for typo in documentation (say) the change being pushed is different than what currently exists.And for the end user the configuration implies that they have a chance to decide whether to accept any new change being pushed or not.
SoftTalker 2y ago
> If your content updates can break clients This is going to be what most customers did not realize. I'm sure Crowdstrike assured them that content updates were completely safe "it's not a change to the software" etc. Well they know differently now.

miohtama 2y ago

I was immediately willing to bet a hundred quid this was C/C++ code :)

formerly_proven 2y ago
Not that interesting a bet considering we know it's a Windows driver.

fire_lake 2y ago

Yes indeed. If you are doing this kind of job, reach for a parser generator framework and fuzz your program. Also go read Parse Don’t Validate https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...

teeheelol 2y ago
Yep. Looking at how this whole thing is pasted together, there's probably a regex engine in one of those sys files somewhere that was doing the "parsing"...
lolinder 2y ago
> reach for a parser generator framework and fuzz your program I agree to the second but disagree on the first. Parser generator frameworks produce a lot of code that is hard to read and understand and they don't necessarily do a better job of error handling than you would. A hand-written recursive descent parser will usually be more legible, will clearly line up with the grammar that you're supposed to be parsing, and will be easier to add better error handling to. Once you're aware of the risks of a bad parser you're halfway there. Write a parser with proper parsing theory in mind and in a language that forces you to handle all cases. Then fuzz the program, turn bad inputs that turn up into permanent regression tests, and write your own tests with your knowledge of the inner workings of your parser in mind. This isn't like rolling your own crypto because the alternative isn't a battle-tested open source library, it's a framework that generates a brand new library that only you will use and maintain. If you're going to end up with a bespoke library anyway, you ought to understand it well.
- fire_lake 2y ago
  You place your trust in: 1. The battle testing of the generator (e.g. lex yacc is widely used) 2. The readability of the grammar file, which enables good review You don’t (often) read the generated code, only test it. Much like you wouldn’t read the binary generated by a compiler.

mtlmtlmtlmtl 2y ago

There's at least five different things that went wrong simultaneously. 1. Poorly written code in the kernel module crashed the whole OS, and kept trying to parse the corrupted files, causing a boot loop. Instead of handling the error gracefully and deleting/marking the files as corrupt. 2. Either the corrupted files slipped through internal testing, or there is no internal testing. 3. Individual settings for when to apply such updates were apparently ignored. It's unclear whether this was a glitch or standard practice. Either way I consider it a bug(it's just a matter of whether it's a software bug or a bug in their procedures). 4. This was pushed out everywhere simultaneously instead of staggered to limit any potential damage. 5. Whatever caused the corruption in the first place, which is anyone's guess.

simonh 2y ago
There is a story out that the problem was introduced in a post processing step after testing. That makes more sense than that there was no testing. If true it means they thought they’d tested the update, but actually hadn’t.
rwmj 2y ago
Zero effort to fuzz test the parser too. I mean, we know how to harden parsers against bugs and attacks, and any semi-competent fuzzer would have caught such a trivial bug.
- chrisjj 2y ago
  The triggering file was all zeros. It is not possible that only this pattern caused the crash, and fuzzing omitted to try this unfuzzy pattern?
  - monsieurbanana 2y ago
    In my limited experience, I thought any serious fuzzing program does test for all "standard" patters like only null bytes, empty strings, etc...
  - gliptic 2y ago

bradley13 2y ago

No bet. There are two failures here. (1) Failing to check the data for validity, and (2) Failing to handle an error gracefully. Both of these are undergraduate-level techniques. Heck, they are covered in most first-semester programming courses. Either of these failures is inexcusable in a professional product, much less one that is running with kernel-level privileges. Bet: CrowdStrike has outsourced much of its development work.

ahoka 2y ago
What do you mean by outsourced?
- Rinzler89 2y ago
  He probably means work was sent offshore to offices with cheaper labor that's less skilled or less vested into delivering quality work. Though there's no proof of that yet, people just like to throw the blame on offshoring whoever $BIG_CORP fucks up, as if all programmers in the US are John Carmack and they can never cause catastrophic fuckups with their code or processes.
  - jojobas 2y ago
    Not everyone in the US might be Carmack, but it's ridiculously nearsighted to assert that cultural differences don't play into people desire and ability to Do It Right.
    - Rinzler89 2y ago
      It's not cultural differences that make the difference in output quality, it's pay and quality standards of the output set by the team/management, which is also mostly a function of pay, since underpaid and unhappy developers tend not to care at all beyond doing the bare minimum to not getting fired (#notmyjob, laying flat movement, etc). You think everyone writing code in the US would give two shits about the quality of their output if they see the CEO pocketing another private jet while they can barley make big-city rent? Hell, even well paid devs at top companies in the US can be careless and lazy if their company doesn't care about quality. Have you seen some of the vulnerabilities and bugs that make it into the Android source code and on Pixel devices? And guess what, that code was written by well paid developers in the US, hired at Google leetcode standards, yet would give far-east sweatshops a run for their money in terms of carelessness. It's what you get when you have a high barrier of entry but a low barrier of output quality where devs just care about "rest and vest".

variadix 2y ago

More or less. Binary parsers are the easiest place to find exploits because of how hard it is to do correctly. Bounds checks, overflow checks, pointer checks, etc. Especially when the data format is complicated.

hnthrowaway0328 2y ago
Is there any reading about this topic? By saying binary parsing I guess you meant code that parses say PNG or WAD files?

praptak 2y ago

> imperative languages allow us to do so This problem has a promising solution, WUFFS, "a memory-safe programming language (and a standard library written in that language) for Wrangling Untrusted File Formats Safely." HN discussion: https://news.ycombinator.com/item?id=40378433 https://news.ycombinator.com/item?id=40378433 HN discussion of Wuffs implementation of PNG parser: https://news.ycombinator.com/item?id=26714831 https://news.ycombinator.com/item?id=26714831

noobermin 2y ago

So, I also have near zero cybersecurity expertise (I took an online intro course on cryptography due to curiousity) and no expertise in writing kernel modules actually, but why if ever would you parse an array of pointers...in a file...instead of any other way of serializing data that doesn't include hardcoded array offsets in an on-disk file... Ignore this failure which was catastrophic, this was a bad design asking to be exploited by criminals.

Jare 2y ago
Performance, I assume. Right now it may look like the wrong tradeoff, but every day in between incidents like this we're instead complaining that software is slow. Of course it doesn't have to be either/or; you can have fast + secure, but it costs a lot more to design, develop, maintain and validate. What you can't have is a "why don't they just" simple and obvious solution that makes it cheap without making it either less secure, less performant, or both. Given all the other mishaps in this story, it is very well possible that the software is insecure (we know that), slow and also still very expensive. There's a limit to how high you can push the triangle, but there's not bottom to how bad it can get.
deaddodo 2y ago
I'm curious, how else would you store direct memory offsets? No matter how you store/transmit them, eventually you're going to need those same offsets. The problem wasn't storing raw memory offsets, it was not having some way to validate the data at runtime.
- noobermin 2y ago
  You don't store direct memory offsets if possible. You have a system of handles and store the handle keys instead ofndirect pointer addresses. Yes you still need to validate the keys to avoid logic errors, but you can avoid the memory errors.
  - deaddodo 2y ago
    In this case, the direct memory addresses are literally needed. The addresses aren't being generated internal to the program, so there are no "handles". They are referencing external data by design. That's like saying "you shouldn't use a hard-coded volatile pointer to reference a hardware device". No, you literally need to do that sometimes; especially in embedded software.

lol768 2y ago

> I'm happy to up the ante by £50 to account for my second theory What's that, three pints in a pub inside the M25? :P Completely agree with this sentiment though, we've known that handling of binary data in memory unsafe languages has been risky for yonks. At the very least, fuzzing should've been employed here to try and detect these sorts of issues. More fundamentally though, where was their QA? These "channel files" just went out of the door without any idea as to their validity? Was there no continuous integration check to just .. ensure they parsed with the same parser as was deployed to the endpoints? And why were the channel files not deployed gradually?

TeMPOraL 2y ago
FWIW, before someone brings up JSON, GP's bet only makes sense when "binary" includes parsing text as well. In fact, most notorious software bugs are related to misuse of textual formats like SQL or JS.

1992spacemovie 2y ago

Interesting observation. As a non-developer, what can one do to enhance coverage for these types of scenerios? Fuzz testing?

rwmj 2y ago
Fuzz testing absolutely should be used whenever you parse anything.
- SoftTalker 2y ago
  Yeah, even if you are only parsing "safe" inputs such as ones you created yourself. Other bugs and sometimes even truly random events can corrupt data.

throw0101d 2y ago

> Approximately 100% of CVEs, crashes, bugs, slowdowns, and pain points of computing have to do with various forms of deserialising binary data back into machine-readable data structures. For the record, the top 25 common weaknesses for 2023 are listed at: * https://cwe.mitre.org/top25/archive/2023/2023_top25_list.html https://cwe.mitre.org/top25/archive/2023/2023_top25_list.htm... Deserialization of Untrusted Data (CWE-502) was number fifteen. Number one was Out-of-bounds Write (CWE-787), Use After Free (CWE-416) was number four. CWEs that have been in every list since they started doing this (2019): * https://cwe.mitre.org/top25/archive/2023/2023_stubborn_weaknesses.html https://cwe.mitre.org/top25/archive/2023/2023_stubborn_weakn...

lioeters 2y ago
# Top Stubborn Software Weaknesses (2019-2023) Out-of-bounds Write Improper Neutralization of Input During Web Page Generation (‘Cross-site Scripting’) Improper Neutralization of Special Elements used in an SQL Command (‘SQL Injection’) Use After Free Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') Improper Input Validation Out-of-bounds Read Improper Limitation of a Pathname to a Restricted Directory (‘Path Traversal’) Cross-Site Request Forgery (CSRF) NULL Pointer Dereference Improper Authentication Integer Overflow or Wraparound Deserialization of Untrusted Data Improper Restriction of Operations within Bounds of a Memory Buffer Use of Hard-coded Credentials
- TeMPOraL 2y ago
  Yup. Almost all of them are various flavor of fucking up a parser or misusing it (in particular, all the injection cases are typically caused by writing stupid code that glues strings together instead of proper parsing).
  - lolinder 2y ago
    That's not parsing, that's the inverse of parsing. It's taking untrusted data and injecting it into a string that will later be parsed into code without treating the data as untrusted and adapting accordingly. It's compiling, of a sort. Parsing is the reverse—taking an untrusted string (or binary string) that is meant to be code and converting it into a data structure. Both are the result of taking untrusted data and assuming it'll look like what you expect, but both are not parsing issues.

eru 2y ago

> Approximately 100% of CVEs, crashes, bugs, slowdowns, and pain points of computing have to do with various forms of deserialising binary data back into machine-readable data structures. All because a) human programmers forget to account for edge cases, and b) imperative programming languages allow us to do so. I wouldn't blame imperative programming. Eg Rust is imperative, and pretty good at telling you off when you forgot a case in your switch. By contrast the variant of Scheme I used twenty years ago was functional, but didn't have checks for covering all cases. (And Haskell's ghc didn't have that checked turned on by default a few years ago. Not sure if they changed that.)

seymore_12 2y ago

>Approximately 100% of CVEs, crashes, bugs, slowdowns, and pain points of computing have to do with various forms of deserialising binary data back into machine-readable data structures. All because a) human programmers forget to account for edge cases, and b) imperative programming languages allow us to do so. This. One year ago UK air traffic control collapsed due to inability to properly parse "faulty" flight plan: https://news.ycombinator.com/item?id=37461695 https://news.ycombinator.com/item?id=37461695

cedws 2y ago

I’d say that it is a bug by definition if your program ungracefully crashes when it’s passed malformed data at runtime.

stefan_ 2y ago

People are target fixating too much. Sure, this parser crashed and caused the system to go down. But in an alternative universe they push a definition file that rejects every openat() or connect() syscall. Your system is now equally as dead, except it probably won't even have the grace to restart. The whole concept of "we fuck with the system in kernel based on data downloaded from the internet" is just not very sound and safe.

hello_moto 2y ago
It's not and that's the sad state of AV in Windows

xxs 2y ago

>(for the non-British, that's £100) next time you'd be adding /s to your posts

back_to_basics 2y ago

"human programmers forget to account for edge cases" Which is precisely the rationale which led to Standard Operating Procedures and Best Practices (much like any other Sector of business has developed). I submit to you, respectfully, that a corporation shall never rise to a $75 Billion Market Cap without a bullet-proof adherence to such, and thus, this "event" should be properly characterized and viewed as a very suspicious anomaly, at the least https://news.ycombinator.com/item?id=41023539 https://news.ycombinator.com/item?id=41023539 fleshes out the proper context.

divan 2y ago

Related talk: 28c3: The Science of Insecurity (2011) https://www.youtube.com/watch?v=3kEfedtQVOY https://www.youtube.com/watch?v=3kEfedtQVOY

nonrandomstring 2y ago
Excellent talk. So long ago and what since?

smsm42 2y ago

> combination of said bad binary data and a poorly-written parser that didn't error out correctly upon reading invalid data By now, if you write any parser that deals with any outside data and don't fuzz the heck out of it, you are willfully negligent. Fuzzers are pretty easy to use, automatic and would likely catch any such problem pretty soon. So, did they fuzz and got very very unlucky or do they just like to live dangerously?

Log_out_ 2y ago

Soon, update revert watchdogs