6 ms·

Some sympathy for the devil: operating systems are just so…big. Like, there’s an arbitrary code execution issue fixed here in ColorSync. ColorSync! Who thinks

Some sympathy for the devil: operating systems are just so…big. Like, there’s an arbitrary code execution issue fixed here in ColorSync. ColorSync! Who thinks to attack the color management system, and then how do you set up your org structure to protect from attacks on that? Because…assuming they have much of a team on it at all, as it’s probably a bit of a solved problem…your core team is probably going to be focused on color, and not buffer overflows. macOS has a lot of old components now, and just getting a grasp of where you could possibly have security issues must be nigh impossible.

jrockway 5y ago
I think you prevent these "corrupt file corrupts important memory" bugs in two ways: 1) use a memory safe language 2) write fuzz tests just like you'd write unit tests. 1 is becoming extremely viable, infrastructure for 2 is starting to be included with modern programming languages and so will soon become the norm.
- DrBenCarson 5y ago
  You want all of every OS rewritten in Rust? Boy I’ve got bad news for you
  - howinteresting 5y ago
    Directionally, yes, I want more parts of more OSes written in Rust and other languages that allow building safe abstractions over unsafe code. This is table stakes in 2022.
    - deleted 5y ago
      [deleted]
  - dangus 5y ago
    Apple is already writing more of the OS in Swift: https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/ https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-...
    - ksec 5y ago
      Most of them are Apps. Not what I considered as "OS"
      - pjmlp 5y ago
        To be fair, they need to start somewhere, and if userspace gets more secure, it is an easier sell to non-belivers. Think a bit like Android, versus the hard battle from .NET on Windows with WinDev love for C++/COM.
  - McMiniBurger 5y ago
    could you elaborate on the "bad news" part? Is there something rust inherently can't do but c/c++ can?
    - izacus 5y ago
      Exist 20 years ago when the OS was written.
      - Someone 5y ago
        Even ColorSync, which the top of this thread mentioned, is older than that. It was first released in 1993 (https://en.wikipedia.org/wiki/List_of_macOS_components#ColorSync_Utility https://en.wikipedia.org/wiki/List_of_macOS_components#Color...) The work to remove all AT&T code from BSD started in 1989, so other parts of MacOS probably are over 30 years old, too. I don’t think anything of the original Mac OS remains, lots of it being written in 68k assembly and pascal.
    - vineyardmike 5y ago
      Yea, rust doesn't already exist. Rewriting a whole OS is an impossibly high ask.
    - deleted 5y ago
      [deleted]
  - vineyardmike 5y ago
    Rust? You mean swift right?
  - pjmlp 5y ago
    It is quite telling that every single time one talks about secure OSes, someone has to bring out Rust. The parent only mentioned "use a memory safe language", yet Rust it must be. No, it could have been JOVIAL, NEWP, PL/I, BLISS, Modula-2, Modula-3, Oberon, Object Pascal, D, Nim, System C#, Ada, Swift, ....
    - ksec 5y ago
      That is why i keep mentioning Ada in the hope to balance the "Memory Safe language" selection bias. I only mention it a few times whenever the topic of memory safe language and Rust came up. And I have already been asked to stop.
      - tasogare 5y ago
        Please keep fighting. The automatic association here on memory safe language with Rust annoys me as well, just as the one about mixing languages on a VM and WebAssembly. Both features are available since decades on C# and the CLR, and Java and the JVM to name a few.
      - pjmlp 5y ago
        I guess there are multiple issues with memory safe language bringing Rust into the talk. One is the naive one, someone that lacks the proper background and for whatever reasons thinks it is the only way, thus Rust. Then we have those that are aware, but think all the other safety approaches are not valid for whatever reason, thus Rust. Then we have those that kind of feel threatened by the security awareness that Rust brings into the picture, so whenever we talk about security, there is pavlovian reaction that it must be Rust when the subject is security. Picking any language that embraces bounds checking enabled by default, and proper string/arrays, is already a major improvement.
        bigbizisverywyz 5y ago
        >Picking any language that embraces bounds checking enabled by default, and proper string/arrays, is already a major improvement. Even bog standard C can be made safe, and include bounds checking if you create and consistently use an API that supports that mindset - see for e.g. Microsoft's string safe lib (not that I'm saying that is the epitome of such a thing). However the will and the effort has to be there, C is an incredibly flexible and capable language and can be as safe as you want it to be. The last Random ascii article I read, he found an easily exploitable buffer overflow in iOS simply by checking the code to see where it used memmove and worked out the exploit from there, guessing correctly that there was no bounds checking. So it seems that the usual culprits are fairly easy to spot, but somebody needs to replace the worst APIs with something safer (memmove, memcpy, memset, malloc, a = b).
  - tialaramex 5y ago
    The quote was "corrupt file corrupts important memory" which is exactly what WUFFS is for. In Rust you can screw up badly enough to still have the bug, doing so is harder but not impossible - but in WUFFS the stupidest file format parsing you can do is merely wrong and can't corrupt anything. In WUFFS possible outcomes of incompetent morons reading an image file include: the colours in the image are wrong, the image is just a solid black or white rectangle because they didn't remember to actually render the image, bits of it are smashed up somehow, and so on. Notice how it's always about the image? WUFFS can't express programs that do other stuff. General purpose languages are cool, but their unlimited expressiveness is exactly what you don't want for security. Why is this PNG loader able to make network connections and control my camera? You know a bad guy will find a way to force it to use those capabilities right?
- giantrobot 5y ago
  > write fuzz tests just like you'd write unit tests You're missing a key point the GP made, operating systems are big. There's also a finite number of person-hours in a development cycle. Third, fuzzing is a complicated and open ended process that's difficult to analyze. Fuzzing is not a simple subject. Testing fuzzed input requires a lot of instrumentation that can affect measurements if it's instrumentation not enabled in production builds. Not all code is easily tested with fuzzed input even when it can be properly instrumented. Because there's limited amounts of time in a day, every working moment can't be spent on the near infinite permutations of fuzzed input for every component on a system. Even automated test suites take time to run and then can only test known workflows or code paths. An OS has thousands of components with complex graphs of those interacting components. There's vast surface area for bugs and limited time to explore it all. Just switching languages and adding more tests aren't going to make bugs go away.
  - jacobr1 5y ago
    Big makes it challenging. But also we are talking about big companies. The OSX/Apple, Windows/Microsoft and linux/IBM/Goog/Amazon/long-tail are big enough that they already invest lots in security and can invest even more. This is where central tooling teams and come into play.
    - giantrobot 5y ago
      I was at Apple for more than 15 years in SWE. The scale of OS builds isn't just big but just this side of intractable. They have tool and build teams. It's not an easily solved problem and you can't just throw money at it to make it so. The OS has thousands of components with combinatorial amounts of different interactions and states. You can have multiple bug free components that create an exploitable situation in combination. The same is true for Windows or Linux or any large software project. Unit tests are not magic. Neither is fuzzing. Even if you've got a great fuzzing infrastructure and coverage it still takes non-zero time for an engineer to analyze, understand, and then fix a problem. Many components in an OS require a fair amount of domain knowledge so engineers aren't fungible. You can't grab Sally from the Mail team and give her a bunch of CoreMedia bugs to fix. Keep in mind all the companies you mention invest significantly in security and just barely keep ahead of exploits and often can't keep ahead of exploits. Hackers don't have the same burden. They don't need to find every bug. They only need to find a profitable one (depending on the hat they're wearing). They also don't need to maintain their exploit in future versions. They just move on to find the next exploit.
bumblebritches5 5y ago
brundolf 5y ago
Which is exactly why we need to use safe languages. People can't even prevent vulnerabilities in the code they are scrutinizing, much less the other 90% they aren't. (Beating a dead horse, I know, and not blaming the devs because this utility has been around forever, but regardless)
- perardi 5y ago
  It’s a horse worth beating, or at least kicking, because it’s a tough set of incentives. Even if you have functionally unlimited money like Apple, it’s tough to prioritize rewriting a somewhat obscure piece of the OS solely to prevent security issues. Because if you have a team that’s capable of fully reimplementing this component, and getting it through a QA process to make sure it doesn’t break another component of the OS…well damn, that’s a team you probably want on a higher-profile project, and not doing cleanup, right?
  - brundolf 5y ago
    Yeah, rewriting is hard and involves lots of complex cost tradeoffs. But at least when writing new software, with today's options, I think it's much harder to justify starting out with an unsafe language.
    - perardi 5y ago
      Well, web apps. But that’s me being argumentative. 100% agree for “system software” like this.
    - pjmlp 5y ago
      Today's options? The options were always there since around 1958, then about 15 years later, a group of folks doing their tiny OS decided to ignore them when creating their fun little language, because it was more fun to do their own thing. The problem is that this tiny OS and accompaning language grew to be everywhere thanks to its license, so now we are in this mess. But options? They have always been there for those that actually care.
    - 5y ago
SulphurCrested 5y ago
Since you pick on that example, yes I am unsurprised about an exploit in ColorSync. The bugs in ColorSync in combination with Photoshop are legend: for example, there was a "black crush" bug doing the rounds of photo blogs and forums which persisted for months. More recently, it was not possible to disable ColorSync when printing, which made it impossible to calibrate printers (that is, measure the printed result with a spectrophotometer in order to derive new ColorSync tables). And furthermore, I don't think it is actually the first colour profile exploit I've heard of, although I forget whether it was Apple's or someone else's. However much of a team they have on ColorSync, it needs to be bigger. If they don't have code fixes to make, there's plenty of other things for them to get on with. I didn't think much of the API documentation, and multiple people who seemed credible to me claimed there wasn't enough detail for the kind of implementation Adobe and others need, as several calls have been deprecated. And, since Apple no longer offer their own competing product it would make sense to assist Adobe in getting it reliably right. More than that, the calibration software for monitors is third party and, at least for the photo monitor I have... could be better. These monitors use onboard hardware LUTs instead of doing it in the GPU, so the software is hardware-specific. Fortunately, there are only a couple of brands who offer these high-end monitors, maybe 3 if you count LG, so there isn't all that much hardware to support. It's another case where the overall experience could be improved for a segment of Apple's customers if they put the effort in. There may even be money to be made. ColorSync shouldn't have been an "old" component, if Apple had kept their eyes on what their users needed. Instead they went and changed Music again, which has apparently not pleased its users. It's a company whose management seems obsessed with a fairly narrow range of applications like home automation and instant messaging, the kinds of things shown in 1970s TV programs about the "future". If they widened their horizons more they would end up cycling through more of the many components of macOS more often, so things got timely refreshes.
- kosolam 5y ago
  Could be the reason is they are putting the effort where 80% of the users are?
  - SulphurCrested 5y ago
the-golden-one 5y ago
Would it not be possible to detect these kind of errors using machine learning now? To go through the code base and find unsafe operations? There must be enough security bug fixes out there on GitHub that certain classes of security error become easily detectable.
- SSLy 5y ago
  There are no techbro startups to invest into curating datasets like this.

jrockway 5y ago

I think you prevent these "corrupt file corrupts important memory" bugs in two ways: 1) use a memory safe language 2) write fuzz tests just like you'd write unit tests. 1 is becoming extremely viable, infrastructure for 2 is starting to be included with modern programming languages and so will soon become the norm.

DrBenCarson 5y ago
You want all of every OS rewritten in Rust? Boy I’ve got bad news for you
- howinteresting 5y ago
  Directionally, yes, I want more parts of more OSes written in Rust and other languages that allow building safe abstractions over unsafe code. This is table stakes in 2022.
  - deleted 5y ago
    [deleted]
- dangus 5y ago
  Apple is already writing more of the OS in Swift: https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/ https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-...
  - ksec 5y ago
    Most of them are Apps. Not what I considered as "OS"
    - pjmlp 5y ago
      To be fair, they need to start somewhere, and if userspace gets more secure, it is an easier sell to non-belivers. Think a bit like Android, versus the hard battle from .NET on Windows with WinDev love for C++/COM.
- McMiniBurger 5y ago
  could you elaborate on the "bad news" part? Is there something rust inherently can't do but c/c++ can?
  - izacus 5y ago
    Exist 20 years ago when the OS was written.
    - Someone 5y ago
      Even ColorSync, which the top of this thread mentioned, is older than that. It was first released in 1993 (https://en.wikipedia.org/wiki/List_of_macOS_components#ColorSync_Utility https://en.wikipedia.org/wiki/List_of_macOS_components#Color...) The work to remove all AT&T code from BSD started in 1989, so other parts of MacOS probably are over 30 years old, too. I don’t think anything of the original Mac OS remains, lots of it being written in 68k assembly and pascal.
  - vineyardmike 5y ago
    Yea, rust doesn't already exist. Rewriting a whole OS is an impossibly high ask.
  - deleted 5y ago
    [deleted]
- vineyardmike 5y ago
  Rust? You mean swift right?
- pjmlp 5y ago
  It is quite telling that every single time one talks about secure OSes, someone has to bring out Rust. The parent only mentioned "use a memory safe language", yet Rust it must be. No, it could have been JOVIAL, NEWP, PL/I, BLISS, Modula-2, Modula-3, Oberon, Object Pascal, D, Nim, System C#, Ada, Swift, ....
  - ksec 5y ago
    That is why i keep mentioning Ada in the hope to balance the "Memory Safe language" selection bias. I only mention it a few times whenever the topic of memory safe language and Rust came up. And I have already been asked to stop.
    - tasogare 5y ago
      Please keep fighting. The automatic association here on memory safe language with Rust annoys me as well, just as the one about mixing languages on a VM and WebAssembly. Both features are available since decades on C# and the CLR, and Java and the JVM to name a few.
    - pjmlp 5y ago
      I guess there are multiple issues with memory safe language bringing Rust into the talk. One is the naive one, someone that lacks the proper background and for whatever reasons thinks it is the only way, thus Rust. Then we have those that are aware, but think all the other safety approaches are not valid for whatever reason, thus Rust. Then we have those that kind of feel threatened by the security awareness that Rust brings into the picture, so whenever we talk about security, there is pavlovian reaction that it must be Rust when the subject is security. Picking any language that embraces bounds checking enabled by default, and proper string/arrays, is already a major improvement.
      - bigbizisverywyz 5y ago
        >Picking any language that embraces bounds checking enabled by default, and proper string/arrays, is already a major improvement. Even bog standard C can be made safe, and include bounds checking if you create and consistently use an API that supports that mindset - see for e.g. Microsoft's string safe lib (not that I'm saying that is the epitome of such a thing). However the will and the effort has to be there, C is an incredibly flexible and capable language and can be as safe as you want it to be. The last Random ascii article I read, he found an easily exploitable buffer overflow in iOS simply by checking the code to see where it used memmove and worked out the exploit from there, guessing correctly that there was no bounds checking. So it seems that the usual culprits are fairly easy to spot, but somebody needs to replace the worst APIs with something safer (memmove, memcpy, memset, malloc, a = b).
- tialaramex 5y ago
  The quote was "corrupt file corrupts important memory" which is exactly what WUFFS is for. In Rust you can screw up badly enough to still have the bug, doing so is harder but not impossible - but in WUFFS the stupidest file format parsing you can do is merely wrong and can't corrupt anything. In WUFFS possible outcomes of incompetent morons reading an image file include: the colours in the image are wrong, the image is just a solid black or white rectangle because they didn't remember to actually render the image, bits of it are smashed up somehow, and so on. Notice how it's always about the image? WUFFS can't express programs that do other stuff. General purpose languages are cool, but their unlimited expressiveness is exactly what you don't want for security. Why is this PNG loader able to make network connections and control my camera? You know a bad guy will find a way to force it to use those capabilities right?
giantrobot 5y ago
> write fuzz tests just like you'd write unit tests You're missing a key point the GP made, operating systems are big. There's also a finite number of person-hours in a development cycle. Third, fuzzing is a complicated and open ended process that's difficult to analyze. Fuzzing is not a simple subject. Testing fuzzed input requires a lot of instrumentation that can affect measurements if it's instrumentation not enabled in production builds. Not all code is easily tested with fuzzed input even when it can be properly instrumented. Because there's limited amounts of time in a day, every working moment can't be spent on the near infinite permutations of fuzzed input for every component on a system. Even automated test suites take time to run and then can only test known workflows or code paths. An OS has thousands of components with complex graphs of those interacting components. There's vast surface area for bugs and limited time to explore it all. Just switching languages and adding more tests aren't going to make bugs go away.
- jacobr1 5y ago
  Big makes it challenging. But also we are talking about big companies. The OSX/Apple, Windows/Microsoft and linux/IBM/Goog/Amazon/long-tail are big enough that they already invest lots in security and can invest even more. This is where central tooling teams and come into play.
  - giantrobot 5y ago
    I was at Apple for more than 15 years in SWE. The scale of OS builds isn't just big but just this side of intractable. They have tool and build teams. It's not an easily solved problem and you can't just throw money at it to make it so. The OS has thousands of components with combinatorial amounts of different interactions and states. You can have multiple bug free components that create an exploitable situation in combination. The same is true for Windows or Linux or any large software project. Unit tests are not magic. Neither is fuzzing. Even if you've got a great fuzzing infrastructure and coverage it still takes non-zero time for an engineer to analyze, understand, and then fix a problem. Many components in an OS require a fair amount of domain knowledge so engineers aren't fungible. You can't grab Sally from the Mail team and give her a bunch of CoreMedia bugs to fix. Keep in mind all the companies you mention invest significantly in security and just barely keep ahead of exploits and often can't keep ahead of exploits. Hackers don't have the same burden. They don't need to find every bug. They only need to find a profitable one (depending on the hat they're wearing). They also don't need to maintain their exploit in future versions. They just move on to find the next exploit.

bumblebritches5 5y ago

brundolf 5y ago

Which is exactly why we need to use safe languages. People can't even prevent vulnerabilities in the code they are scrutinizing, much less the other 90% they aren't. (Beating a dead horse, I know, and not blaming the devs because this utility has been around forever, but regardless)

perardi 5y ago
It’s a horse worth beating, or at least kicking, because it’s a tough set of incentives. Even if you have functionally unlimited money like Apple, it’s tough to prioritize rewriting a somewhat obscure piece of the OS solely to prevent security issues. Because if you have a team that’s capable of fully reimplementing this component, and getting it through a QA process to make sure it doesn’t break another component of the OS…well damn, that’s a team you probably want on a higher-profile project, and not doing cleanup, right?
- brundolf 5y ago
  Yeah, rewriting is hard and involves lots of complex cost tradeoffs. But at least when writing new software, with today's options, I think it's much harder to justify starting out with an unsafe language.
  - perardi 5y ago
    Well, web apps. But that’s me being argumentative. 100% agree for “system software” like this.
  - pjmlp 5y ago
    Today's options? The options were always there since around 1958, then about 15 years later, a group of folks doing their tiny OS decided to ignore them when creating their fun little language, because it was more fun to do their own thing. The problem is that this tiny OS and accompaning language grew to be everywhere thanks to its license, so now we are in this mess. But options? They have always been there for those that actually care.
  - 5y ago

SulphurCrested 5y ago

Since you pick on that example, yes I am unsurprised about an exploit in ColorSync. The bugs in ColorSync in combination with Photoshop are legend: for example, there was a "black crush" bug doing the rounds of photo blogs and forums which persisted for months. More recently, it was not possible to disable ColorSync when printing, which made it impossible to calibrate printers (that is, measure the printed result with a spectrophotometer in order to derive new ColorSync tables). And furthermore, I don't think it is actually the first colour profile exploit I've heard of, although I forget whether it was Apple's or someone else's. However much of a team they have on ColorSync, it needs to be bigger. If they don't have code fixes to make, there's plenty of other things for them to get on with. I didn't think much of the API documentation, and multiple people who seemed credible to me claimed there wasn't enough detail for the kind of implementation Adobe and others need, as several calls have been deprecated. And, since Apple no longer offer their own competing product it would make sense to assist Adobe in getting it reliably right. More than that, the calibration software for monitors is third party and, at least for the photo monitor I have... could be better. These monitors use onboard hardware LUTs instead of doing it in the GPU, so the software is hardware-specific. Fortunately, there are only a couple of brands who offer these high-end monitors, maybe 3 if you count LG, so there isn't all that much hardware to support. It's another case where the overall experience could be improved for a segment of Apple's customers if they put the effort in. There may even be money to be made. ColorSync shouldn't have been an "old" component, if Apple had kept their eyes on what their users needed. Instead they went and changed Music again, which has apparently not pleased its users. It's a company whose management seems obsessed with a fairly narrow range of applications like home automation and instant messaging, the kinds of things shown in 1970s TV programs about the "future". If they widened their horizons more they would end up cycling through more of the many components of macOS more often, so things got timely refreshes.

kosolam 5y ago
Could be the reason is they are putting the effort where 80% of the users are?
- SulphurCrested 5y ago

the-golden-one 5y ago

Would it not be possible to detect these kind of errors using machine learning now? To go through the code base and find unsafe operations? There must be enough security bug fixes out there on GitHub that certain classes of security error become easily detectable.

SSLy 5y ago
There are no techbro startups to invest into curating datasets like this.