9 ms·

Checked C: Making C Safer by Extension

84 points by matt_d 8y ago

kyberias 8y ago
October 1, 2018 ? :O
- skookumchuck 8y ago
  A buffer overflow?
- srujn 8y ago
  That's likely when the paper will be presented at the https://cybersecurity.ieee.org/blog/2018/01/31/ieee-secdev-2018-call-for-papers/ https://cybersecurity.ieee.org/blog/2018/01/31/ieee-secdev-2...
profquail 8y ago
Checked C source code: https://github.com/Microsoft/checkedc https://github.com/Microsoft/checkedc
saagarjha 8y ago
The problem here is you’re not writing C, you’re writing a language that kind of resembles C but is nonportable. So you might as well just use a different, memory safe language at that point.
- chongli 8y ago
  Yes. This is why people put so much effort into projects like Valgrind [1] and american fuzzy lop [2]. Preserving portability by keeping the same language and trying to find as many bugs as possible using clever tooling. [1] http://valgrind.org http://valgrind.org [2] http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/
- ychen306 8y ago
  It's about the control you can have. You can gradually add in annotations without having to rewrite the whole program as well as being able to selectively check buffer-overflows (i.e. checking everything vs checking code regions that you don't have security guarantee but would like to have). Additionally they also have features that allows you check overflow statically on limited cases.
- ramshorns 8y ago
  Well, probably most C code is not strictly standard-compliant C, because of undefined behaviour, GNU extensions, or whatever else.
- chubot 8y ago
  I think you're missing an important point: Checked C’s design is distinguished by its focus on backward-compatibility, incremental conversion, developer control, and enabling highly performant code The problem is that there is a huge body of decades-old C code out there that we all depend on. A typical Docker container with a web app contains maybe 10K lines of your own code in Python/Ruby/JS, 100K lines of framework code, and 1M or 10M lines of C code (i.e. the interpreter itself, the web server, SSL, the base image, etc.) So I welcome new languages focused on incremental conversion. That's what I'm doing with Unix shell: http://www.oilshell.org/blog/2018/01/28.html http://www.oilshell.org/blog/2018/01/28.html I applaud projects like Corrode, a C to unsafe-Rust translator, although undoubtedly they would have an easier time if Rust was designed in the first place for conversion. https://github.com/jameysharp/corrode https://github.com/jameysharp/corrode So Checked C seems like a great idea to me. I've noticed that most programmers seem to wildly overestimate the rate at which code gets rewritten. I think it's more accurate to say that code piles up over time. And unfortunately when the foundations are unstable, what you build on top is also unstable.
  - bluejekyll 8y ago
    There is also a new tool, c2rust, https://c2rust.com https://c2rust.com that I believe was inspired by Corrode. In addition, we also have the bindgen tool in Rust that will auto generate the Rust FFI from C header files. It works really well. At that point you wrap the FFI in more Rust idiomatic interfaces, and expose those as the preferred library for others to use. I mention this, because it is a path toward allowing for the carrying forward of a C codebase, but using a safe language like Rust for all future work, perhaps going back and rewriting the C if it’s deemed important. There are many examples of this pattern being applied to projects. In addition to that, there is a cbindgen tool, that does the inverse of the bindgen tool, which would allow you to write Rust code, and quickly generate C header files for the FFI, allowing for critical areas of C to be replaced with Rust, without converting the entire project. Putting all of that together might be a reasonable thing for large projects to convert iteratively, should they decide it’s an important thing for them to do.
    - sivakon 8y ago
- pjmlp 8y ago
  No, the problem here is that no one is going to re-write UNIX derived OSes into something else, nor change their culture around C. So there are only two options left, make C safe in some way no matter what, or just throw everything away. The Linux Security Summit 2018 set of talks regarding kernel mitigation exploits is pretty clear how bad security exploits due to C are turning out to be.
  - int0x80 8y ago
    The thing is, C can not be made "safe" without breaking old code and specified semantics. Even if you for example follow all the strict aliasing rules, you can still 1) make a mistake 2) access/modify arbitrary memory. Then you have overflows, threading races ... It is ok, C is meant to be like that. You can make C safer, make things better (arguably), that is a different thing. And it is beeing done, slowly and without breaking (in theory) stuff.
    - pjmlp 8y ago
      I agree the solution is not perfect and ideally I would nuke the language out of orbit. But that approach will never win the hearts of UNIXy kernel devs, and I guess until Quantum computers take over we are past the point of having alternative OSes to UNIX based ones, because no one is going to spend the required money to do that. So a migration path towards safety is required.
      - vinceguidry 8y ago
        I think, eventually, the whole stack will get rewritten from the ground up. Someone is going to come up with a minimal OS that can be run on virtual machines that implements LLVM in a provably correct way, the OS hooks will get reimplemented as libraries for all the languages, and over a few years or so the entire cloud infrastructure will simply switch over. I think we're already half the way there, Amazon Linux is how widely deployed already? The surface area between application logic and OS code is shrinking by the year, once serverless goes mainstream then even application code will start to get rare, pushed to only those specific projects that require human interaction. Everything else is 'mere' computation, something that can be specified purely algorithmically and executed by anything. The OS will only be relevant in the context of personal computing, the human interface to the machine kingdom which shed those trappings decades ago. 20-30 years down the road I see a translation layer unifying all three major, now solely personal, operating systems, and a fundamental, free software core being driven out, eventually supplanting proprietary development. Apple will do its best to retain tight coupling between their hardware and software, but they'll slowly fade just like Microsoft did as programming itself is revealed to consist of two main activities, intricate UX and comparatively dumb data plumbing, with a small priesthood of hardware driver maintainers / database developers for those who still want to do 'real' programming. Everybody will be able to code at some level and it'll be amazing.
- emilfihlman 8y ago
  Do you see the difference between "Here's a very small change, easy to implement and add." and "Let's just rewrite the whole thing and buy new devs" ?
  - saagarjha 8y ago
    Yes, I do. But the unfortunate reality is that your "small change" makes your code much harder to work with for other people, since you are now writing a C derivative instead of standard C.
    - stefano 8y ago
      This is the same argument that was used when Microsoft launched Typescript. As it turns out, their strategy of extending an existing language to make it safer worked great, while brand new languages that are safer than Typescript are now miles behind it in terms of adoption. So it seems to be a pretty solid strategy, and if it brings people closer to a safe language, even if not 100% safe, it's still an improvement on the status quo.
      - vmchale 8y ago
        > As it turns out, their strategy of extending an existing language to make it safer worked great, while brand new languages that are safer than Typescript are now miles behind it in terms of adoption Did it? Adoption seems to me to be a pretty silly metric given all the politics of language choice.
        pjmlp 8y ago
        Given that the goal is getting C developers to adopt best practices, adoption is the most relevant metric here.
- sureaboutthis 8y ago
  Completely agree. It's interesting to see Microsoft extend C and others, here, are embracing it so quickly.
  - saagarjha 8y ago
    That's an interesting choice of words you have there. Are you trying to allude to what I think you're alluding to?
deleted 8y ago
[deleted]
Animats 8y ago
I wonder how this will work out. GCC had checked fat pointers for years, used by almost nobody. Microsoft can make it stick on their platform, because they own the libraries and can make them use fat pointers. I hope this works out better than Microsoft Managed C++.
- sterlind 8y ago
  As far as I know, Managed C++ is mostly used as an interop layer between C/C++ and C#, sort of like the inverse of P/Invoke. Since P/Invoke can't marshal C++ classes, Managed C++ gives you a way to expose such things. It's a very special-purpose chimera.
- pjmlp 8y ago
  Managed C++ has been replaced by C++/CLI in 2005 and it pretty much alive. Its purpose was to make .NET interop easier with C++ libraries and has been quite successful at it, instead of having to play around with P/Invoke until everything works properly. Regarding Windows, Microsoft is pretty much on the ongoing transition to C++, where these kind of issues are easier to tackle. I bet their goals regarding Checked C have more to do with the adoption of UNIX (Linux/BSD) based technologies for Azure deployments and IoT devices like Azure Sphere.
- wott 8y ago
  There has been dozens of "safe" additions, variants and extensions to C along the years (from commercial companies, from individuals, from university labs, ...). Not one of them ever took off. This one did not either. We always just say "hmmm... right... this one may look a bit better than the previous one (or doesn't)" and we never bother to use them. After 20 years like this, we can conclude that we like C as it is and are not interested in extra layers.
  - stefano 8y ago
    > After 20 years like this, we can conclude that we like C as it is and are not interested in extra layers. This says a lot about how little the industry cares about software quality. You'd think a bunch of people who make a living out of automating tasks would happily use tools that automate the checking of an importnt source of bugs. Instead, many prefer to use tools that look simpler on the surface and then end up manually testing for the same kind of bugs over and over. So you have people writing in dynamically typed languages who rely on (manually written!) unit tests to check that their program is not mixing up types, and others using unsafe languages and then relying on tests to catch memory access violations. Inevitably, tests never catch all issues and you end up with type errors and segfaults in production.
Someone 8y ago
This does nothing for use-after-free or integer under/overflow, so “Making C SafeR by Extension” would be a better title. It’s good to see people attempting this, though. I would think chances are good Microsoft will use it internally, and that already would be a win. Also, it’s a clang/LLVM extension. That makes it easier for third parties to start using it. On the other hand, their choice to use Ptr<T> for familiarity with C++ requires users who want to keep their code compatible with C compilers that don’t support this a bit harder (it will require wrapping that in some macro, say _ptrTo(T)). That isn’t ideal for adoption in open source programs.
- dang 8y ago
  Ok, we've squeezed in an 'r'.
- ainar-g 8y ago
  Re. syntax. I don't understand, why can't they just make _Ptr an attribute like _Atomic, and provide a header <stdchecked.h>. That is, #include <stdchecked.h> void read_next(int *b, int idx, checked int *out) { int tmp = *(b+idx); *out = tmp; }
- vmchale 8y ago
  > integer under/overflow very few languages actually solve this completely. you'd need refinement types to do so.
styfle 8y ago
There is some precedent for a new language that gradually improves safety of an existing language; TypeScript did this to JavaScript and is arguably successful. I don’t know if Checked C will have the same success story but if devs can easily add this to their existing projects and it catches bugs, that seems like a win to me.

84 points by matt_d 8y ago

kyberias 8y ago

October 1, 2018 ? :O

skookumchuck 8y ago
A buffer overflow?
srujn 8y ago
That's likely when the paper will be presented at the https://cybersecurity.ieee.org/blog/2018/01/31/ieee-secdev-2018-call-for-papers/ https://cybersecurity.ieee.org/blog/2018/01/31/ieee-secdev-2...

profquail 8y ago

Checked C source code: https://github.com/Microsoft/checkedc https://github.com/Microsoft/checkedc

saagarjha 8y ago

The problem here is you’re not writing C, you’re writing a language that kind of resembles C but is nonportable. So you might as well just use a different, memory safe language at that point.

chongli 8y ago
Yes. This is why people put so much effort into projects like Valgrind [1] and american fuzzy lop [2]. Preserving portability by keeping the same language and trying to find as many bugs as possible using clever tooling. [1] http://valgrind.org http://valgrind.org [2] http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/
ychen306 8y ago
It's about the control you can have. You can gradually add in annotations without having to rewrite the whole program as well as being able to selectively check buffer-overflows (i.e. checking everything vs checking code regions that you don't have security guarantee but would like to have). Additionally they also have features that allows you check overflow statically on limited cases.
ramshorns 8y ago
Well, probably most C code is not strictly standard-compliant C, because of undefined behaviour, GNU extensions, or whatever else.
chubot 8y ago
I think you're missing an important point: Checked C’s design is distinguished by its focus on backward-compatibility, incremental conversion, developer control, and enabling highly performant code The problem is that there is a huge body of decades-old C code out there that we all depend on. A typical Docker container with a web app contains maybe 10K lines of your own code in Python/Ruby/JS, 100K lines of framework code, and 1M or 10M lines of C code (i.e. the interpreter itself, the web server, SSL, the base image, etc.) So I welcome new languages focused on incremental conversion. That's what I'm doing with Unix shell: http://www.oilshell.org/blog/2018/01/28.html http://www.oilshell.org/blog/2018/01/28.html I applaud projects like Corrode, a C to unsafe-Rust translator, although undoubtedly they would have an easier time if Rust was designed in the first place for conversion. https://github.com/jameysharp/corrode https://github.com/jameysharp/corrode So Checked C seems like a great idea to me. I've noticed that most programmers seem to wildly overestimate the rate at which code gets rewritten. I think it's more accurate to say that code piles up over time. And unfortunately when the foundations are unstable, what you build on top is also unstable.
- bluejekyll 8y ago
  There is also a new tool, c2rust, https://c2rust.com https://c2rust.com that I believe was inspired by Corrode. In addition, we also have the bindgen tool in Rust that will auto generate the Rust FFI from C header files. It works really well. At that point you wrap the FFI in more Rust idiomatic interfaces, and expose those as the preferred library for others to use. I mention this, because it is a path toward allowing for the carrying forward of a C codebase, but using a safe language like Rust for all future work, perhaps going back and rewriting the C if it’s deemed important. There are many examples of this pattern being applied to projects. In addition to that, there is a cbindgen tool, that does the inverse of the bindgen tool, which would allow you to write Rust code, and quickly generate C header files for the FFI, allowing for critical areas of C to be replaced with Rust, without converting the entire project. Putting all of that together might be a reasonable thing for large projects to convert iteratively, should they decide it’s an important thing for them to do.
  - sivakon 8y ago
pjmlp 8y ago
No, the problem here is that no one is going to re-write UNIX derived OSes into something else, nor change their culture around C. So there are only two options left, make C safe in some way no matter what, or just throw everything away. The Linux Security Summit 2018 set of talks regarding kernel mitigation exploits is pretty clear how bad security exploits due to C are turning out to be.
- int0x80 8y ago
  The thing is, C can not be made "safe" without breaking old code and specified semantics. Even if you for example follow all the strict aliasing rules, you can still 1) make a mistake 2) access/modify arbitrary memory. Then you have overflows, threading races ... It is ok, C is meant to be like that. You can make C safer, make things better (arguably), that is a different thing. And it is beeing done, slowly and without breaking (in theory) stuff.
  - pjmlp 8y ago
    I agree the solution is not perfect and ideally I would nuke the language out of orbit. But that approach will never win the hearts of UNIXy kernel devs, and I guess until Quantum computers take over we are past the point of having alternative OSes to UNIX based ones, because no one is going to spend the required money to do that. So a migration path towards safety is required.
    - vinceguidry 8y ago
      I think, eventually, the whole stack will get rewritten from the ground up. Someone is going to come up with a minimal OS that can be run on virtual machines that implements LLVM in a provably correct way, the OS hooks will get reimplemented as libraries for all the languages, and over a few years or so the entire cloud infrastructure will simply switch over. I think we're already half the way there, Amazon Linux is how widely deployed already? The surface area between application logic and OS code is shrinking by the year, once serverless goes mainstream then even application code will start to get rare, pushed to only those specific projects that require human interaction. Everything else is 'mere' computation, something that can be specified purely algorithmically and executed by anything. The OS will only be relevant in the context of personal computing, the human interface to the machine kingdom which shed those trappings decades ago. 20-30 years down the road I see a translation layer unifying all three major, now solely personal, operating systems, and a fundamental, free software core being driven out, eventually supplanting proprietary development. Apple will do its best to retain tight coupling between their hardware and software, but they'll slowly fade just like Microsoft did as programming itself is revealed to consist of two main activities, intricate UX and comparatively dumb data plumbing, with a small priesthood of hardware driver maintainers / database developers for those who still want to do 'real' programming. Everybody will be able to code at some level and it'll be amazing.
emilfihlman 8y ago
Do you see the difference between "Here's a very small change, easy to implement and add." and "Let's just rewrite the whole thing and buy new devs" ?
- saagarjha 8y ago
  Yes, I do. But the unfortunate reality is that your "small change" makes your code much harder to work with for other people, since you are now writing a C derivative instead of standard C.
  - stefano 8y ago
    This is the same argument that was used when Microsoft launched Typescript. As it turns out, their strategy of extending an existing language to make it safer worked great, while brand new languages that are safer than Typescript are now miles behind it in terms of adoption. So it seems to be a pretty solid strategy, and if it brings people closer to a safe language, even if not 100% safe, it's still an improvement on the status quo.
    - vmchale 8y ago
      > As it turns out, their strategy of extending an existing language to make it safer worked great, while brand new languages that are safer than Typescript are now miles behind it in terms of adoption Did it? Adoption seems to me to be a pretty silly metric given all the politics of language choice.
      - pjmlp 8y ago
        Given that the goal is getting C developers to adopt best practices, adoption is the most relevant metric here.
sureaboutthis 8y ago
Completely agree. It's interesting to see Microsoft extend C and others, here, are embracing it so quickly.
- saagarjha 8y ago
  That's an interesting choice of words you have there. Are you trying to allude to what I think you're alluding to?

deleted 8y ago

[deleted]

Animats 8y ago

I wonder how this will work out. GCC had checked fat pointers for years, used by almost nobody. Microsoft can make it stick on their platform, because they own the libraries and can make them use fat pointers. I hope this works out better than Microsoft Managed C++.

sterlind 8y ago
As far as I know, Managed C++ is mostly used as an interop layer between C/C++ and C#, sort of like the inverse of P/Invoke. Since P/Invoke can't marshal C++ classes, Managed C++ gives you a way to expose such things. It's a very special-purpose chimera.
pjmlp 8y ago
Managed C++ has been replaced by C++/CLI in 2005 and it pretty much alive. Its purpose was to make .NET interop easier with C++ libraries and has been quite successful at it, instead of having to play around with P/Invoke until everything works properly. Regarding Windows, Microsoft is pretty much on the ongoing transition to C++, where these kind of issues are easier to tackle. I bet their goals regarding Checked C have more to do with the adoption of UNIX (Linux/BSD) based technologies for Azure deployments and IoT devices like Azure Sphere.
wott 8y ago
There has been dozens of "safe" additions, variants and extensions to C along the years (from commercial companies, from individuals, from university labs, ...). Not one of them ever took off. This one did not either. We always just say "hmmm... right... this one may look a bit better than the previous one (or doesn't)" and we never bother to use them. After 20 years like this, we can conclude that we like C as it is and are not interested in extra layers.
- stefano 8y ago
  > After 20 years like this, we can conclude that we like C as it is and are not interested in extra layers. This says a lot about how little the industry cares about software quality. You'd think a bunch of people who make a living out of automating tasks would happily use tools that automate the checking of an importnt source of bugs. Instead, many prefer to use tools that look simpler on the surface and then end up manually testing for the same kind of bugs over and over. So you have people writing in dynamically typed languages who rely on (manually written!) unit tests to check that their program is not mixing up types, and others using unsafe languages and then relying on tests to catch memory access violations. Inevitably, tests never catch all issues and you end up with type errors and segfaults in production.

Someone 8y ago

This does nothing for use-after-free or integer under/overflow, so “Making C SafeR by Extension” would be a better title. It’s good to see people attempting this, though. I would think chances are good Microsoft will use it internally, and that already would be a win. Also, it’s a clang/LLVM extension. That makes it easier for third parties to start using it. On the other hand, their choice to use Ptr<T> for familiarity with C++ requires users who want to keep their code compatible with C compilers that don’t support this a bit harder (it will require wrapping that in some macro, say _ptrTo(T)). That isn’t ideal for adoption in open source programs.

dang 8y ago
Ok, we've squeezed in an 'r'.
ainar-g 8y ago
Re. syntax. I don't understand, why can't they just make _Ptr an attribute like _Atomic, and provide a header <stdchecked.h>. That is, #include <stdchecked.h> void read_next(int *b, int idx, checked int *out) { int tmp = *(b+idx); *out = tmp; }
vmchale 8y ago
> integer under/overflow very few languages actually solve this completely. you'd need refinement types to do so.

styfle 8y ago

There is some precedent for a new language that gradually improves safety of an existing language; TypeScript did this to JavaScript and is arguably successful. I don’t know if Checked C will have the same success story but if devs can easily add this to their existing projects and it catches bugs, that seems like a win to me.