7 ms·

This is a tool issue and not a language issue. I think that if you do some thing in code that results in unintended behavior, then that's a bug that should be c

This is a tool issue and not a language issue. I think that if you do some thing in code that results in unintended behavior, then that's a bug that should be communicated to the developer during development. A buffer over-run isn't a bug, its the symptom of a a bug in the arithmetic that computes how to access the buffer. The way to fix that is to communicate and address the bug during development, not to try to mitigate the symptom at run time. I am a C programmer, and to me buffer over-runs are a rare class of bugs, and a class of bugs that are (at least on the heap, and often on the stack) trivial to find and fix with the tools i have.

xxpor 6y ago
How are they trivial? The issue with buffer overruns is that they're usually silent. If you have an array of pointers, and you overrun that array, you could end up assigning garbage to another pointer variable. Then when you deref that, if you're lucky it'll be outside the mapped memory addresses for that process and you crash. If you're unlucky, you read or write to a random location. Basically the point is, it's very easy to end up in a situation where you know you have a bug but you're 2+ levels of indirection from the actual logic error. Those sorts of issues are always the biggest time sink.
- quelsolaar 6y ago
  I have a debug mode in my code base, where I wrap malloc, free, and realloc with debug versions that over allocate all allocations that are made, and put magic numbers before and after the memory blocks. These are periodically checked to make sure nothing has been overwritten that shouldn't be. You can find this code on my website, www.gamepipeline.org as part of the "forge" library. The system tells me very precisely what has gone wrong with buffer over-runs, and it also make memory leaks and double frees trivial to find. Visual studios debug mode does something similar with buffer overruns on the stack, but I tend not to put very many buffers on the stack so that much more rare as an issue. If you want to go even more hardcore, windows has "gflags" that you can turn on to find overruns using memory protection. LLVM also have the "Checked C" compiler that also finds these issues. My own system is so good that, I haven't used gflags in a few years and I also haven't tested Checked C so I cant vouch for it, but there are loads of possibilities in this space that i hope to explore further in the future.
  - xxpor 6y ago
    These sound good, and I've used things like asan and ubsan which do a lot of the same thing. The issue is not everyone's running in a traditional environment where you have the conveniences of things like... an OS.
    - quelsolaar 6y ago
      I sometimes write code for micro controllers, and I tend to wrap all system calls so that I can run the same code on my PC just to get access to better tools. On small hardware platforms its common that you are forced to use C because of the limited resources, so the options are limited.
  - mbrubeck 6y ago
    This testing-based approach is great at finding buffer overflows that occur within your test suite. But especially for code like curl that runs on untrusted data, you need a more adversarial mindset. Unless your tests cover every possible input, a clever attacker might always find input data that you didn't think to test.
    - deleted 6y ago
      [deleted]
    - mbrubeck 6y ago
      Here's an example from one of your own projects, where code that works fine on normal input has a buffer overrun on malicious or corrupted data: https://github.com/quelsolaar/HxA/issues/4 https://github.com/quelsolaar/HxA/issues/4
anchpop 6y ago
> I am a C programmer, and to me buffer over-runs are a rare class of bugs, and a class of bugs that are (at least on the heap, and often on the stack) trivial to find and fix with the tools i have. What does your testing setup look like? Does it include regular fuzzing and security audits? Buffer overruns are so common because normally your mental model of how the code should work assumes they don't happen, so you forget to test for their presence (or prove their absence, like Rust nudges you to).
- quelsolaar 6y ago
  I tend to have tests on a lot of the time (see below about my test lib). As someone who has worked with C for a long time, I think my mental model is all about direct memory access, so it probably comes more natural to me.
- megous 6y ago
  That sounds like the opposite of the mindset I have when writing C. Keeping buffer limits in mind all the time, and having them communicated around the code/function calls and respected, is one of the most basic things. I don't do fuzzing, because I leave string parsing to tools like re2c. I tried it in the past with my mail message parser written in C/re2c and it was not very helpful.
- benibela 6y ago
  There are also other low-level languages besides C and Rust I use Pascal, and I never have any buffer overruns in my code. If there were any, Pascal's automatic bound checking on array indices would raise an exception The only issue is, should I keep the bound checking in the release builds for maximum safety, or disable them for maximum performance?
jetrink 6y ago
Why are they so trivial to detect for you while they bedevil the rest of the software industry? Is it possible that there are many such bugs hiding in your code that have not been found, because your code has not been subjected to the amount of scrutiny that projects like cURL, Windows and Chrome have?
- quelsolaar 6y ago
  I think i have fewer, bugs but this is a wild guess as I have no data supporting this even though i have significant number of users. I dont know why others struggle with it, but I suspect: -Most programmers are not used to thinking about memory the way you should when using C. -Most developers are not aware of the tools available for debugging C.
  - indemnity 6y ago
    You should write an operating system, clearly Torvalds and the Microsoft and Apple kernel developers have not reached your level yet.
    - quelsolaar 6y ago
      I don't find either of the Kernels you mention to be particularly unsafe, despite being written in "unsafe" languages. Can you point to better kernels written in Rust?
      - amluto 6y ago
        Redox? That being said, a kernel is a huge undertaking. There is nothing else with the scope of Linux. And substantial parts of writing a kernel involve dealing with architecture nastiness, and, in the absence of formal architecture models, no language will help much here. The Linux x86 low level code is a terrifying mess, and it’s also the best and most capable implementation I’m aware of. (I’m obviously biased.) A language like Rust would help only a tiny bit. (Something like NMI handling on x86 is fundamentally memory unsafe. If you get nested NMIs, your stack gets clobbered. Thanks AMD. Linux has code, mostly in assembly, that detects and recovers. Good luck writing it in any memory safe language.) I once skimmed seL4, and I would not use its x86 code for serious work in its current form.
eat_veggies 6y ago
> The way to fix that is to communicate and address the bug during development, not to try to mitigate the symptom at run time. You'll write bugs and they'll make it past the development phase because bugs are inevitable. So now the choice is whether they're a crash or an rce.
deleted 6y ago
[deleted]
- deleted 6y ago
  [deleted]

xxpor 6y ago

How are they trivial? The issue with buffer overruns is that they're usually silent. If you have an array of pointers, and you overrun that array, you could end up assigning garbage to another pointer variable. Then when you deref that, if you're lucky it'll be outside the mapped memory addresses for that process and you crash. If you're unlucky, you read or write to a random location. Basically the point is, it's very easy to end up in a situation where you know you have a bug but you're 2+ levels of indirection from the actual logic error. Those sorts of issues are always the biggest time sink.

quelsolaar 6y ago
I have a debug mode in my code base, where I wrap malloc, free, and realloc with debug versions that over allocate all allocations that are made, and put magic numbers before and after the memory blocks. These are periodically checked to make sure nothing has been overwritten that shouldn't be. You can find this code on my website, www.gamepipeline.org as part of the "forge" library. The system tells me very precisely what has gone wrong with buffer over-runs, and it also make memory leaks and double frees trivial to find. Visual studios debug mode does something similar with buffer overruns on the stack, but I tend not to put very many buffers on the stack so that much more rare as an issue. If you want to go even more hardcore, windows has "gflags" that you can turn on to find overruns using memory protection. LLVM also have the "Checked C" compiler that also finds these issues. My own system is so good that, I haven't used gflags in a few years and I also haven't tested Checked C so I cant vouch for it, but there are loads of possibilities in this space that i hope to explore further in the future.
- xxpor 6y ago
  These sound good, and I've used things like asan and ubsan which do a lot of the same thing. The issue is not everyone's running in a traditional environment where you have the conveniences of things like... an OS.
  - quelsolaar 6y ago
    I sometimes write code for micro controllers, and I tend to wrap all system calls so that I can run the same code on my PC just to get access to better tools. On small hardware platforms its common that you are forced to use C because of the limited resources, so the options are limited.
- mbrubeck 6y ago
  This testing-based approach is great at finding buffer overflows that occur within your test suite. But especially for code like curl that runs on untrusted data, you need a more adversarial mindset. Unless your tests cover every possible input, a clever attacker might always find input data that you didn't think to test.
  - deleted 6y ago
    [deleted]
  - mbrubeck 6y ago
    Here's an example from one of your own projects, where code that works fine on normal input has a buffer overrun on malicious or corrupted data: https://github.com/quelsolaar/HxA/issues/4 https://github.com/quelsolaar/HxA/issues/4

anchpop 6y ago

> I am a C programmer, and to me buffer over-runs are a rare class of bugs, and a class of bugs that are (at least on the heap, and often on the stack) trivial to find and fix with the tools i have. What does your testing setup look like? Does it include regular fuzzing and security audits? Buffer overruns are so common because normally your mental model of how the code should work assumes they don't happen, so you forget to test for their presence (or prove their absence, like Rust nudges you to).

quelsolaar 6y ago
I tend to have tests on a lot of the time (see below about my test lib). As someone who has worked with C for a long time, I think my mental model is all about direct memory access, so it probably comes more natural to me.
megous 6y ago
That sounds like the opposite of the mindset I have when writing C. Keeping buffer limits in mind all the time, and having them communicated around the code/function calls and respected, is one of the most basic things. I don't do fuzzing, because I leave string parsing to tools like re2c. I tried it in the past with my mail message parser written in C/re2c and it was not very helpful.
benibela 6y ago
There are also other low-level languages besides C and Rust I use Pascal, and I never have any buffer overruns in my code. If there were any, Pascal's automatic bound checking on array indices would raise an exception The only issue is, should I keep the bound checking in the release builds for maximum safety, or disable them for maximum performance?

jetrink 6y ago

Why are they so trivial to detect for you while they bedevil the rest of the software industry? Is it possible that there are many such bugs hiding in your code that have not been found, because your code has not been subjected to the amount of scrutiny that projects like cURL, Windows and Chrome have?

quelsolaar 6y ago
I think i have fewer, bugs but this is a wild guess as I have no data supporting this even though i have significant number of users. I dont know why others struggle with it, but I suspect: -Most programmers are not used to thinking about memory the way you should when using C. -Most developers are not aware of the tools available for debugging C.
- indemnity 6y ago
  You should write an operating system, clearly Torvalds and the Microsoft and Apple kernel developers have not reached your level yet.
  - quelsolaar 6y ago
    I don't find either of the Kernels you mention to be particularly unsafe, despite being written in "unsafe" languages. Can you point to better kernels written in Rust?
    - amluto 6y ago
      Redox? That being said, a kernel is a huge undertaking. There is nothing else with the scope of Linux. And substantial parts of writing a kernel involve dealing with architecture nastiness, and, in the absence of formal architecture models, no language will help much here. The Linux x86 low level code is a terrifying mess, and it’s also the best and most capable implementation I’m aware of. (I’m obviously biased.) A language like Rust would help only a tiny bit. (Something like NMI handling on x86 is fundamentally memory unsafe. If you get nested NMIs, your stack gets clobbered. Thanks AMD. Linux has code, mostly in assembly, that detects and recovers. Good luck writing it in any memory safe language.) I once skimmed seL4, and I would not use its x86 code for serious work in its current form.

eat_veggies 6y ago

> The way to fix that is to communicate and address the bug during development, not to try to mitigate the symptom at run time. You'll write bugs and they'll make it past the development phase because bugs are inevitable. So now the choice is whether they're a crash or an rce.

deleted 6y ago

[deleted]

deleted 6y ago
[deleted]