4 ms·

If anyone from Microsoft is listening: I'm a very experienced C programmer and what I would like, much more then a new language version, are some basic OS func

If anyone from Microsoft is listening: I'm a very experienced C programmer and what I would like, much more then a new language version, are some basic OS functions where I can query the status of a pointer. Is it readable? Is it writable? Where does the memory allocation start and end? Is the memory in the heap or stack, or a memory mapped file? All the information is available inside the OS, but I cant get at it. If it was available it would be possible to write all kinds of instrumentation. If it was possible (in debug mode) to get the layout of the stack, that would also be super useful. Buffer overruns are really mostly a problem when they are on the stack. By warping malloc and free, you can easily find buffer over runs, memory leaks and double frees. They become a none issue. But with stack memory you have very little control. I avoid using the stack for operations where this might be an issue, just because its so much harder to debug. I would love to be able to write APIs that have built in tests (in debug mode) where all pointers are checked, and if anything is broken the API can fail gracefully and give out precise errors.

murderfs 6y ago
https://devblogs.microsoft.com/cppblog/addresssanitizer-asan-for-windows-with-msvc/ https://devblogs.microsoft.com/cppblog/addresssanitizer-asan...
- quelsolaar 6y ago
  I would like to be able to query it. Windows used to support IsBadReadPtr and IsBadWritePtr but they have been deprecated. I'm not aware of something similar on Linux.
  - tinus_hn 6y ago
    It’s not very useful, if a pointer is bad and you access it you’ll crash anyway. The problem is when the pointer is manipulated to be good, but not pointing at what you expect.
    - quelsolaar 6y ago
      Let me give you an example: Lets say you have an API that implements a function similar to memcpy. It interacts with more then one user provided pointer. Lets also assume that the API is something big and complicated like a graphics driver, possibly closed source. If you give it a broken pointer it may crash, but it wont tell you what pointer was bad and in what way. Maybe no pointer was bad, maybe the implementation is broken, or some other param is broken. An API can check other params to make sure they are valid, and many APIs do, bu its not yet possible to check poiners. You have to dig in to the implementation to do that and that's a lot of work even if its opensource. Crashing in your code is great, because the debugger will tell you what is going on, but crashing in someone elses code is much harder to debug. Worse, the code may not crash at all, it may over write another allocation by the same process. The implementation cant check the mallocs, it only crashes if the pointer steps outside of a memory block owned by the process.
      - tinus_hn 6y ago
        Why would an api be able to tell you anything the debugger can’t? If you want to do what you describe you have to make such changes that the language really isn’t c anymore.
      - quelsolaar 6y ago
        > Why would an api be able to tell you anything the debugger can’t? An error message is much easier to read then a stacktrace, especially if you dont have source-code. > If you want to do what you describe you have to make such changes that the language really isn’t c anymore. I dont want any changes to C. I just want to be able to query a process allocations. the OS has to keep the information in order to implement realloc and free, so the information is already there.
        tinus_hn 6y ago
        The operating system doesn’t handle malloc, realloc and free. That task is handled by the C library. All the operating system provides is a way for these functions to claim and release large areas of memory. Applications use malloc to allocate some memory they then use to store a collection of data. Pointers often don’t point to the exact start of that memory area, which is what you need to do memory management. So from a pointer you can’t necessarily realloc or free. And the problem is, if a buffer overruns for instance, you end up writing memory that is allocated and valid, just not for the thing the developer thought. Alternative implementations of malloc exist that try to help with this, by encasing memory between guard pages. Also, with for example glibc malloc you can use the MALLOC_TRACE environment variable to get a log of all allocations.
Hydraulix989 6y ago
Well, I don't know how NT does it, but for POSIX, malloc and free at the OS layer are fixed-length pages from something like sbrk, not arbitrary blocks. The real implementations are in user space in libc. If I had to guess, NT does something similar though. I don't think the OS should be in charge of memory safety in this way since this can all be done in user space.
- quelsolaar 6y ago
  I'm doing it as much as i can in userspace, but i do run in to some limitations: thes say if i want to implement a library and i want a debug/developer mode where i can check pointers given to the API by an application i didn't write. In that case i cant wrap that applications memory allocations to keep track of memory.
tinus_hn 6y ago
Check out Valgrind, it kind of does what you want. For as far as it is possible.
- quelsolaar 6y ago
  I have used Valgrind, Its nice but on the slow side. gflags is another great option. I would still like more options for instrumenting my own code. I have a fairly capable wrapper that takes care of most of my memory issues: http://gamepipeline.org/forge_Debugging_.html http://gamepipeline.org/forge_Debugging_.html Most of why i want this kind of functionality is not to test my code but to be able to build tools. I have written a live memory viewer/poker. Memory query would really help that kind of effort: https://www.youtube.com/watch?v=pvkn9Xz-xks https://www.youtube.com/watch?v=pvkn9Xz-xks
bregma 6y ago
You're describing a very intrusive garbage collector. So, a weakly typed language with garbage collection? Where have we heard of that before? I can imagine that if most of your code is for checking that the remaining fraction reads correctly from rvalues and writes correctly to lvalues and each check requires a round trip through a context switch, you're not going to be wringing very much productivity out of either your hardware as the CPU thrashes like mad or the one developer you manage to keep on staff.
- quelsolaar 6y ago
  Its not a garbage collector. All i want is to be able to query the list of allocation made by a process. Performance would not need to be great, I would even assume it would be quite bad. But being able to write APIs, that have debug mode that tells you precisely when you give a API call broken pointers would be very useful.
  - bregma 6y ago
    The ability to know which memory is valid and which is garbage is indistinguishable from what a garbage collector does.
Const-me 6y ago
> Is it readable? Is it writable? VirtualQuery but the granularity is coarse, pages are 4kb. > Where does the memory allocation start and end? HeapSize if that's on heap, and you know the heap. > Is the memory in the heap or stack, or a memory mapped file? QueryVirtualMemoryInformation > APIs that have built in tests (in debug mode) where all pointers are checked, and if anything is broken the API can fail gracefully and give out precise errors. Visual C++ in debug builds does lots of runtime checks on all levels, C runtime, C++ runtime, and even compiler. For instance, they fill uninitialized and freed memory with magic numbers.
- quelsolaar 6y ago
  Visual studio is great at catching loads of stuff in debug mode. HeapSize is close, but cant be used to check incoming pointers since it has to be the start of the allocation, and again it doesn't recognize the stack. VirtualQuery is better but as you say a little course.
- pjmlp 6y ago
  Indeed, unfortunately outside of Windows, there isn't that much culture in doing that. C++ Builder is also quite similar.
deleted 6y ago
[deleted]

murderfs 6y ago

https://devblogs.microsoft.com/cppblog/addresssanitizer-asan-for-windows-with-msvc/ https://devblogs.microsoft.com/cppblog/addresssanitizer-asan...

quelsolaar 6y ago
I would like to be able to query it. Windows used to support IsBadReadPtr and IsBadWritePtr but they have been deprecated. I'm not aware of something similar on Linux.
- tinus_hn 6y ago
  It’s not very useful, if a pointer is bad and you access it you’ll crash anyway. The problem is when the pointer is manipulated to be good, but not pointing at what you expect.
  - quelsolaar 6y ago
    Let me give you an example: Lets say you have an API that implements a function similar to memcpy. It interacts with more then one user provided pointer. Lets also assume that the API is something big and complicated like a graphics driver, possibly closed source. If you give it a broken pointer it may crash, but it wont tell you what pointer was bad and in what way. Maybe no pointer was bad, maybe the implementation is broken, or some other param is broken. An API can check other params to make sure they are valid, and many APIs do, bu its not yet possible to check poiners. You have to dig in to the implementation to do that and that's a lot of work even if its opensource. Crashing in your code is great, because the debugger will tell you what is going on, but crashing in someone elses code is much harder to debug. Worse, the code may not crash at all, it may over write another allocation by the same process. The implementation cant check the mallocs, it only crashes if the pointer steps outside of a memory block owned by the process.
    - tinus_hn 6y ago
      Why would an api be able to tell you anything the debugger can’t? If you want to do what you describe you have to make such changes that the language really isn’t c anymore.
    - quelsolaar 6y ago
      > Why would an api be able to tell you anything the debugger can’t? An error message is much easier to read then a stacktrace, especially if you dont have source-code. > If you want to do what you describe you have to make such changes that the language really isn’t c anymore. I dont want any changes to C. I just want to be able to query a process allocations. the OS has to keep the information in order to implement realloc and free, so the information is already there.
      - tinus_hn 6y ago
        The operating system doesn’t handle malloc, realloc and free. That task is handled by the C library. All the operating system provides is a way for these functions to claim and release large areas of memory. Applications use malloc to allocate some memory they then use to store a collection of data. Pointers often don’t point to the exact start of that memory area, which is what you need to do memory management. So from a pointer you can’t necessarily realloc or free. And the problem is, if a buffer overruns for instance, you end up writing memory that is allocated and valid, just not for the thing the developer thought. Alternative implementations of malloc exist that try to help with this, by encasing memory between guard pages. Also, with for example glibc malloc you can use the MALLOC_TRACE environment variable to get a log of all allocations.

Hydraulix989 6y ago

Well, I don't know how NT does it, but for POSIX, malloc and free at the OS layer are fixed-length pages from something like sbrk, not arbitrary blocks. The real implementations are in user space in libc. If I had to guess, NT does something similar though. I don't think the OS should be in charge of memory safety in this way since this can all be done in user space.

quelsolaar 6y ago
I'm doing it as much as i can in userspace, but i do run in to some limitations: thes say if i want to implement a library and i want a debug/developer mode where i can check pointers given to the API by an application i didn't write. In that case i cant wrap that applications memory allocations to keep track of memory.

tinus_hn 6y ago

Check out Valgrind, it kind of does what you want. For as far as it is possible.

quelsolaar 6y ago
I have used Valgrind, Its nice but on the slow side. gflags is another great option. I would still like more options for instrumenting my own code. I have a fairly capable wrapper that takes care of most of my memory issues: http://gamepipeline.org/forge_Debugging_.html http://gamepipeline.org/forge_Debugging_.html Most of why i want this kind of functionality is not to test my code but to be able to build tools. I have written a live memory viewer/poker. Memory query would really help that kind of effort: https://www.youtube.com/watch?v=pvkn9Xz-xks https://www.youtube.com/watch?v=pvkn9Xz-xks

bregma 6y ago

You're describing a very intrusive garbage collector. So, a weakly typed language with garbage collection? Where have we heard of that before? I can imagine that if most of your code is for checking that the remaining fraction reads correctly from rvalues and writes correctly to lvalues and each check requires a round trip through a context switch, you're not going to be wringing very much productivity out of either your hardware as the CPU thrashes like mad or the one developer you manage to keep on staff.

quelsolaar 6y ago
Its not a garbage collector. All i want is to be able to query the list of allocation made by a process. Performance would not need to be great, I would even assume it would be quite bad. But being able to write APIs, that have debug mode that tells you precisely when you give a API call broken pointers would be very useful.
- bregma 6y ago
  The ability to know which memory is valid and which is garbage is indistinguishable from what a garbage collector does.

Const-me 6y ago

> Is it readable? Is it writable? VirtualQuery but the granularity is coarse, pages are 4kb. > Where does the memory allocation start and end? HeapSize if that's on heap, and you know the heap. > Is the memory in the heap or stack, or a memory mapped file? QueryVirtualMemoryInformation > APIs that have built in tests (in debug mode) where all pointers are checked, and if anything is broken the API can fail gracefully and give out precise errors. Visual C++ in debug builds does lots of runtime checks on all levels, C runtime, C++ runtime, and even compiler. For instance, they fill uninitialized and freed memory with magic numbers.

quelsolaar 6y ago
Visual studio is great at catching loads of stuff in debug mode. HeapSize is close, but cant be used to check incoming pointers since it has to be the start of the allocation, and again it doesn't recognize the stack. VirtualQuery is better but as you say a little course.
pjmlp 6y ago
Indeed, unfortunately outside of Windows, there isn't that much culture in doing that. C++ Builder is also quite similar.

deleted 6y ago

[deleted]