4 ms·

Translation: This is a euphemism for saying Linux blatantly violates the language specification with no remorse. (Yes, I realize Linux is the kernel, etc.)

Liquid_Fire 7y ago
Which part of the specification does this violate?
- mehrdadn 7y ago
  Like, every part? It doesn't actually allocate the space requested, it doesn't indicate failure by returning NULL (or by any other reliable means for that matter)...
  - Liquid_Fire 7y ago
    I had a look, and the C standard doesn't actually say anything about indicating failure. Here is all it says about malloc: > The malloc function allocates space for an object whose size is specified by size and whose value is indeterminate. > The malloc function returns either a null pointer or a pointer to the allocated space. I think it's certainly debatable whether overcommitting while lazily allocating counts as actually allocating.
    - mehrdadn 7y ago
      > I had a look, and the C standard doesn't actually say anything about indicating failure. > The malloc function returns either a null pointer or a pointer to the allocated space. Uhm, either malloc returns a pointer to the allocated space, or it returns null. I don't see what's so complicated about this. There's no provision for "return a non-null pointer to unallocated space".
      - brianfen 7y ago
        The malloc() function requests memory from the kernel using mmap(). If mmap() returns successfully, it means that the kernel is saying that the memory is allocated. So from malloc()'s perspective the memory allocation has been successful and it must return a non-null pointer to the caller. The standard does not require malloc() to distrust what the kernel said and try to actually write to that memory to double-check if there is any physical memory mapped to it or not. The standard also does not impose anything on the kernel. I see no section of the standard being violated here. Section 7.22.3.1: The order and contiguity of storage allocated by successive calls to the aligned_alloc, calloc, malloc, and realloc functions is unspecified. The pointer returned if the allocation succeeds is suitably aligned so that it may be assigned to a pointer to any type of object with a fundamental alignment requirement and then used to access such an object or an array of such objects in the space allocated (until the space is explicitly deallocated). The lifetime of an allocated object extends from the allocation until the deallocation. Each such allocation shall yield a pointer to an object disjoint from any other object. The pointer returned points to the start (lowest byte address) of the allocated space. If the space cannot be allocated, a null pointer is returned. If the size of the space requested is zero, the behavior is implementation-defined: either a null pointer is returned, or the behavior is as if the size were some nonzero value, except that the returned pointer shall not be used to access an object. Section 7.22.3.4: The malloc function allocates space for an object whose size is specified by size and whose value is indeterminate.
        mehrdadn 7y ago
        The standard literally could not care less if there's a kernel underneath. It doesn't care how malloc is implemented or what the kernel, if any, looks like "from its perspective". The perspective that has relevance is that of the caller of malloc. From your own quotes: > The pointer returned if the allocation succeeds is suitably aligned so that [...] > If the space cannot be allocated, a null pointer is returned. If you use mmap and "from your perspective" that's considered success then it's your perspective that's faulty. The standard is literally telling you right here^ there are 2 possibilities: either you allocate the space and return a pointer to the space, or you don't and you return null. There is no third option of "space cannot be allocated but you return non-null anyway". That's quite literally the end of the story.
  - brianfen 7y ago
    You can't be serious about "every part". This definitely does not violate section 1.1 which only talks about the scope of the standard. It also definitely does not violate 1.2 which talks about what is not within the scope of the standard. So once again, can you cite the exact section number from the standard that you think is being violated here?
    - mehrdadn 7y ago
      See reply to sibling comment.
  - KirinDave 7y ago
    There is a deep irony in folks choosing to implement one of the few things the C standard DOES clearly define by papering over it with an alternative memory model. Ironically, it's a memory model that greatly reduces the performance of modern systems, and als impacts its safety.
potyl 7y ago
> [...] Linux blatantly violates the language specification [...] We can't expect a kernel to be aware of all language specifications. Yes, I know that in this case C is both the program's and the Kernel's language in this example but even then. Languages go through iterations (versions) and a Kernel can't be required to obey all languages which might run under it.
- mehrdadn 7y ago
  If you're suggesting Linux did not intend its malloc implement the corresponding function in the C standard then you're just lying to yourself. That's exactly why it's there and that's exactly what it's used for.
  - cesarb 7y ago
    > If you're suggesting Linux did not intend its malloc implement Linux (the kernel) doesn't have malloc. It's part of the C library, which is a completely separate project. What Linux implements is brk and mmap.
    - mehrdadn 7y ago
      Playing loose with the word Linux here. Play along a bit maybe?
simias 7y ago
Linux is a kernel, not a C runtime so that's not really relevant. You say you realize that but then what are you arguing for exactly? Arguably you could complain that the glibc (or whatever libc you're using) is not working around that by, for instance, scrubbing the pages in malloc() to force the kernel to allocate memory. But even then I'm not convinced that anything here is non-standard, at worse maybe we're in a bit of a grey area. As long as the kernel maintains its smoke and mirrors whether and how it allocates memory is irrelevant from the point of view of the standard. The C language has a rather simplistic memory model, it doesn't impose a lot on the implementation. Now the problem occurs when the program attempts to access virtually-allocated memory and the kernel realizes that it can't find any physical memory to map it to. In this situation several things can happen but in general the process will be killed. Is it against the standard for the OS to kill a program for arbitrary reasons? I can't imagine why. It could also freeze the program, waiting for more memory to become available. Again, not against the standard as far as I can tell. Or maybe kill some other program to free memory. If you have some specific part of the C standard in mind please do tell, I always find these language lawyering arguments interesting, somehow.
- mehrdadn 7y ago
  > If you have some specific part of the C standard in mind please do tell See here: https://news.ycombinator.com/item?id=20145604 https://news.ycombinator.com/item?id=20145604 Also note the POSIX standard: Upon successful completion with size not equal to 0, malloc() shall return a pointer to the allocated space. If size is 0, either a null pointer or a unique pointer that can be successfully passed to free() shall be returned. Otherwise, it shall return a null pointer and set errno to indicate the error. In neither case is there any provision for returning a non-null pointer to anything other than an allocated block of memory of at least the given size.
  - simias 7y ago
    Thank you for your reply but I still don't buy it. As far as the program is concerned it is returned a memory block, what this "memory" is effectively behind the scenes is none of the standard's business. As long as the implementation manages to maintain the illusion it's perfectly fine AFAIK. The problem is when this breaks down and the kernel realizes that it can no longer maintain the masquerade. If at this point it did something stupid like map this block nowhere or remap something that's already been allocated and let the program continue like this, then yes that would be a clear violation of the contract. But it does no such thing, it kills the program instead. If there's no program then there's no problem. At no point did a single C instruction get executed in an environment that did not maintain a coherent memory model. >In neither case is there any provision for returning a non-null pointer to anything other than an allocated block of memory of at least the given size. A pointer to virtual memory is a pointer to memory. The rest is an implementation detail. If, when dereferenced, the kernel issues an order to Amazon for more RAM and waits for it to be installed to resume the execution, that's none of the C standard's business.
dooglius 7y ago
The C spec actually states at the beginning, "This International Standard does not specify ... the size or complexity of a program and its data that will exceed the capacity of any specific data-processing system or the capacity of a particular processor;"
- mehrdadn 7y ago
  Yes and that's why malloc can return null.
kazinator 7y ago
> Linux blatantly violates the language specification with no remorse. We may be able to argue that it doesn't. ISO C says in 1. Scope, this: This International Standard does not specify [...] — the size or complexity of a program and its data that will exceed the capacity of any specific data-processing system or the capacity of a particular processor. It seems that these weasel words have an interpretation that can be bent around overcommitted memory allocation.
- mehrdadn 7y ago
  No it wouldn't. They clearly included the ability to return null to allow for when the data exceeds the capacity of the system. There is no provision for returning a pointer to memory that cannot be used.
  - kazinator 7y ago
    You might think, but that's not how how conformance (of an ISO C implementation to the standard) works. For an implementation to be deemed conforming, it just has to be demonstrated to successfully translate and execute one program that tests each of the minimum implementation limits. See 5.2.4.1 in http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf Only if no such program can be found can we then conclude that the implementation is nonconforming (and if the reason for not finding such a program is this overcommit issue, then we can blame that issue). Your Linux system is indeed nonconforming if it is so low on memory that, for instance, no C program can allocate a 65535 byte object (that can be actually initialized, and used: a real object). Basically the memory situation has to be so severe that it takes the implementation below the minimum limits, whereby we can clearly demonstrate nonconformance.

Liquid_Fire 7y ago

Which part of the specification does this violate?

mehrdadn 7y ago
Like, every part? It doesn't actually allocate the space requested, it doesn't indicate failure by returning NULL (or by any other reliable means for that matter)...
- Liquid_Fire 7y ago
  I had a look, and the C standard doesn't actually say anything about indicating failure. Here is all it says about malloc: > The malloc function allocates space for an object whose size is specified by size and whose value is indeterminate. > The malloc function returns either a null pointer or a pointer to the allocated space. I think it's certainly debatable whether overcommitting while lazily allocating counts as actually allocating.
  - mehrdadn 7y ago
    > I had a look, and the C standard doesn't actually say anything about indicating failure. > The malloc function returns either a null pointer or a pointer to the allocated space. Uhm, either malloc returns a pointer to the allocated space, or it returns null. I don't see what's so complicated about this. There's no provision for "return a non-null pointer to unallocated space".
    - brianfen 7y ago
      The malloc() function requests memory from the kernel using mmap(). If mmap() returns successfully, it means that the kernel is saying that the memory is allocated. So from malloc()'s perspective the memory allocation has been successful and it must return a non-null pointer to the caller. The standard does not require malloc() to distrust what the kernel said and try to actually write to that memory to double-check if there is any physical memory mapped to it or not. The standard also does not impose anything on the kernel. I see no section of the standard being violated here. Section 7.22.3.1: The order and contiguity of storage allocated by successive calls to the aligned_alloc, calloc, malloc, and realloc functions is unspecified. The pointer returned if the allocation succeeds is suitably aligned so that it may be assigned to a pointer to any type of object with a fundamental alignment requirement and then used to access such an object or an array of such objects in the space allocated (until the space is explicitly deallocated). The lifetime of an allocated object extends from the allocation until the deallocation. Each such allocation shall yield a pointer to an object disjoint from any other object. The pointer returned points to the start (lowest byte address) of the allocated space. If the space cannot be allocated, a null pointer is returned. If the size of the space requested is zero, the behavior is implementation-defined: either a null pointer is returned, or the behavior is as if the size were some nonzero value, except that the returned pointer shall not be used to access an object. Section 7.22.3.4: The malloc function allocates space for an object whose size is specified by size and whose value is indeterminate.
      - mehrdadn 7y ago
        The standard literally could not care less if there's a kernel underneath. It doesn't care how malloc is implemented or what the kernel, if any, looks like "from its perspective". The perspective that has relevance is that of the caller of malloc. From your own quotes: > The pointer returned if the allocation succeeds is suitably aligned so that [...] > If the space cannot be allocated, a null pointer is returned. If you use mmap and "from your perspective" that's considered success then it's your perspective that's faulty. The standard is literally telling you right here^ there are 2 possibilities: either you allocate the space and return a pointer to the space, or you don't and you return null. There is no third option of "space cannot be allocated but you return non-null anyway". That's quite literally the end of the story.
- brianfen 7y ago
  You can't be serious about "every part". This definitely does not violate section 1.1 which only talks about the scope of the standard. It also definitely does not violate 1.2 which talks about what is not within the scope of the standard. So once again, can you cite the exact section number from the standard that you think is being violated here?
  - mehrdadn 7y ago
    See reply to sibling comment.
- KirinDave 7y ago
  There is a deep irony in folks choosing to implement one of the few things the C standard DOES clearly define by papering over it with an alternative memory model. Ironically, it's a memory model that greatly reduces the performance of modern systems, and als impacts its safety.

potyl 7y ago

> [...] Linux blatantly violates the language specification [...] We can't expect a kernel to be aware of all language specifications. Yes, I know that in this case C is both the program's and the Kernel's language in this example but even then. Languages go through iterations (versions) and a Kernel can't be required to obey all languages which might run under it.

mehrdadn 7y ago
If you're suggesting Linux did not intend its malloc implement the corresponding function in the C standard then you're just lying to yourself. That's exactly why it's there and that's exactly what it's used for.
- cesarb 7y ago
  > If you're suggesting Linux did not intend its malloc implement Linux (the kernel) doesn't have malloc. It's part of the C library, which is a completely separate project. What Linux implements is brk and mmap.
  - mehrdadn 7y ago
    Playing loose with the word Linux here. Play along a bit maybe?

simias 7y ago

Linux is a kernel, not a C runtime so that's not really relevant. You say you realize that but then what are you arguing for exactly? Arguably you could complain that the glibc (or whatever libc you're using) is not working around that by, for instance, scrubbing the pages in malloc() to force the kernel to allocate memory. But even then I'm not convinced that anything here is non-standard, at worse maybe we're in a bit of a grey area. As long as the kernel maintains its smoke and mirrors whether and how it allocates memory is irrelevant from the point of view of the standard. The C language has a rather simplistic memory model, it doesn't impose a lot on the implementation. Now the problem occurs when the program attempts to access virtually-allocated memory and the kernel realizes that it can't find any physical memory to map it to. In this situation several things can happen but in general the process will be killed. Is it against the standard for the OS to kill a program for arbitrary reasons? I can't imagine why. It could also freeze the program, waiting for more memory to become available. Again, not against the standard as far as I can tell. Or maybe kill some other program to free memory. If you have some specific part of the C standard in mind please do tell, I always find these language lawyering arguments interesting, somehow.

mehrdadn 7y ago
> If you have some specific part of the C standard in mind please do tell See here: https://news.ycombinator.com/item?id=20145604 https://news.ycombinator.com/item?id=20145604 Also note the POSIX standard: Upon successful completion with size not equal to 0, malloc() shall return a pointer to the allocated space. If size is 0, either a null pointer or a unique pointer that can be successfully passed to free() shall be returned. Otherwise, it shall return a null pointer and set errno to indicate the error. In neither case is there any provision for returning a non-null pointer to anything other than an allocated block of memory of at least the given size.
- simias 7y ago
  Thank you for your reply but I still don't buy it. As far as the program is concerned it is returned a memory block, what this "memory" is effectively behind the scenes is none of the standard's business. As long as the implementation manages to maintain the illusion it's perfectly fine AFAIK. The problem is when this breaks down and the kernel realizes that it can no longer maintain the masquerade. If at this point it did something stupid like map this block nowhere or remap something that's already been allocated and let the program continue like this, then yes that would be a clear violation of the contract. But it does no such thing, it kills the program instead. If there's no program then there's no problem. At no point did a single C instruction get executed in an environment that did not maintain a coherent memory model. >In neither case is there any provision for returning a non-null pointer to anything other than an allocated block of memory of at least the given size. A pointer to virtual memory is a pointer to memory. The rest is an implementation detail. If, when dereferenced, the kernel issues an order to Amazon for more RAM and waits for it to be installed to resume the execution, that's none of the C standard's business.

dooglius 7y ago

The C spec actually states at the beginning, "This International Standard does not specify ... the size or complexity of a program and its data that will exceed the capacity of any specific data-processing system or the capacity of a particular processor;"

mehrdadn 7y ago
Yes and that's why malloc can return null.

kazinator 7y ago

> Linux blatantly violates the language specification with no remorse. We may be able to argue that it doesn't. ISO C says in 1. Scope, this: This International Standard does not specify [...] — the size or complexity of a program and its data that will exceed the capacity of any specific data-processing system or the capacity of a particular processor. It seems that these weasel words have an interpretation that can be bent around overcommitted memory allocation.

mehrdadn 7y ago
No it wouldn't. They clearly included the ability to return null to allow for when the data exceeds the capacity of the system. There is no provision for returning a pointer to memory that cannot be used.
- kazinator 7y ago
  You might think, but that's not how how conformance (of an ISO C implementation to the standard) works. For an implementation to be deemed conforming, it just has to be demonstrated to successfully translate and execute one program that tests each of the minimum implementation limits. See 5.2.4.1 in http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf Only if no such program can be found can we then conclude that the implementation is nonconforming (and if the reason for not finding such a program is this overcommit issue, then we can blame that issue). Your Linux system is indeed nonconforming if it is so low on memory that, for instance, no C program can allocate a 65535 byte object (that can be actually initialized, and used: a real object). Basically the memory situation has to be so severe that it takes the implementation below the minimum limits, whereby we can clearly demonstrate nonconformance.