4 ms·

> On an average desktop/server system the OS would automatically take care of putting whatever fits in RAM and the rest on disk. This is not true, unless you’r

> On an average desktop/server system the OS would automatically take care of putting whatever fits in RAM and the rest on disk. This is not true, unless you’re referring to swap (which is a configuration of the system and may not be big enough to actually fit it either, many people run with only a small amount of swap or disable it altogether.) You may be referring to mmap(2), which will map the on-disk dataset to a region of memory that is paged in on-demand, but somehow I doubt that’s what OP was referring to either. If you just read() the file into memory and work on it, you’re going to be using a ton of RAM. The OS will only put “the rest on disk” if it swaps, which is a degenerate performance case, and it may not even be the dataset itself that gets swapped (the kernel may opt to swap everything else on the system to fit your dataset into RAM. All pages are equal in the eyes of the virtual memory layer, and the ranking algorithm is basically an LRU cache.)

geocar 3y ago
read() doesn't read from the disk if the blocks are already in memory.
- ninkendo 3y ago
  That’s a really misleading thing to say. If the kernel already has the thing you’re read()’ing cached, then yes the kernel can skip the disk read as an optimization. But by reading, you’re taking those bytes and putting a copy of them in the process’s heap space, which makes it no longer just a “cache”. You’re now “using memory”. read() is not mmap(). You can’t just say “oh I’ll read the file in and the OS will take care of it”. It doesn’t work that way.
  - geocar 3y ago
    > If the kernel already has the thing you’re read()’ing cached which it would do, if you have just downloaded the file. > But by reading, you’re taking those bytes and putting a copy of them in the process’s heap space i mean, you just downloaded it from the network, so unless you think mmap() can hold buffers on the network card, there's definitely going to be a copy going on. now it's downloaded, you don't need to do it again, so we're only talking the one copy here. > You can’t just say “oh I’ll read the file in and the OS will take care of it”. It doesn’t work that way. i can and do. and you've already explained swap sufficiently that i believe you know you can do exactly that also.
    - ninkendo 3y ago
      Please keep in mind the context of the discussion. A prior poster made a claim that they can read a file into memory, and that it won’t actually use any additional memory because the kernel will “automatically take care” of it somehow. This is plainly false. You come in and say something to the effect of “but it may not have to read from disk because it’s cached”, which… has nothing to do with what was being discussed. We’re not talking about whether it incurs a disk read, we’re talking about whether it will run your system out of memory trying to load it into RAM. > i mean, you just downloaded it from the network, so unless you think mmap() can hold buffers on the network card, there's definitely going to be a copy going on. now it's downloaded, you don't need to do it again, so we're only talking the one copy here. What in god’s holy name are you blathering about? If I “just downloaded it from the network”, it’s on-disk. If I mmap() the disk contents, there’s no copy going on, it’s “mapped” to disk. If I read() the contents, which is what you said I should do, then another copy of the data is now sitting in a buffer in my process’s heap. This extra copy is now "using" memory, and if I keep doing this, I will run the system out of RAM. This is characteristically different from mmap(), where a region of memory maps to a file on-disk, and contents are faulted into memory as I read them. The reason this is an extremely important distinction, is that in the mmap scenario, the kernel is free to free the read-in pages any time it wants, and they will be faulted back again in if I try to read them again. Contrast this with using read(), which makes it so the kernel can't free the pages, because they're buffers in my process's heap, and are not considered file-backed from the kernel's perspective. > i can and do. and you've already explained swap sufficiently that i believe you know you can do exactly that also. Swap is disabled on my system. Even if it wasn’t, I’d only have so much of it. Even if I had a ton of it, read()’ing 100GB of data and relying on swap to save me is going to grind the rest of the system to a halt as the kernel tries to make room for it (because the data is in my heap, and thus isn’t file-backed, so the kernel can’t just free the pages and read them back from the file I read it from.) read() is not mmap(). Please don’t conflate them.
      - SoftTalker 3y ago
        > Swap is disabled on my system. Yep I do the same. If I have a server with hundreds of GB or even TB of RAM (not uncommon these days) I'm not setting up swap. If you're exhausting that much RAM, swap is only going to delay the inevitable. Fix your program.
Chabsff 3y ago
Why do you think OP isn't refering to mmap()? Its behavior is pretty much what they describe, and a common way it's used.
- ninkendo 3y ago
  Fair enough, it’s totally possible that’s what they meant. But the complaint of “every package assumes you loaded the whole dataset in memory” seems to imply the package just naively reads the file in. I mean, if the package was mmapping it, they probably wouldn’t have had much trouble with memory enough for it to be an issue they’ve had to complain about. Also, you may not always have the luxury of mmap()’ing, if you’re reading data from a socket (network connection, stdout from some other command, etc.) I don’t do much python but I used to do a lot of ruby, and it was rare to see anyone mmap’ing anything, most people just did File.read(path) and called it a day. If the norm in the python ecosystem is to mmap things, then you’re probably right.

geocar 3y ago

read() doesn't read from the disk if the blocks are already in memory.

ninkendo 3y ago
That’s a really misleading thing to say. If the kernel already has the thing you’re read()’ing cached, then yes the kernel can skip the disk read as an optimization. But by reading, you’re taking those bytes and putting a copy of them in the process’s heap space, which makes it no longer just a “cache”. You’re now “using memory”. read() is not mmap(). You can’t just say “oh I’ll read the file in and the OS will take care of it”. It doesn’t work that way.
- geocar 3y ago
  > If the kernel already has the thing you’re read()’ing cached which it would do, if you have just downloaded the file. > But by reading, you’re taking those bytes and putting a copy of them in the process’s heap space i mean, you just downloaded it from the network, so unless you think mmap() can hold buffers on the network card, there's definitely going to be a copy going on. now it's downloaded, you don't need to do it again, so we're only talking the one copy here. > You can’t just say “oh I’ll read the file in and the OS will take care of it”. It doesn’t work that way. i can and do. and you've already explained swap sufficiently that i believe you know you can do exactly that also.
  - ninkendo 3y ago
    Please keep in mind the context of the discussion. A prior poster made a claim that they can read a file into memory, and that it won’t actually use any additional memory because the kernel will “automatically take care” of it somehow. This is plainly false. You come in and say something to the effect of “but it may not have to read from disk because it’s cached”, which… has nothing to do with what was being discussed. We’re not talking about whether it incurs a disk read, we’re talking about whether it will run your system out of memory trying to load it into RAM. > i mean, you just downloaded it from the network, so unless you think mmap() can hold buffers on the network card, there's definitely going to be a copy going on. now it's downloaded, you don't need to do it again, so we're only talking the one copy here. What in god’s holy name are you blathering about? If I “just downloaded it from the network”, it’s on-disk. If I mmap() the disk contents, there’s no copy going on, it’s “mapped” to disk. If I read() the contents, which is what you said I should do, then another copy of the data is now sitting in a buffer in my process’s heap. This extra copy is now "using" memory, and if I keep doing this, I will run the system out of RAM. This is characteristically different from mmap(), where a region of memory maps to a file on-disk, and contents are faulted into memory as I read them. The reason this is an extremely important distinction, is that in the mmap scenario, the kernel is free to free the read-in pages any time it wants, and they will be faulted back again in if I try to read them again. Contrast this with using read(), which makes it so the kernel can't free the pages, because they're buffers in my process's heap, and are not considered file-backed from the kernel's perspective. > i can and do. and you've already explained swap sufficiently that i believe you know you can do exactly that also. Swap is disabled on my system. Even if it wasn’t, I’d only have so much of it. Even if I had a ton of it, read()’ing 100GB of data and relying on swap to save me is going to grind the rest of the system to a halt as the kernel tries to make room for it (because the data is in my heap, and thus isn’t file-backed, so the kernel can’t just free the pages and read them back from the file I read it from.) read() is not mmap(). Please don’t conflate them.
    - SoftTalker 3y ago
      > Swap is disabled on my system. Yep I do the same. If I have a server with hundreds of GB or even TB of RAM (not uncommon these days) I'm not setting up swap. If you're exhausting that much RAM, swap is only going to delay the inevitable. Fix your program.

Chabsff 3y ago

Why do you think OP isn't refering to mmap()? Its behavior is pretty much what they describe, and a common way it's used.

ninkendo 3y ago
Fair enough, it’s totally possible that’s what they meant. But the complaint of “every package assumes you loaded the whole dataset in memory” seems to imply the package just naively reads the file in. I mean, if the package was mmapping it, they probably wouldn’t have had much trouble with memory enough for it to be an issue they’ve had to complain about. Also, you may not always have the luxury of mmap()’ing, if you’re reading data from a socket (network connection, stdout from some other command, etc.) I don’t do much python but I used to do a lot of ruby, and it was rare to see anyone mmap’ing anything, most people just did File.read(path) and called it a day. If the norm in the python ecosystem is to mmap things, then you’re probably right.