3 ms·

> A prior poster made a claim that they can read a file into memory, and that it won’t actually use any additional memory because the kernel will “automatically

• ninkendo 3y ago
  You're basically just gish galloping at this point and there's no need to respond to you any more. All your points about swap are irrelevant to the discussion. All your points about disk cache are irrelevant to the discussion. You have a very, very, very incorrect understanding of how operating system kernels work if you think mmap() is just a less efficient read(): > Every word of that is nonsense. The disk is attached to a serial bus. Even if you're using fancy nvme "disks" there's a queue that operates in a (reasonably) serial fashion. The reason mmap() is referred to zero-copy is because it can reuse the block cache if it has been recently downloaded -- but if the data is paged out, there is absolutely a copy and it's more expensive than just read() by a long way. Please do a basic web search for what "virtual memory" means. You seem to think that handwavey nonsense about disk cache means that read() doesn't copy the data into your working set. You should look at the manpage for read() and maybe ponder why it requires you to pass your own buffer. This buffer would have to be something you've malloc()'d ahead of time. Hence why you're using more memory by using read() than you would using mmap() > You only have so much storage, this is life, but I can tell you as someone who does operate a 1tb of ram machine that downloads 300gb of logfiles every day, read() and write() work just fine -- I just can't speak for python (or why python people don't do it) because i don't like python. You should definitely learn what the mmap syscall does and why it exists. You really don't need to use 300gb of RAM to read a 300gb log file. You should probably look up how text editors like vim actually work, and why you can seek to the end of a 300gb log file without vim taking up a bunch of RAM. Maybe you've never been curious about this before, I dunno. Try making a 20gb file called "bigfile", and run this C program: #include <sys/mman.h> #include <sys/stat.h> #include <fcntl.h> #include <stdio.h> #include <stdlib.h> #include <inttypes.h> #include <unistd.h> int main(int argc, char *argv[]) { char *mmap_buffer; int fd; struct stat sb; ssize_t s; fd = open("./bigfile", O_RDONLY); fstat(fd, &sb); // get the size // do the mmap mmap_buffer = mmap(NULL, sb.st_size, PROT_READ, MAP_PRIVATE, fd, 0); uint8_t sum; // Ensure the whole buffer is touched by doing a dumb math operation on every byte for (size_t i = 0; i < sb.st_size; ++i) { sum += mmap_buffer[i]; // overflow is fine, sorta a dumb checksum } fprintf(stderr, "done! sum=%d\n", sum); sleep(1000); } And wait for "done!" to show up in stderr. It will sleep 1000 seconds waiting for you to ctrl+c at this point. At this point we will have (1) mmap'd the entire file into memory, and (2) read the whole thing sequentially, adding each byte to a `sum` variable (with expected overflow.) While it's sleeping, check /proc/<pid>/status and take note of the memory stats. You'll see that VmSize is as big as the file you read in (for me, bigfile is more than 20GB): VmSize: 20974160 kB But the actual resident set is 968kb: VmRSS: 968 kB So, my program is using 968 kB even though it has a 20GB in-memory buffer that just read the whole file in! My system only has 16GB of RAM and swap is disabled. How is this possible? Because mmap lets you do this. The kernel will read in pages from bigfile on demand, but is also free to free them at any point. There is no controversy here, every modern operating system has supported this for decades. Compare this to a similar program using read(): #include <sys/mman.h> #include <sys/stat.h> #include <fcntl.h> #include <stdio.h> #include <stdlib.h> #include <inttypes.h> #include <unistd.h> int main(int argc, char *argv[]) { int fd; struct stat sb; ssize_t s; char *read_buffer; fd = open("./bigfile", O_RDONLY); fstat(fd, &sb); // get the size // do the read read_buffer = malloc(sb.st_size); read(fd, read_buffer, sb.st_size); uint8_t sum; // Ensure the whole buffer is touched by doing a dumb math operation on every byte for (size_t i = 0; i < sb.st_size; ++i) { sum += read_buffer[i]; // overflow is fine, sorta a dumb checksum } fprintf(stderr, "done! sum=%d\n", sum); sleep(1000); } And do the same thing (in this instance I shrunk bigfile to 2 GB because I don't have enough physical RAM to do this with 20GB.) You'll see this in /proc/<pid>/status: VmPeak: 2099928 kB VmSize: 2099928 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 2098056 kB VmRSS: 2098056 kB Oops! I'm using 2 GB of resident set size. If I were to do this with a file that's bigger than RAM, I'd get OOM-killed. This is why you shouldn't read() in large datasets. Your strategy is to have servers with 1TB of ram and massive amounts of swap, and I'm telling you you don't need this to process this big of files. mmap() does so without requiring things to be read into RAM ahead of time. Oh, and guess what: Take out the `sleep(1000)`, and the mmap version is faster than the read() version: $ time ./mmap done! sum=225 ./mmap 6.89s user 0.28s system 99% cpu 7.180 total $ time ./read done! sum=246 ./read 6.86s user 1.72s system 99% cpu 8.612 total Why is it faster? Because we don't have to needlessly copy the data into the process's heap. We can just read the blocks directly from the mmap'd address space, and let page faults read them in for us.
  • geocar 3y ago
    [flagged]
    • ninkendo 3y ago
      Begone troll. (Edit: because I noticed this too, and it got me curious, the reason for the incorrect result is that I didn't check the result of the read() call, and it was actually reading fewer bytes, by a small handful. read() is allowed to do this, and it's up to callers to call it again. It was reading 2147479552 bytes when the file size was 2147483648 bytes. If anything this should have made the read implementation faster, but mmap still wins even though it read more bytes. A fixed version follows, and now produces the same "sum" as the mmap): #include <sys/mman.h> #include <sys/stat.h> #include <fcntl.h> #include <stdio.h> #include <stdlib.h> #include <inttypes.h> #include <unistd.h> int main(int argc, char *argv[]) { int fd; struct stat sb; ssize_t s; ssize_t read_result; ssize_t bytes_read; char *read_buffer; fd = open("./bigfile", O_RDONLY); fstat(fd, &sb); // get the size // do the read read_buffer = malloc(sb.st_size); bytes_read = 0; do { read_result = read(fd, read_buffer + bytes_read, sb.st_size - bytes_read); bytes_read += read_result; } while (read_result != 0); uint8_t sum; // Ensure the whole buffer is touched by doing a dumb math operation on every byte for (size_t i = 0; i < sb.st_size; ++i) { sum += read_buffer[i]; // overflow is fine, sorta a dumb checksum } fprintf(stderr, "done! sum=%d", sum); }
      • geocar 3y ago
        > because I noticed this too When it was pointed out to you. Maybe if you read what I wrote and try benchmarking what I am saying you will “notice” more of your errors.