10 ms·

Or you could stream parse the file. eg. parse it one chunk at a time.

• saagarjha 8y ago
  That’s what they were doing before?
  • 0xffff2 8y ago
    Before they were reading the whole file into memory and parsing it. Now, they are using mmap to read parts of the file into memory when they are used. If you're only doing a single linear pass through a file there is a third option; you could read part of the file in a single chunk at a time, parse that chunk, then read the next part of the file into the same chunk of memory. I don't know enough about the process being discussed here to know whether it needs to do lookups at random offsets in the file, but iff it doesn't then the chunked read solution could be a simpler way to reduce memory usage.