4 ms·

I think i’m missing something here, could you not just mmap() the file in and let the kernel take care of memory pressure for you?

• sschnei8 3y ago
  https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/
  • rubiquity 3y ago
    Good paper and overall the right opinion but not very relevant to a relatively simple system as OP described since they weren't dealing with safety, transactions, etc.
    • legulere 3y ago
      Also not access times, so OP doesn’t care if the OS frees the pages again.
• hakunin 3y ago
  I'm no systems programmer, but I remember trying to research mmap approach. It's been 3 years, so I'm not sure what stopped me, but something didn't feel right. Perhaps it was the lack of control over how memory is used. I could clearly see how not use it, and didn't want any fluctuations. Edit: oh and I think I did come across some article like the one linked in the neighbor comment. It's starting to come back.
• lisper 3y ago
  Sure -- until your system crashes and data is left in an inconsistent state because some of it was written to non-volatile storage and some of it wasn't and your OS had no concept of transactional consistency because mmap is a leaky abstraction. [UPDATE] I missed a crucial part of the problem setup, which is that the data is read-only. That obviously moots my objection.
  • posnet 3y ago
    Not a problem here. Read the top level comment. "static key-value data. ... a const (i.e. a hash table)"
    • lisper 3y ago
      Ah. I stand corrected.
• legulere 3y ago
  And Flatbuffers for data-layout and access.