4 ms·

I was heavily involved in the push to switch Neo4j off of memory mapping and on to an in house page cache. In retrospect, I’d say it was something like 50% dri

I was heavily involved in the push to switch Neo4j off of memory mapping and on to an in house page cache. In retrospect, I’d say it was something like 50% driven by Java memory mapping having insane issues on Windows, 20% Java memory mapping having insane issues on every platform, 25% me being a noob and 5% actual issues with memory mapping on Linux. I think if could say “this DB is posix only” I would try memory mapping for the next DB I build

kjeetgill 4y ago
> Java memory mapping having insane issues on every platform I'm very interested, what issues? I've occasionally considered using mmap for a few things, but never had justification to. I'm curious what issues I might have run into. Did you use JNI/JNA to mmap or via RandomAccessFile?
- RyanHamilton 4y ago
  2GB limit. Cost of indirect addressing. Inability to address RAM and disk using same method calls without overhead. I've implemented a modest database in both C and Java with memory mapping. It's hard to explain the feeling of programming both but in general: with java I found myself trying to add intelligence at the cost of complexity to make things run faster. In C I could get speed by removing overheads and doing less work.
  - kjeetgill 4y ago
    Thanks! Insightful. I wonder if you could get what you needed via unsafe but I agree: that's fighting the system in Java.
apavlo 4y ago
> I think if could say “this DB is posix only” I would try memory mapping for the next DB I build Great! Please reach out to us after you learn (again) that this is a bad idea!
icedchai 4y ago
I worked at a company that built a custom database on top of mmap, back in the early 2000's. It was single threaded, intended for very limited OLTP use cases, not a generic DBMS, but worked wonderfully for those use cases. A couple gigabytes was more than enough address space. During this time, a machine with 4 gigabytes RAM was considered "high end"! It only ran on a couple of posix platforms. At startup, it would read everything into memory so everything was hot, ready to go. The "database" was intended to run on a dedicated node with no other applications running. It also kept a write ahead log for transaction recovery.

kjeetgill 4y ago

> Java memory mapping having insane issues on every platform I'm very interested, what issues? I've occasionally considered using mmap for a few things, but never had justification to. I'm curious what issues I might have run into. Did you use JNI/JNA to mmap or via RandomAccessFile?

RyanHamilton 4y ago
2GB limit. Cost of indirect addressing. Inability to address RAM and disk using same method calls without overhead. I've implemented a modest database in both C and Java with memory mapping. It's hard to explain the feeling of programming both but in general: with java I found myself trying to add intelligence at the cost of complexity to make things run faster. In C I could get speed by removing overheads and doing less work.
- kjeetgill 4y ago
  Thanks! Insightful. I wonder if you could get what you needed via unsafe but I agree: that's fighting the system in Java.

apavlo 4y ago

> I think if could say “this DB is posix only” I would try memory mapping for the next DB I build Great! Please reach out to us after you learn (again) that this is a bad idea!

icedchai 4y ago

I worked at a company that built a custom database on top of mmap, back in the early 2000's. It was single threaded, intended for very limited OLTP use cases, not a generic DBMS, but worked wonderfully for those use cases. A couple gigabytes was more than enough address space. During this time, a machine with 4 gigabytes RAM was considered "high end"! It only ran on a couple of posix platforms. At startup, it would read everything into memory so everything was hot, ready to go. The "database" was intended to run on a dedicated node with no other applications running. It also kept a write ahead log for transaction recovery.