3 ms·
That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list ab
by uroni 3mo ago
That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list about it). The performance advantage is probably negligible in many cases (since malloc implementations often already cache), while causing confusing memory usage behavior.
Idk, if it was your issue, but for long running write transactions it doesn't spill to disk. So you have all the changes being written to disk at the end of the transaction. One would think enabling write mapping fixes this, but it needs to mark all the pages as clean before commit, so same effect there.
I fixed this for 0.9 here https://github.com/uroni/hs5/tree/main/external/lmdb https://github.com/uroni/hs5/tree/main/external/lmdb . Will have to investigate if it is improved with 1.0, or if I need to redo the changes.
Edit: Just noticed that the issue is about free list in the file. Never had a problem with that, but I also had to replace that MIDL structure with something more scalable for the spilling.
- markasoftware 3mo agoFWIW I had this issue even with the MDB_NOSYNC flag so it shouldnt be force flushing to disk unless I'm out of ram or whatever
- hyc_symas 3mo agoLMDB 1.0 no longer uses a P_DIRTY flag, it no longer has to explicitly mark pages as clean.
- keynha 3mo agoDropping the explicit P_DIRTY flag in 1.0 is a neat change. What tracks which pages still need to be written back at commit now that the flag is gone?
- hyc_symas 3mo agoThe txnID was added to the page header to enable support for incremental backup. As a consequence, it's sufficient to compare a page's txnID to the current txnID to know if it's dirty or not, and spilled pages don't need a cleanup pass to clear their dirty bit on commit so commits of large txns are faster now.
- hyc_symas 3mo agoBy the way, you're wrong on both points - the cache of page mallocs is not infinite, and it does spill dirty pages to disk when necessary. And the latter is what bounds the number of malloc'd pages.