hyc_symas

9 ms·

61.
by hyc_symas 3y ago
Transactions in the filesystem API is misguided and useless. All you need is to expose command tagging to user level, probably with some fcntl: https://www.spinics.net/lists/linux-fsdevel/msg70047.html Trying to i
62.
by hyc_symas 3y ago
> Clearly, MMAP cannot achieve this Clearly you have no idea what you're talking about. With a read-only mmap you don't need to do anything, OOM won't happen*. All of the pages of the map are clean; whenever there's m
63.
by hyc_symas 3y ago
Prefix compression adds a lot of complexity. If you insert a new record that makes an existing prefix non-unique, you then have to expand and rewrite every key in the page, and that will ripple up to every parent page as well. And the expan
64.
by hyc_symas 3y ago
Yeah, we knew that LMDB would be unaffected. Other folks demonstrated that too. https://www.pugetsystems.com/labs/hpc/Intel-CPU-flaw-kernel-...
65.
by hyc_symas 3y ago
Before we go any further here, you should all read the original LMDB design paper, to get a grasp of what we traded away first. https://openldap.org/pub/ You can also watch the presentation at CMU https://w
66.
by hyc_symas 3y ago
> That worked well when the storage system was made of spinning disks, and you were incentivized to use data structures laying data out appropriate for that access pattern. It was all complementary and cooperated pretty well. > Then S
67.
by hyc_symas 3y ago
If they knew what they were talking about they wouldn't have used a raw filesystem benchmarking tool to prove their arguments. They would have used a pair of actual, comparable DBMS implementations. Single-writer is all you need if eac
68.
by hyc_symas 3y ago
Andy and I have had this conversation many times over. https://www.youtube.com/watch?v=tEa5sAh-kVk Feel free to point out specific arguments that you think need further fleshing out. Every aspect of LMDB's design and p
69.
by hyc_symas 3y ago
500 lines of code is still 500 lines of added complexity. For LMDB that'd be a 7% increase in LOCs, which would also introduce the need for manual cache configuration and tuning (further increasing complexity for the end user) and a 50
70.
by hyc_symas 3y ago
You can say the performance analysis is wrong, when you find any benchmarks showing that LMDB performance suffers because of it. Just like those paper's authors can claim using mmap increases complexity when they can find any DBMS with
71.
by hyc_symas 3y ago
> hand-wave away I/O stalls > A DBMS might be servicing many requests at once; Yes, a DBMS may be servicing many requests at once, which is why individual stalls don't matter. Other threads will be making progress. > a da
72.
by hyc_symas 3y ago
There's an obvious tradeoff. With a 4K page size, the breakeven point is around 768 bytes per record. For larger records, the WAL approach is less efficient than LMDB. For smaller records, the WAL is more efficient in terms of write am
73.
by hyc_symas 3y ago
A read txn starts with reading the last page of the file and searching backward for a valid metapage. This can be time-consuming if the previous use crashed while a large write was in progress. Yet another reason LMDB doesn't use appen
74.
by hyc_symas 3y ago
I had never read any papers on shadow paging when I wrote LMDB. I refer to it as double-buffering in the original LMDB paper because of my own prior experience writing graphics code. You can find the (L)MDB paper here https://ope
75.
by hyc_symas 3y ago
For a 4KB pagesize, the breakeven point in write amplification vs WALs is around 768 bytes. http://www.lmdb.tech/bench/ondisk/ Above that record size, LMDB is more efficient. For OpenLDAP, with LDAP entries typica
76.
by hyc_symas 3y ago
Just use OpenLDAP.
77.
by hyc_symas 3y ago
typedef all structs - yes, helps with conciseness. Use typedefs liberally, I say. But only typedef the things themselves, not pointers to the things. You can always use (type *) when you need a pointer. In particular, for function pointers,
78.
by hyc_symas 3y ago
InnoDB is a bloated pig. No, it hasn't gotten 5 orders of magnitude faster in the past 10 years. http://www.lmdb.tech/bench/memcache/ And the overhead of SQL processing vs LDAP protocol hasn't improved a
79.
by hyc_symas 3y ago
Wouldn't need to upgrade server hardware as often...
80.
by hyc_symas 3y ago
Got a link to a report on that? OpenLDAP serves queries at line rate on multi-gigabit NICs, with latencies indistinguishable from ping RTTs. Other databases aren't even close.
81.
by hyc_symas 3y ago
Still an order of magnitude slower / less efficient than OpenLDAP / LMDB.
82.
by hyc_symas 3y ago
That's all just false. Just because you're single-writer at the application level doesn't mean the OS isn't queueing enough writes to saturate storage at the device level. We've benchmarked plenty of high speed NVMe
83.
by hyc_symas 3y ago
Every HDD since the 1980s has guaranteed atomic sector writes: > Currently all hard drive/SSD manufacturers guarantee that 512 byte sector writes are atomic. As such, failure to write the 106 byte header is not something we accoun
84.
by hyc_symas 3y ago
The mention of a large shared flat address space implied no paging, to me. Maybe I just read something into it that wasn't there.
85.
by hyc_symas 3y ago
I think we've gone off on a tangent. At any rate, both LevelDB and RocksDB are still single-writer so whatever point seems to have been lost along the way.
86.
by hyc_symas 3y ago
> The page says "updates," not "writes." Updates are a constrained form of write where you are writing to an existing key. Updates, importantly, do not affect your index structure, while writes do. OK, I see your poin
87.
by hyc_symas 3y ago
> RocksDB/LevelDB which implicitly assume that your database is a lot bigger than RAM. Where are you getting that assumption from? LevelDB was built to be used in Google Chrome, not for multi-TB DBs. RocksDB was optimized specific
88.
by hyc_symas 3y ago
All of that is true, but I don't think it's a realistic concern. You're going to be sharding your data across multiple nodes before it gets that large. Nobody wants to sit around backing up or restoring a monolithic 256 TiB d
89.
by hyc_symas 3y ago
> I am also a little surprised at the lack of a random writes benchmark. Eh? This was 20% random writes, 80% random reads. LMDB is for read-heavy workloads. > That is an insanely high cache hit rate, which should have probably set off
90.
by hyc_symas 3y ago
msync lets you force a flush so you can control the latest possible moment for a writeout. But the OS can flush before that, and you have no way to detect or control that. So you can only control the late side of the timing, not the early s

More ›