hyc_symas

5 ms·

31.
by hyc_symas 6mo ago
I walked through the design at Monerokon in 2019 here https://www.youtube.com/watch?v=4Hkd-n1W_e4
32.
by hyc_symas 8mo ago
Ah yes, the ever popular "mongoDB's developers were incompetent therefore mmap is bad" paper. Pure tripe. https://www.symas.com/post/are-you-sure-you-want-to-use-mmap...
33.
by hyc_symas 8mo ago
How often does anyone care about using data on a different system than it was created on? These days, any C struct you built on amd64 will work identically on arm64. There really aren't any other architectures that matter. And yes, man
34.
by hyc_symas 8mo ago
On BSD, read() was already implemented in the kernel by page-faulting in the desired pages of the file, to then be copied into the user-supplied buffer. So from the first time mmap was ever implemented, it was always the fastest input mecha
35.
by hyc_symas 8mo ago
Network data and most serialization formats are big endian because it's easiest to shift bits in and out of a shift register onto a serial comm channel in that order. If you used little endian, the shifter on output would have to opera
36.
by hyc_symas 8mo ago
In-memory DBs were always a dead-end waste of time. There will always be bigger slower cheaper secondary storage, no matter how cheap main memory gets. And workloads always grow to exceed the available main memory (unless a system is dying)
37.
by hyc_symas 10mo ago
Fun footnote: SQLite only got on board with mmap after I demonstrated how slow their code was without it. I.e., getting a 22x speedup by replacing SQLite's btree code with LMDB https://github.com/LMDB/sqlightning
38.
by hyc_symas 10mo ago
Andy's critiques are only valid on dedicated database servers. https://www.symas.com/post/are-you-sure-you-want-to-use-mmap... LMDB uses mmap and Andy recommends LMDB, in the very article this thread is about.
39.
by hyc_symas 10mo ago
A lot of potential treatments are too easily available and can't be patented. If a big pharma company can't make massive profit from it, they won't bother bringing it to market. Consider that a not-good reason. Other treatmen
40.
by hyc_symas 11mo ago
You don't have to take my word for it. Plenty of other developers know. https://www.youtube.com/watch?v=CfiQ0h4bGWM
41.
by hyc_symas 11mo ago
Not every. LMDB is optimized for read-heavy workloads. I make no particular claims about write-heavy workloads. Because it's so efficient, it can retain more useful data in-memory than other DBs for a given RAM size. For DBs much large
42.
by hyc_symas 11mo ago
Plenty of proof. http://www.lmdb.tech/bench/
43.
by hyc_symas 11mo ago
Identical hardware, same RAM size, same data volume.
44.
by hyc_symas 11mo ago
It has a single writer transaction mutex, yes. But it's a process-shared mutex, so it will serialize write transactions across an arbitrary number of processes. And of course, read transactions are completely lockfree/waitfree acr
45.
by hyc_symas 11mo ago
Wrong, LMDB fully supports multiprocess concurrency as well as DBs multiple orders of magnitude larger than RAM. Wherever you got your info from is dead wrong. Among embedded key/value stores, only LMDB and BerkeleyDB support multiproc
46.
by hyc_symas 11mo ago
> So much C library code is documented in ad-hoc ways - often through doxygen, which is a disaster. Eg here's the documentation for LMDB. LMDB is one of the most thoroughly documented C APIs I've seen, but I find this almost to
47.
by hyc_symas 1y ago
That's such an obvious error in their benchmark code. In my benchmark code I make sure to touch the data so at least the 1st page is actually paged in from disk. https://github.com/LMDB/dbbench/blob/12815
48.
by hyc_symas 1y ago
Thanks for that. It doesn't look like there is an open bug report for this yet. I understand your commit description but I'm still a bit puzzled by it. cursor_to_update shouldn't have a deeper stack than cursor, even if it wa
49.
by hyc_symas 1y ago
What's the bug ID, have you submitted the fix back?
50.
by hyc_symas 1y ago
LMDB is for read-heavy workloads. The opposite of RocksDB. RocksDB can use thousands of file descriptors at once, on larger DBs. Makes it unsuitable for servers that may also need to manage thousands of client connections at once. LMDB uses
51.
by hyc_symas 1y ago
A system built on LMDB will work better for this use case than RocksDB. And OSM Express already uses it. https://wiki.openstreetmap.org/wiki/OSM_Express
52.
by hyc_symas 2y ago
> LMDB has MDB_PANIC, documented as "Update of meta page failed or environment had fatal error". Yes. That doesn't mean there was anything bad in the program logic. It most likely means your storage device had a fatal I&#x
53.
by hyc_symas 3y ago
Welp, back in college I wrote a virtual memory demand pager for IBM 370 mainframe. I also read the entire 4.3BSD kernel source, as well as the Minix kernel source, and hacked on both of them. Also ported NFS server and client to the mainfra
54.
by hyc_symas 3y ago
> That link claims the test is 16 cores but my hunch given that the benchmark is from 2013 is that it’s 8+HT. We tell no lies. Those tests were run on an HP DL585 G5 server with 128GB of RAM and 4 quad core AMD Opteron 8354 CPUs. If cros
55.
by hyc_symas 3y ago
If there's any effect (like frame stuttering) then it will impact the max and 99th percentile latencies. Given that LMDB's latencies are always the tightest of any DB, it still boils down to "even if the effect is real, it&#x
56.
by hyc_symas 3y ago
Yes, you can use raw block devices in Linux. LMDB works on raw block devices as well. You get write-through cache behavior; writes are immediate but everything still goes into the page cache. I've not yet bothered to check how things w
57.
by hyc_symas 3y ago
Replacing the NIC sounds expensive, on a college student budget. You could've just reprogrammed the MAC EEPROM, no?
58.
by hyc_symas 3y ago
It took many years before mobile carriers in the US supported IPv6. I don't think any carriers here in Ireland do. My gigabit fiber provider only supports IPv4. Granted, they're just a VNO, using PPPoE on top of Eircom's netw
59.
by hyc_symas 3y ago
What you call optimal performance with InnoDB, I call pathetically slow. http://www.lmdb.tech/bench/memcache/ And if you'd read the reliability references linked in the article, you'd see that MariaDB&#x
60.
by hyc_symas 3y ago
If this was a significant factor, then it should have a greater impact in high concurrency workloads, getting progressively worse across multi-socket machines. Benchmarks all show that LMDB performance scales perfectly linearly across arbit

More ›