hyc_symas

9 ms·

91.
by hyc_symas 3y ago
The jump in address sizes starts to get too unwieldy. 32 bit addresses were ok, 64 bit addresses start to get clunky, 128 bit would be exorbitant for CPU real estate. There's a reason AMD64 still only supported 40 physical address bits
92.
by hyc_symas 3y ago
Huge pages aren't pageable, so they wouldn't be particularly advantageous for a mmap DB anyway, you'd have to do traditional I/O & buffer management for everything.
93.
by hyc_symas 3y ago
"fit entirely in memory" is not a requirement. LMDB is not a main-memory database, it is an on-disk database that uses memory mapping.
94.
by hyc_symas 3y ago
Yeah, no. Not with a DB 50x larger than RAM, anyway. http://www.lmdb.tech/bench/hyperdex/ RAM is relatively cheap too, there's no real reason to be running multi-TB databases at greater than a 50x ratio.
95.
by hyc_symas 3y ago
> Do you have benchmarks of lmdb when the working set is much larger than memory? I couldn't find any. Where did you look? This is a sample using DB 5x and 50x larger than RAM http://www.lmdb.tech/bench/hyperde
96.
by hyc_symas 3y ago
You don't need more than single-writer concurrency if your write txns are fast enough. Our experience with OpenLDAP was that multi-writer concurrency cost too much overhead. Even though you may be writing primary records to independent
97.
by hyc_symas 3y ago
Yes, LMDB is an embedded key/value store but it can be used as the backing store of any other DB model you care for. E.g. as a backend to MySQL, or SQLite, or OpenLDAP, or whatever.
98.
by hyc_symas 3y ago
1 - for reading any uncached data, the I/O stalls are unavoidable. Whatever client requested that data is going to have to wait regardless. 2 - complexity? this is simply false. LMDB's ACID txns using MVCC are much simpler than an
99.
by hyc_symas 3y ago
>but I don't think GP meant "one process" literally. Neither that is realistic nor is it necessary. The point was simply about other processes that could be competing for resources - CPU, memory, or I/O. It is expensi
100.
by hyc_symas 3y ago
It wouldn't be much use without the B+tree as well; it's the B+tree's cache friendliness that allows applications to run so efficiently without the OS knowing any specifics of the app's usage patterns.
101.
by hyc_symas 3y ago
Correct, and thank you. I wrote LMDB, wrote a lot of OpenLDAP, and worked on BerkeleyDB for many years. And actually Andy Pavlo invited me to CMU to give a lecture on LMDB a few years back. https://www.youtube.com/watch?v=tE
102.
by hyc_symas 3y ago
> There's nothing special about kernel programmers. Yes, that was a shorthand generalization for "people who've studied computer architecture" - which most application developers never have. > no DBA worth their sa
103.
by hyc_symas 3y ago
This is a pretty old argument and IMO it's far out of date/obsolete. Taking full control of your I/O and buffer management is great if (a) your developers are all smart and experienced enough to be kernel programmers and (b)
104.
by hyc_symas 4y ago
Yeah, I explicitly LD_PRELOAD libeatmydata for a few things, like my web browser and before running a software upgrade. So much junk being overwritten multiple times, manpages regenerated, init ramdisks regenerated, over and over again in a
105.
by hyc_symas 4y ago
kernel pagesize is returned by sysconf(_SC_PAGESIZE)
106.
by hyc_symas 4y ago
> Thanks for lmdb btw! Thanks, that's great feedback! Happy to hear that you're both using it successfully and that you actually read the code and learned from it.
107.
by hyc_symas 4y ago
LMDB performance suffers when used on a journaling filesystem. Consequently, LMDB supports using a plain block device, to eliminate all FS overheads. And yes, all of the perf benefits go out the window if you're running on several laye
108.
by hyc_symas 4y ago
Write barriers have been discussed for years and years. Personally I prefer grouped writes. https://www.spinics.net/lists/linux-fsdevel/msg70047.html The underlying SCSI and SATA protocols would support this with
109.
by hyc_symas 4y ago
strlcpy() doesn't fix the problem of getting the length parameter wrong. The right solution is this: char *stecpy(char *d, const char *s, const char *e) { if (e) e--; while (d < e && *s) *d++ =
110.
by hyc_symas 4y ago
Hm, in March 2016 I got a 64GB USB flash drive from amazon.co.uk for £24.99 and today a similar 64GB drive is only £6.99. A 500GB Samsung 850 EVO SSD was £110.99. Now a 500GB Samsung 870 EVO SSD is £68.47. Samsung has just started mass prod
111.
by hyc_symas 4y ago
Filesystem-level compression isn't magic, you still have to write data in sector-sized chunks to the underlying device. And while individual flash chips still use 512 byte sectors (pages), modern SSDs flash controllers gang them up in
112.
by hyc_symas 4y ago
You don't need a new hardware interface. You only need a sensible API for write barriers. In particular, you want "semi-permeable" barriers, which is just another way of saying tagged commands, or grouped writes. 1) An f
113.
by hyc_symas 4y ago
The paper's analysis of LMDB is wrong. https://news.ycombinator.com/item?id=32462537
114.
by hyc_symas 4y ago
Yes, deprecated. I thought it had already been dropped but apparently not yet. https://docs.microsoft.com/en-us/windows/win32/fileio/about-...
115.
by hyc_symas 4y ago
The description of LMDB's behavior and subsequent analysis are flat wrong. https://twitter.com/hyc_symas/status/1558909442737012736 To assume that any newbie has hit upon a potential failure condition that we
116.
by hyc_symas 4y ago
You could use an LDAP SASL Bind instead, using any of a variety of strong authentication mechanisms then without plaintext passwords.
117.
by hyc_symas 4y ago
I'm guessing you mean "and the software is fast enough ", because OpenLDAP itself has been the world's fastest distributed database for over a decade, and nothing else even comes close. Since you built on top of SQLite,
118.
by hyc_symas 4y ago
OpenLDAP's license is a pretty standard BSD 3-clause.
119.
by hyc_symas 4y ago
> size slapd text data bss dec hex filename 2787425 78888 570048 3436361 346f49 slapd PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
120.
by hyc_symas 4y ago
1) was rectified pretty easily, there were X.500 implementations on top of TCP/IP. 2) was definitely annoying. They seem to have loosened restrictions on access to the specs these days. X.500 (and X.400) got a lot of pushback in the IE

More ›