4 ms·

If this was a significant factor, then it should have a greater impact in high concurrency workloads, getting progressively worse across multi-socket machines.

If this was a significant factor, then it should have a greater impact in high concurrency workloads, getting progressively worse across multi-socket machines. Benchmarks all show that LMDB performance scales perfectly linearly across arbitrarily many CPUs. If this effect were an actual issue then the performance scaling should be significantly sub-linear. It's not. See e.g. http://www.lmdb.tech/bench/inmem/scaling.html http://www.lmdb.tech/bench/inmem/scaling.html or even the redb benchmark https://mastodon.social/@hyc/111887593549586249 https://mastodon.social/@hyc/111887593549586249 The notion that "maybe their performance floor was higher" is laughable, since no other DBMS delivers anywhere near LMDB's read performance.

vlovich123 3y ago
> The notion that "maybe their performance floor was higher" is laughable, since no other DBMS delivers anywhere near LMDB's read performance. What I said was that lmdb’s read performance may be so good, that at its worst when you’re seeing cross tlb shootdowns, it’s still better than the other DBs. You seem to contradict me and then restate the exact same thing. > Benchmarks all show that LMDB performance scales perfectly linearly across arbitrarily many CPUs As I said, synthetic benchmarks may not reproduce the cross tlb shootdown scenario frequently enough for you to notice it. It’s like frame stuttering - you can’t argue that it’s not happening by saying the average frame rate was 70fps. The data you’re using to support your argument isn’t a relevant measure. You’d have to actually measure how frequently cross tlb shootdowns are happening (via dtrace probably) and ideally compare it against a robust lmdb read path written against iouring to get an accurate measure of the issue.
- hyc_symas 3y ago
  If there's any effect (like frame stuttering) then it will impact the max and 99th percentile latencies. Given that LMDB's latencies are always the tightest of any DB, it still boils down to "even if the effect is real, it's unmeasurable and therefore negligible." http://www.lmdb.tech/bench/memcache/ http://www.lmdb.tech/bench/memcache/ And of course, none of this is working in a realtime environment, so a couple nanoseconds here or there just doesn't matter.
  - vlovich123 3y ago
    Except even your link shows a substantial jump in the log graph from ~500ms->1s going p99 to pmax. And cross tlb shoot down times can vary wildly but nanoseconds is the wrong order of magnitude. They also get significantly worse with the number of cores. That link claims the test is 16 cores but my hunch given that the benchmark is from 2013 is that it’s 8+HT. I really don’t understand this defensiveness and doubling down on something that’s pretty well studied and documented. Keep in mind that the original paper looked at raw mmap vs raw io api issues regardless of the surrounding DB: https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf It’s a really well written and accessible paper and this is not the hill to die on if you’re disagreeing. I agree mmap for the read path only is a weakness in their paper and that idea I agree has some merit. But cross tlb shootdowns are a real performance downside of this approach.
    - arandomusername 3y ago
      Are we looking at the same benchmark? All of LMDB's MAX read perf is under 1ms. Him doubling down makes sense, given that LMDB continues to offer the best read performance in any benchmark, regardless of who performed it. If tlb shoot down times are such a problem, why aren't we seeing it being represented in any DB benchmark?
    - hyc_symas 3y ago
      > That link claims the test is 16 cores but my hunch given that the benchmark is from 2013 is that it’s 8+HT. We tell no lies. Those tests were run on an HP DL585 G5 server with 128GB of RAM and 4 quad core AMD Opteron 8354 CPUs. If cross-TLB shootdowns were actually as big a problem as they claim, there should be massive latency spikes due to cross-socket communication, far worse than for inter-core single socket. You're definitely misreading the graphs, the Y axis is clearly labeled "msec". You're going to have to pay better attention if you want to discuss this further. Otherwise we're just wasting our time. > Keep in mind that the original paper looked at raw mmap vs raw io api issues regardless of the surrounding DB Yes, but that's exactly why the paper is utterly worthless and never should have made it past peer review. Their initial claim is this: > Unfortunately, mmap has a hidden dark side with many sordid problems that make it undesirable for file I/O in a DBMS. As we describe in this paper, these problems involve both data safety and system performance concerns. We contend that the engineering steps required to overcome them negate the purported simplicity of working with mmap. For these reasons, we believe that mmap adds too much complexity with no commensurate performance benefit and strongly urge DBMS developers to avoid using mmap as a replacement for a traditional buffer pool. To prove their claims they would have to have demonstrated that a DBMS using mmap was more complex and less reliable than one using a traditional buffer pool. They also had to demonstrate that a DBMS using mmap had no performance benefit vs a DBMS using a traditional buffer pool. They didn't demonstrate any of these things, and LMDB is concrete proof that all of these claims are wrong: at only 7KLOCs it is simpler than every other traditional DBMS, its reliability is literally perfect, and its read performance is always orders of magnitude than all traditional designs. The reviewers should have known just from reading the abstract that this paper was a dud: > Such problems make it difficult, if not impossible, to use mmap correctly and efficiently in a modern DBMS. Their claim required them to prove a negative, which obviously defies the rules of logic. Everything after that was just noise.

vlovich123 3y ago

> The notion that "maybe their performance floor was higher" is laughable, since no other DBMS delivers anywhere near LMDB's read performance. What I said was that lmdb’s read performance may be so good, that at its worst when you’re seeing cross tlb shootdowns, it’s still better than the other DBs. You seem to contradict me and then restate the exact same thing. > Benchmarks all show that LMDB performance scales perfectly linearly across arbitrarily many CPUs As I said, synthetic benchmarks may not reproduce the cross tlb shootdown scenario frequently enough for you to notice it. It’s like frame stuttering - you can’t argue that it’s not happening by saying the average frame rate was 70fps. The data you’re using to support your argument isn’t a relevant measure. You’d have to actually measure how frequently cross tlb shootdowns are happening (via dtrace probably) and ideally compare it against a robust lmdb read path written against iouring to get an accurate measure of the issue.

hyc_symas 3y ago
If there's any effect (like frame stuttering) then it will impact the max and 99th percentile latencies. Given that LMDB's latencies are always the tightest of any DB, it still boils down to "even if the effect is real, it's unmeasurable and therefore negligible." http://www.lmdb.tech/bench/memcache/ http://www.lmdb.tech/bench/memcache/ And of course, none of this is working in a realtime environment, so a couple nanoseconds here or there just doesn't matter.
- vlovich123 3y ago
  Except even your link shows a substantial jump in the log graph from ~500ms->1s going p99 to pmax. And cross tlb shoot down times can vary wildly but nanoseconds is the wrong order of magnitude. They also get significantly worse with the number of cores. That link claims the test is 16 cores but my hunch given that the benchmark is from 2013 is that it’s 8+HT. I really don’t understand this defensiveness and doubling down on something that’s pretty well studied and documented. Keep in mind that the original paper looked at raw mmap vs raw io api issues regardless of the surrounding DB: https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf It’s a really well written and accessible paper and this is not the hill to die on if you’re disagreeing. I agree mmap for the read path only is a weakness in their paper and that idea I agree has some merit. But cross tlb shootdowns are a real performance downside of this approach.
  - arandomusername 3y ago
    Are we looking at the same benchmark? All of LMDB's MAX read perf is under 1ms. Him doubling down makes sense, given that LMDB continues to offer the best read performance in any benchmark, regardless of who performed it. If tlb shoot down times are such a problem, why aren't we seeing it being represented in any DB benchmark?
  - hyc_symas 3y ago
    > That link claims the test is 16 cores but my hunch given that the benchmark is from 2013 is that it’s 8+HT. We tell no lies. Those tests were run on an HP DL585 G5 server with 128GB of RAM and 4 quad core AMD Opteron 8354 CPUs. If cross-TLB shootdowns were actually as big a problem as they claim, there should be massive latency spikes due to cross-socket communication, far worse than for inter-core single socket. You're definitely misreading the graphs, the Y axis is clearly labeled "msec". You're going to have to pay better attention if you want to discuss this further. Otherwise we're just wasting our time. > Keep in mind that the original paper looked at raw mmap vs raw io api issues regardless of the surrounding DB Yes, but that's exactly why the paper is utterly worthless and never should have made it past peer review. Their initial claim is this: > Unfortunately, mmap has a hidden dark side with many sordid problems that make it undesirable for file I/O in a DBMS. As we describe in this paper, these problems involve both data safety and system performance concerns. We contend that the engineering steps required to overcome them negate the purported simplicity of working with mmap. For these reasons, we believe that mmap adds too much complexity with no commensurate performance benefit and strongly urge DBMS developers to avoid using mmap as a replacement for a traditional buffer pool. To prove their claims they would have to have demonstrated that a DBMS using mmap was more complex and less reliable than one using a traditional buffer pool. They also had to demonstrate that a DBMS using mmap had no performance benefit vs a DBMS using a traditional buffer pool. They didn't demonstrate any of these things, and LMDB is concrete proof that all of these claims are wrong: at only 7KLOCs it is simpler than every other traditional DBMS, its reliability is literally perfect, and its read performance is always orders of magnitude than all traditional designs. The reviewers should have known just from reading the abstract that this paper was a dud: > Such problems make it difficult, if not impossible, to use mmap correctly and efficiently in a modern DBMS. Their claim required them to prove a negative, which obviously defies the rules of logic. Everything after that was just noise.