3 ms·

I really like the work Joran and his team have done around DST, distributed systems awareness, and performance practices. I especially love the craziness around

I really like the work Joran and his team have done around DST, distributed systems awareness, and performance practices. I especially love the craziness around no dependencies (well, could you consider an underlying OS a dependecy?). But I've always felt the way they treat normal OLTP (they call OLGP) seems unfair. For example, comparisons using clear sub-optimal interactive SQL transactions for financial workloads, like locking rows rather than using condition checks at commit time, because "that's how OLTP was intended to be used when it was designed ~50(?) years ago". In their cited https://tigerbeetle.com/#performance https://tigerbeetle.com/#performance the lowest the slider can go is 1% contention. Do you think Stripe has 1% contention directly on an OLTP DB? Definitely not. You can build systems that _expect contention_, and elegantly handle it at REALLY high throughput. These systems protect the DB from contention, so you can continue to scale. From talking to folks working on these systems, I roughly know the kinds of transactional (financial) throughput of DBs like Stripe's and other systems - they have _many_ more zeros behind them than their performance comparison page proposes they could possibly have at even 0.01% contention. Their marketing largely ignores this fact, and treats everyone like they just slam the DB with junior engineer-designed interactive transactions. Most developers (I hope) are smarter than that if they're working at a payments company. There's even the title "payments engineer" for the kind of person that's thinks about scaling contention and correctness all day. TigerBeetle is great, but I find the pattern of being quite misleading about other OLTPs off putting.

sgarland 1y ago
> Do you think Stripe has 1% contention directly on an OLTP DB? Definitely not. Stripe runs on top of MongoDB, which is horrifying in its own right, but in any case comparing them to a shop running an RDBMS is apples to oranges.
- redwood 1y ago
  Stripe delivers incredible uptime. This type of comment is beneath this platform.
  - sgarland 1y ago
    I didn’t say they’re bad at their job, I said I find the technology choice horrifying. That’s a personal take, not a value statement.
xxpor 1y ago
>well, could you consider an underlying OS a dependecy? I've worked on systems that ran entirely in memory, and continued running during kexec. You can't do syscalls. So yeah :)
jorangreef 1y ago
Hey Dan, appreciate the compliment. Do you think it would be more fair to suggest that OLTP workloads have 0% contention? That debit/credit sets never intersect? In practice, we're seeing that our customers are already some of the largest brokerages, wealth managements or exchanges, in their jurisdictions, even national projects, and all of them with decades of Postgres etc. experience, some even running installations with 200 Postgres/MySQL machines backing their sharded core ledger. They're not junior engineers. For example, they know about stored procedures, but the problem of concurrency runs deeper, into the storage engine itself. At least for our customers, for the seasoned payments engineers, OLTP contention is a killer. They're tired of scaling outside the DBMS, and the complexity of expensive reconciliation systems. Several of them are even leaving their Chief Architect positions and starting TB-related startups—I know of at least 4 personally, from fintech brands you will recognize, and probably use. I hope the spirit of our talks, trying to get the big ideas across, to make people aware of contention and Amdahl's Law, is clear, and that you take it in good faith, to the extent that we show it ourselves.
- dangoodmanUT 1y ago
  Oh I totally agree the world needs to be more aware of Amdahls law, and I appreciate that you make it known (I believe I even learned it from you). I just think that the communication is sometimes too much "it's our way or the bad way", if that makes sense? I don't think they have 0% contention, and I agree that contention is the bane of a dev's existence, but I don't think it's as cut and dry as only looking at Amdahl's. Specialized systems have done really well to handle contention: My go-to example is LMAX. I think I remember a talk where the devs said they were pulling 6M orders per second, and ofc the stock market has an aggressive power law. If you design around contention, you can still push silly high perf (as you've shown). FWIW I think we both agree that Postgres is not the best DB for this either :P
n_u 1y ago
> comparisons using clear sub-optimal interactive SQL transactions for financial workloads, like locking rows rather than using condition checks at commit time Could you give an example of how the same transaction could be written poorly with "locking rows" and then more optimally with "using condition checks at commit time"?
- dangoodmanUT 1y ago
  subtract 50 if balance >= 50 is optimistic (the value at commit can be different than at read) vs. "lock this row so it can't change while I subtract 50 from 50,000,000,000,000..." you get the point
  - n_u 1y ago
    I'm more confused now. Those two have different logic? In the second example the balance can go negative. Also aren't both of those are going to have to lock the row to modify it? Even if you don't explicitly take out a lock on a row the DBMS will do its own concurrency control to provide transaction isolation, which will require a write lock for the row. Edit: Oh are you saying the "poorly written" case is an interactive transaction where the logic happens in the application code and involves multiple round trips while holding a lock? Instead of a transaction (or maybe even stored proc) where the logic happens in the DB, so less time holding the lock. Less contention. Ah ok. Yeah interactive transactions like that aren't great.
deleted 1y ago
[deleted]

sgarland 1y ago

> Do you think Stripe has 1% contention directly on an OLTP DB? Definitely not. Stripe runs on top of MongoDB, which is horrifying in its own right, but in any case comparing them to a shop running an RDBMS is apples to oranges.

redwood 1y ago
Stripe delivers incredible uptime. This type of comment is beneath this platform.
- sgarland 1y ago
  I didn’t say they’re bad at their job, I said I find the technology choice horrifying. That’s a personal take, not a value statement.

xxpor 1y ago

>well, could you consider an underlying OS a dependecy? I've worked on systems that ran entirely in memory, and continued running during kexec. You can't do syscalls. So yeah :)

jorangreef 1y ago

Hey Dan, appreciate the compliment. Do you think it would be more fair to suggest that OLTP workloads have 0% contention? That debit/credit sets never intersect? In practice, we're seeing that our customers are already some of the largest brokerages, wealth managements or exchanges, in their jurisdictions, even national projects, and all of them with decades of Postgres etc. experience, some even running installations with 200 Postgres/MySQL machines backing their sharded core ledger. They're not junior engineers. For example, they know about stored procedures, but the problem of concurrency runs deeper, into the storage engine itself. At least for our customers, for the seasoned payments engineers, OLTP contention is a killer. They're tired of scaling outside the DBMS, and the complexity of expensive reconciliation systems. Several of them are even leaving their Chief Architect positions and starting TB-related startups—I know of at least 4 personally, from fintech brands you will recognize, and probably use. I hope the spirit of our talks, trying to get the big ideas across, to make people aware of contention and Amdahl's Law, is clear, and that you take it in good faith, to the extent that we show it ourselves.

dangoodmanUT 1y ago
Oh I totally agree the world needs to be more aware of Amdahls law, and I appreciate that you make it known (I believe I even learned it from you). I just think that the communication is sometimes too much "it's our way or the bad way", if that makes sense? I don't think they have 0% contention, and I agree that contention is the bane of a dev's existence, but I don't think it's as cut and dry as only looking at Amdahl's. Specialized systems have done really well to handle contention: My go-to example is LMAX. I think I remember a talk where the devs said they were pulling 6M orders per second, and ofc the stock market has an aggressive power law. If you design around contention, you can still push silly high perf (as you've shown). FWIW I think we both agree that Postgres is not the best DB for this either :P

n_u 1y ago

> comparisons using clear sub-optimal interactive SQL transactions for financial workloads, like locking rows rather than using condition checks at commit time Could you give an example of how the same transaction could be written poorly with "locking rows" and then more optimally with "using condition checks at commit time"?

dangoodmanUT 1y ago
subtract 50 if balance >= 50 is optimistic (the value at commit can be different than at read) vs. "lock this row so it can't change while I subtract 50 from 50,000,000,000,000..." you get the point
- n_u 1y ago
  I'm more confused now. Those two have different logic? In the second example the balance can go negative. Also aren't both of those are going to have to lock the row to modify it? Even if you don't explicitly take out a lock on a row the DBMS will do its own concurrency control to provide transaction isolation, which will require a write lock for the row. Edit: Oh are you saying the "poorly written" case is an interactive transaction where the logic happens in the application code and involves multiple round trips while holding a lock? Instead of a transaction (or maybe even stored proc) where the logic happens in the DB, so less time holding the lock. Less contention. Ah ok. Yeah interactive transactions like that aren't great.

deleted 1y ago

[deleted]