6 ms·
Building a high-performance ticketing system with TigerBeetle
- ripberge 11mo agoHaving built a ticketing system that sold some Oasis level concerts there's a few misconceptions here: Selling an event out takes a long time to do frequently because tickets are VERY frequently not purchased--they're just reserved and then they fall back into open seating. This is done by true fans, but also frequently by bots run by professional brokers or amateur resellers. And Cloudflare and every other state of the art bot detection platform doesn't detect them. Hell, some of the bots are built on Cloudflare workers themselves in my experience... So whatever velocity you achieve in the lab--in the real world you'll do a fraction of it when it comes to actual purchases. That depends upon the event really. Events that fly under the radar may get you a higher actual conversion rate. Also, an act like Oasis is going to have a lot of reserved seating. Running through algorithms to find contiguous seats is going to be tougher than this example and it's difficult to parallelize if you're truly giving the next person in the queue the actual best seats remaining. There are many other business rules that accrue after years of features to win Oasis like business unfortunately that will result in more DB calls and add contention.
- 0cf8612b2e1e 11mo agoDoes that mean that there is some smoke and mirrors when, eg Taylor Swift, says they sold out the concert in minutes? Or are the mega acts truly that high demand?
- ripberge 11mo agoYou can get the seats into "baskets" (reserved) in minutes. In my experience they will not sell out for some time as they usually keep dropping back into inventory. "Sold Out" is a matter of opinion. There are usually lots of single seats left sometimes for weeks or months. The promoter decides when to label the event as "sold out".
- deleted 11mo ago[deleted]
- alamsterdam 11mo agoAgree with the above, we built and run a ticketing platform, the actual transaction of purchasing the ticket at the final step in the funnel is not the bottleneck. The shopping process and queuing process puts considerably more load on our systems than the final purchase transaction, which ultimately is constrained by the size of the venue, which we can control by managing the queue throughput. Even with a queue system in place, you inevitably end up with the thundering heard problem when ticket sales open, as a large majority of users will refresh their browsers regardless of instructions to the contrary
- jorangreef 11mo agoYou would use TigerBeetle for everything: not only the final purchase transaction, but the shopping cart process, inventory management and queuing/reserving. In other words, to count not only the money changing hands, but also the corresponding goods/services being exchanged. These are all transactions: goods/services and the corresponding money.
- jorangreef 11mo ago> Selling an event out takes a long time to do frequently because tickets are VERY frequently not purchased--they're just reserved and then they fall back into open seating. TigerBeetle actually includes native support for "two phase pending transfers" out of the box, to make it easy to coordinate with third party payment systems while users have inventory in their cart: https://docs.tigerbeetle.com/coding/two-phase-transfers/ https://docs.tigerbeetle.com/coding/two-phase-transfers/ > Also, an act like Oasis is going to have a lot of reserved seating. Running through algorithms to find contiguous seats is going to be tougher than this example and it's difficult to parallelize if you're truly giving the next person in the queue the actual best seats remaining. It's actually not that hard (and probably easier) to express this in TigerBeetle using transfers with deterministic IDs. For example, you could check (and reserve) up to 8K contiguous seats in a single query to TigerBeetle, with a P100 less than 100ms. > There are many other business rules that accrue after years of features to win Oasis like business unfortunately that will result in more DB calls and add contention. Yes, contention is the killer. We added an Amdahl's Law calculator to TigerBeetle's homepage to let you see the impact: https://tigerbeetle.com/#general-purpose-databases-have-an-oltp-limit https://tigerbeetle.com/#general-purpose-databases-have-an-o... As you move "the data to the code" in interactive transactions with multiple queries, to process more and more business rules, you're holding row locks across the network. TigerBeetle's design inverts this, to move "the code to the data" in declarative queries, to let the DBMS enforce the transactional business rules directly in the database, with a rich set of debit/credit primitives and audit trail.
- andersmurphy 11mo agoIt's almost like stored procedures were a good idea.
- jorangreef 11mo agoIf only. But you also need to fix the internal concurrency control of the DBMS storage engine. TB here is very different to PG. For example, if you have 8K transactions through 2 accounts, a naive system might read the 2 accounts, update their balances, then write the 2 accounts… for all 8K (!) transactions. Whereas TB does vectorized concurrency control: read the 2 accounts, update them 8K times, write the 2 accounts. This is why stored procedures only get you typically about a 10x win, you don’t see the same 1000x as with TB, especially at power law contention.
- kevinak 11mo agoIs FastAPI just bad with SQLite? I would have expected SQLite to smoke Postgres in terms of ops/s.
- makapuf 11mo agoSQLite is in process, but concurrent write / performance is a complex matter : https://sqlite.org/wal.html https://sqlite.org/wal.html
- kevinak 11mo agoYes, that's why I would expect it to smoke Postgres here, in process is orders of magnitude faster. Do you really need concurrency here when you can do 10-100k+ inserts per second?
- ies7 11mo agoIf 100k users each hit purchase button at the same time will sqlite write it in 1 second? This is different than 1 user doing the purchase for 100k fans
- kevinak 11mo agoI mean it depends on the query and what you're doing of course, but it's not impossible to reach writes/s in the 80k range.
- koakuma-chan 11mo agoI think Python is bad in general if you want “high-performance”
- 0cf8612b2e1e 11mo agoAlso surprised. My yardstick was this post which showed SQLite beating Postgres in a Django app. Benchmarking is hard, and the author said the Postgres results were not tuned to the same degree as SQLite, so buyer beware. https://blog.pecar.me/django-sqlite-benchmark https://blog.pecar.me/django-sqlite-benchmark
- nickmonad 11mo agoIt seems to me that, in practice, you'd want the "LiveBatcher" to have some durability as well. Is there a scenario where a customer could lose their place because of a horribly timed server shutdown, where those transfers hadn't even been sent to TigerBeetle as pending yet? Or am I misunderstanding the architecture here? Edit: Yes, I think I misunderstood something here. The user wouldn't even see their request as having returned a valid "pending" ticket sale since the batcher would be active as the request is active. The request won't return until its own transfer had been sent off to TigerBeetle as pending.
- kelseydh 11mo agoI recently did performance testing of Tigerbeetle for a financial transactions company. The key thing to understand about Tigerbeetle's speed is that it achieves very high speeds through batching transactions. ---- In our testing: For batch transactions, Tigerbeetle delivered truly impressive speeds: ~250,000 writes/sec. For processing transactions one-by-one individually, we found a large slowdown: ~105 writes/sec. This is much slower than PostgreSQL, which row updates at ~5495 sec. (However, in practice PostgreSQL row updates will be way lower in real world OLTP workloads due to hot fee accounts and aggregate accounts for sub-accounts.) One way to keep those faster speeds in Tigerbeetle for real-time workloads is microbatching incoming real-time transactions to Tigerbeetle at an interval of every second or lower, to take advantage of Tigerbeetle's blazing fast batch processing speeds. Nonetheless, this remains an important caveat to understand about its speed.
- nickmonad 11mo agoDid the company end up using it?
- kelseydh 11mo agoWe didn't rule out using Tigerbeetle, but the drop in non-batch performance was disappointing and a reason we haven't prioritised switching our transaction ledger from PostgreSQL to Tigerbeetle. There was also poor Ruby support for Tigerbeetle at the time, but that has improved recently and there is now a (3rd party) Ruby client: https://github.com/antstorm/tigerbeetle-ruby/ https://github.com/antstorm/tigerbeetle-ruby/
- jorangreef 11mo agoI think the drop in non-batch performance was more a function of the PoC than of TB. Would love to see what our team could do for you here! Feel free to reach out to peter@tigerbeetle.com
- NathanaelRea 11mo agoDoesn't the Tigerbeetle client automatically batch requests?
- vivzkestrel 11mo agoWhy cant this be done with PostgreSQL?
- jorangreef 11mo agoThe short answer is that we tried, back in 2020, while working on a central bank payment switch by the Gates Foundation. We found we were hitting the limits of Amdahl's Law, given Postgres' concurrency control with row locks held across the network as well as internal I/O, leading to the design of TigerBeetle. To specialize not for general purpose but only for transaction processing. On the one hand, yes, you could use a general purpose string database to count/move integers, up to a certain scale. But a specialized integer database like TigerBeetle can take you further. It's the same reason, that yes, you could use Postgres as object storage or as queue, or you could use S3 and Kafka and get separation of concerns in your architecture. I did a talk diving into all this recently, looking at the power law, OLTP contention, and how this interacts with Amdahl's Law and Postgres and TigerBeetle: https://www.youtube.com/watch?v=yKgfk8lTQuE https://www.youtube.com/watch?v=yKgfk8lTQuE
- vivzkestrel 11mo agoi am not an exact expert on the limitation you claim to have encountered on postgresql but perhaps someone with more postgresql expertise can chime in on this comment and give some insight
- jorangreef 11mo agoYou don't have to be an expert to understand Amdahl's Law. Definitely watch the talk if you haven't already: https://www.youtube.com/watch?v=yKgfk8lTQuE https://www.youtube.com/watch?v=yKgfk8lTQuE
- ants_a 11mo agoFor updating a single resource where the order of updates matters the best throughput one can hope for is the inverse of locking duration. Typical postgres using applications follow the pattern where a transaction involves multiple round trips between the application and the database to make decisions in the code running on the application server. But this pattern is not required by PostgreSQL, it's possible to run arbitrarily complex transactions all on server side using more complex query patterns and/or stored procedures. In this case the locking time will be mainly determined by time-to-durability. Which, depending on infrastructure specifics, might be one or two orders of magnitude faster. Or in case of fast networks and slow disks, it might not have a huge effect. One can also use batching in PostgreSQL to update the resource multiple times for each durability cycle. This will require some extra care from application writer to avoid getting totally bogged down by deadlocks/serializability conflicts. What will absolutely kill you on PostgreSQL is high contention and repeatable read and higher isolation levels. PostgreSQL handles update conflicts with optimistic concurrency control, and high contention totally invalidates all of that optimism. So you need to be clever enough to achieve necessary correctness guarantees with read committed and the funky semantics it has for update visibility. Or use some external locking to get rid of contention in the database. The option for pessimistic locking would be very helpful for these workloads. What would also help is a different kind of optimism, that would remove durability requirement from lock hold time, which would then result in readers having to wait for durability. Postgres can do tens of thousands of contended updates per second with this model. See the Eventual Durability paper for details.
- overfeed 11mo agoObligatory Jepsen report https://jepsen.io/analyses/tigerbeetle-0.16.11 https://jepsen.io/analyses/tigerbeetle-0.16.11