4 ms·
I run a service that sells theatre tickets and, far more critically, a service to check if a ticket is valid or not. Tickets are checked, by scanning a QR code
by abhibeckert 3y ago
I run a service that sells theatre tickets and, far more critically, a service to check if a ticket is valid or not.
Tickets are checked, by scanning a QR code, as people walk though the door. Not only do people hate sitting in their seat waiting (or worse, standing in line waiting)... you also need to pay a couple hundred employees to stand around and do nothing while waiting - ballpark cost of wages might be $5 for every second you can cut from the process of people walking through the door. On top of that wages cost, people waiting for the show to start don't buy drinks at the bar. A lot of events make more money on drink sales than ticket sales... often a high percentage of ticket sale revenue goes to whoever has a copyright claim to the show. They often get a cut of drink sales too, but it's a smaller cut.
So there are strong incentives to start scanning tickets as late as you possibly can before the performance actually starts. And if there's 1 second of downtime... it might trigger ten minutes of troubleshooting (turn it off and on again, etc) by a dozen people.
A show starting ten minutes late can be _really_ bad. If the show prep starts at 6pm and show pack up finishes at 11pm, that's five hours where some people are working non stop throughout that time - it's not really OK. Live events are dangerous, workplace deaths are far too common and some of the steps taken to try to prevent them don't really leave room for someone to take a meal break during a show. Alec Baldwin was shot and killed by a "prop" gun, in part because crew took a meal break at a critical time on the set, which interrupted routine safety checks... stuff like that can, and does, happen in theatre too.
There are union rules around breaks that everything is planned around and because delays happen in the real world there are financial penalties in place to discourage delays. So instead of $5 per second, that ten minutes might cost you $20 per second... in other words a $12,000 cost that wasn't in your budget... and the producer will have to explain why they ran $12k over budget - keep in mind the show might not actually make $12k profit in a single night's performance. Sometimes they don't even make a profit when they run smoothly.
All of that means your one second of downtime will be followed a formal written incident report and meetings to discuss what happened, and those meetings will inevitably include investigating alternative ticket services. We know little things like that can cost us a fortune, because we win new customers all the time when they tell us they left a competitor after a single brief period of downtime (our biggest competitor recently had a bug where buying tickets didn't work in the shitty browser that ships with certain cheap android phones... they fixed it relatively quickly but we gained a lot of new customers anyway).
Sure - it's only that critical from, say, 7:30pm to 7:55pm in a single building... an hour of downtime at 10am wouldn't be noticed... but if it's a global service, with dozens of cities in every timezone, then that critical state is happening 24/7/365.
One second of downtime will absolutely cause a measurable, and large, cost to your company profits. We deployed a major new system for ticket scanning in February last year. It worked perfectly, we haven't made any changes since February - other than just testing it out, and we're starting to deploy it over the next two months. That's how reluctant we are to break things.
We considered PostgreSQL, but decided it's not reliable enough.
We're using SQLite - with literally thousands of databases — every hand held scanning device as well as the servers run their own database. The servers themselves have a separate database for each individual event (and there are redundant servers too, so multiple server side databases per event). Also the database for critical things like scanning tickets is a separate database to less critical things, like payment records, event details, etc.
PostgreSQL is better than SQLite for the servers at least in a bunch of ways - but the big advantage of SQLite is flexibility - we can do a phased deployment where from the moment we start selling tickets (a few months before the actual event) to the moment of the actual event, there will be no deployments at all to that database unless we think a bug will directly effect the event. A deployment today will typically only apply to events that go on sale starting tomorrow.
Every business case has different needs, but skim reading this article on the future of Postgres, it sounds like they're working to improve some of the reasons I chose not to use it for this specific case.
- backwardation_b 3y agoOn the theme of downtime, how do you handle durability of your SQLite db’s? What if one of the devices goes down?
- abhibeckert 3y agoAs a primary last resort there should be a printed paper backup of the database, but it's a snapshot of ticket sales that at this point would be an hour old, and there's always a rush of last minute writes to the database. The better organised venues write those changes down with a pen, so it's at least up to date, but modern venues have several entrances to get people through the doors faster, so most people won't have the up to date paper backup. Our system uses a cross platform app on ordinary smart phones to scan tickets. So if a device goes down, you just need to get your hands on another phone. There's a limited feature set version of the QR code scanner (scan code, screen flashes red or green) that just runs in a web browser on any device with a half decent camera. All you need is the secret URL for the tonights event — which we also expose via a QR code.
- Demiurge 3y agoI think I understand the use case, and effectively you're sharding the data with SQLite. This approach is applicable with multiple databases/datastores. However, would you really change your approach if PostgreSQL could be upgraded by a minor version without any downtime? > One second of downtime will absolutely cause a measurable, and large, cost to your company profits. I want to believe, but I really am not conviced that 1 second downtime to restart PostgreSQL out of 24/7/365 would make more of a dent on your companys profits, as dedicating two weeks of developers time on optimizing query performance and mobile app performance to potentially improve average response time of your apps by multiple percentage points.
- abhibeckert 3y ago> However, would you really change your approach if PostgreSQL could be upgraded by a minor version without any downtime? Dunno. I wasn't so much answering the "who wouldn't use Postgres" question, I was more answering "who could possibly care about 1 second of downtime" question. I seriously considered Postgres, and I will reconsider that choice regularly. There are things we have in SQLite right now that would absolutely be better in Postgres, but for the moment we've decided those benefits don't justify the overhead of two completely different database system. > I really am not conviced that 1 second downtime to restart PostgreSQL out of 24/7/365 would make more of a dent on your companys profits Our customers are not the people buying tickets. Our customers are the people selling tickets, and our profit is a percentage of their annual revenue. They're moderately sized companies with hundreds of staff. Which means a single dissatisfied person, at the wrong level of management, can cost us hundreds of thousands of dollars. Downtime is inevitable and we don't claim to offer 100% uptime... but we try really really hard to be as close to that as possible. That includes encouraging people to print, on dead tree paper, a copy of their ticket sales database five minutes before they open the doors for a performance. But that copy is always out of date, since they sell tickets after opening the doors, so it's not a great backup.