3 ms·
SQLite is a great database but it's not suited for applications with frequent writes or write-transactions that take longer than a couple of milliseconds. I tr
by dajonker 2y ago
SQLite is a great database but it's not suited for applications with frequent writes or write-transactions that take longer than a couple of milliseconds.
I tried basically everything in this blog post with our Rails business application and none of it really works in practice. I expect the same to be true for Django.
The reason is that with Rails, you usually end up with a bunch of callbacks on models which cause "long" running transactions. For example, the application writes a record to the database, then uses the newly generated primary key to create a bunch of related records, etc. With a reasonable amount of business logic, this means that a transaction can easily run over 100 milliseconds. Not because the database is slow (it's not), but because the application may do all kinds of slow stuff in between the different statements that run in a transaction.
SQLite is single threaded when it comes to writes, so when an average write can take 100 ms, your throughput is already limited to about 10 transactions per second. IMMEDIATE transactions are indeed necessary, to wait dor the database to be ready before attwempting any transaction. Because at least it is easy to have a backoff/retry strategy before letting the framework run its transaction logic.
However, even with just a handful of active users, I needed to make sure that waiting transactions were retried a large number of times, to prevent users from getting a 500 error and requiring them to perform the same action again. However, users were complaining that the application was slow, and I had the metrics that told me the same.
Eventually I just switched to Postgres and all of the issues just disappeared. The users also immediately told me that the application became much more responsive.
I did notice in the metrics that a lot of the read operations actually became slower, as SQLite is really efficient at doing lots of small reads as compared to a client/server model database.
I do still think that SQLite is very suited for production applications, but only when it is read heavy or has very lightweight write transactions.
- anze3db 2y agoI agree 100% with everything you wrote. It was very surprising to me that every transaction and every write operation blocks the whole database and not only the table it's performed upon (it makes sense since it's all a single file, but still). The only workaround for this is splitting your main database into multiple databases, but this bleeds into your application logic and gets messy quickly. If you are in this position, it's best to switch to Postgres as you did!