3 ms·
> you can have any number of concurrent readers, but only a single writer Only if your readers and writers are cleanly segregated. Most languages and web fram
by altfredd 6y ago
> you can have any number of concurrent readers, but only a single writer
Only if your readers and writers are cleanly segregated.
Most languages and web frameworks don't have SQLite drivers out of box (or have extremely bad ones). Unlike SQLite, most databases don't really care about distinction between read-only and writable connections. So there is a good chance, that you will always open writable connection by default, because this is what your framework/ORM does. Furthermore, seemingly read-only web middleware often ends up writing to database on each request for one reason or another. If you try to reuse/pool connections (which is also important under high load), you need to be wary of keeping open writable connections in cache — again, something that does not matter to all major databases other than SQLite.
I was involved in maintenance of a small web app (db size < 5 Mb), written in Django, that had to serve ~1000 dynamic requests per second (the contents of each response were dependent on IP address of caller). The app worked with PostgreSQL, albeit poorly, but immediately ground to halt under load with SQLite — which was our default database choice for historical reason.
We ended up briefly caching results of most database queries in memory, which removed most of load from database (we also did a lot of other optimizations, but this was the decisive one). Eventually the app was able to withstand up to 9000 requests per second, but none of that was an achievement of SQLite — we just evaded database, Django and Python altogether on majority of requests.
While we are on this topic, the most widespread OS in the world, Android, also has extremely low-quality SQLite drivers — despite shipping SQLite as default database for many years. Android has a broken-by-design Cursor implementation (the devs admitted it themselves [1]), that always tries to count query results, even if you don't call getCount(). And a broken connection cache, that does not support read-only connections [2] (that method used to have a "TODO", but eventually they forgot, why they wanted it, so they removed it).
1: https://medium.com/androiddevelopers/large-database-queries-on-android-cb043ae626e8 https://medium.com/androiddevelopers/large-database-queries-...
2: https://android.googlesource.com/platform/frameworks/base/+/android10-release/core/java/android/database/sqlite/SQLiteConnectionPool.java#1039 https://android.googlesource.com/platform/frameworks/base/+/...
- benbjohnson 6y agoSQLite doesn’t require you to know which connections are read and which are write. Connections get promoted to a write lock when you do a write DDL or if you begin an IMMEDIATE transaction. Also, for your small app were you using WAL mode? Multiple readers only works in WAL journaling mode.
- altfredd 6y agoSQLite can promote connections, but it does not demote them back to read-only. As I understand, when a database connection is tainted by a single write, it's lock on SQLite file in promoted to write lock, which prevents other processes from opening any kind of connection to it. Our Django setup needs multiple processes to work around the Grand Interpreter Lock. Disabling connection reuse in Django config slightly changed the behavior we observed, but didn't solve the performance problem.