9 ms·
Django SQLite Production Config
- anze3db 2y agoHey HN! I'm the author of the blog post. I didn't mention this in the post, but I'll try to merge some of these settings into Django by making them the new default or the default for new projects created with the start project command. Any feedback on all of this is greatly appreciated!
- kakulukia 2y agoThx for this!
- stefanos82 2y agoWhen you say " I'll try to merge some of these settings into Django by making them the new default or the default for new projects created with the start project command", you mean you are a Django core developer and you plan to make them as part of the Django project? If you are, YES PLEASE! :D I would love to have those settings as my default configuration, because the majority of my projects are tiny to small to between-small-and-medium size, therefore SQLite is more than enough for me.
- anze3db 2y agoI'm not a Django core dev, but I have managed to get my changes merged into Django already (the transaction_mode setting in 5.1 was my contribution). Carlton does seem to be onboard with my idea[0], so I'm optimistic that we can make it happen. Comments like yours will help me make my case so thank you for that! [0] https://fosstodon.org/@carlton/112605212812578926 https://fosstodon.org/@carlton/112605212812578926
- cqqxo4zV46cp 2y agoIIRC Django has rightfully killed the whole “core dev” thing. And, in reality, new contributors are getting PRs merged constantly.
- anze3db 2y agoI can confirm. My experience contributing has been very positive!
- belter 2y agoDjango is a critical project, and I am sure your contributions are of high quality. But making it too easy to get new contributions in, somehow raises other concerns around the project governance.
- anze3db 2y agoDon't get me wrong, the PR review is still rigorous and can take weeks. But everybody I interacted with during the process was very supportive and helpful, so the overall experience was great!
- kgeist 2y agoI want to warn that one huge issue with SQLite in WAL mode is that if your site gets a high load, the WAL file will grow unboundedly. I ran a stress test of 14k RPS (that's the maximum my PC can pull off in my Go application, but it probably can happen with a more modest RPS) and the WAL file quickly exploded to tens of gigabytes, which can render your machine inoperable (it almost broke my PC). The default checkpointer can't properly function if the DB is continuously written to, due to the internal limitations of SQLite. I managed to solve it by running a separate goroutine (thread) which monitors the WAL size on disk every second: if it goes above the target size of 8 MB, my framework initiates the "slow down" mode where all reads and writes are slowed down by artificially calling sleep(), starting with 16ms and gradually increasing the sleep time according to a few heuristics. This allows the application to have small time gaps where no reads or writes happen and the checkpointer can actually proceed (the goroutine activates it manually). The slow down mode is deactivated when the WAL size is within the target size again. I think SQLite in WAL mode is not really fit for production without this kind of hack.
- anze3db 2y agoI've heard this is a potential issue but I've never encountered it. Do you know about the experimental WAL2 branch[0] that splits the WAL file into two to circumvent this problem? You'd have to compile SQLite yourself from the WAL2 branch to try it out, but it might be worth it at your scale. Python is much slower than Go, so I don't think we could get 14k RPS as easily with Django, but I do have to see if I can reproduce the problem in Django. Topic for a future blog post! Thanks for sharing this, I really appreciate knowing where the limits of SQLite are! [0] https://www.sqlite.org/cgi/src/doc/wal2/doc/wal2.md https://www.sqlite.org/cgi/src/doc/wal2/doc/wal2.md
- baq 2y agoI love this story because it shows you really should read some docs about your storage layer before you jump in head first and just assume it’ll work forever and/or gracefully handle any load including overload. Postgres is the same, it’ll work until it won’t, in some case it’ll break in a way you can’t recover from without stopping production for a long time. (You can guess how I know - you’re right, I didn’t read the relevant part of the manual.) Read your database’s manual, people! Even just going through the table of contents will put you in the top 20%.
- tzot 2y agoIs there a point for the PRAGMA journal_size_limit when we set the database to WAL mode?
- anze3db 2y agoFrom what I know, the journal_size_limit PRAGMA still affects WAL mode, but it doesn't solve the issue of the WAL file potentially growing uncontrollably. Am I missing something?
- kgeist 2y agohttps://sqlite.org/forum/info/54e791a519a225de https://sqlite.org/forum/info/54e791a519a225de >Journal size limit is measured in bytes and only applies to an empty journal file. >The WAL file will grow without bounds until a checkpoint takes place that reaches the very end of the WAL file. Usually, a checkpoint is performed when a commit causes the WAL file to be longer than 1000 pages (PAGES, not bytes). There are conditions when running a checkpoint to completion is not possible, like disabled checkpointing, checkpoint starvation because of open read transactions and large write transactions.
- tzot 2y agoIndeed on rereading my question I see it was not phrased correctly. Yes, the `journal_size_limit` affects the maximum journal/wal file that remains on disk if larger than that — and by the way ensures that these files are not deleted once created. Your setting it to ~25MiB while the default `wal_autocheckpoint` PRAGMA is set to 1000 pages (with the typical page size of 4KiB that means after ~4MiB the WAL file contents will get moved to the main database file if no other transaction is active) is what confused me. 25MiB seems very specific for a file size to keep in the occasion that the WAL file keeps growing beyond 4MiB. Perhaps you also meant to tinker with the `wal_autocheckpoint` PRAGMA but didn't?
- leetrout 2y agoThere's also a great explanation of SQLite capabilities on the server and the various settings and their effects which have some overlap with Anže's settings: https://kerkour.com/sqlite-for-servers https://kerkour.com/sqlite-for-servers Previous (brief) HN discussion on that post: https://news.ycombinator.com/item?id=39383725 https://news.ycombinator.com/item?id=39383725 And, if this piques your interest, there was recently discussion on distributed SQLite from the same author: https://news.ycombinator.com/item?id=39975596 https://news.ycombinator.com/item?id=39975596
- anze3db 2y agoStephen has a whole series of blog posts on SQLite (in Rails) that I highly recommend. I've learned most of what I know about SQLite from him! https://fractaledmind.github.io/2024/04/15/sqlite-on-rails-the-how-and-why-of-optimal-performance/ https://fractaledmind.github.io/2024/04/15/sqlite-on-rails-t... Also, hello Lee! I miss being in a Slack with you!
- hu3 2y agoDoes anyone have experience with running SQLite with mounted Docker volumes in production? I wonder if Docker provides all the i/o features that SQLite requires to function properly.
- deleted 2y ago[deleted]
- kissgyorgy 2y agoI did the exact same thing yesterday, couple of notes: If you subclass sqlite3.base.DatabaseWrapper, it will issue the required PRAGMAs defined by Django, where foreign_keys = ON and legacy_alter_table = OFF. I don't think synchronous = NORMAL worth it, as there is a tiny-tiny chance you will lose data. The relevant section from SQLite doc: "A transaction committed in WAL mode with synchronous=NORMAL might roll back following a power loss or system crash." IMMEDIATE mode might not be needed and your application might never get a database locked error, I would only use that when I see the first error. You can read about mmap_size in depth here: https://oldmoe.blog/2024/02/03/turn-on-mmap-support-for-your-sqlite-connections/ https://oldmoe.blog/2024/02/03/turn-on-mmap-support-for-your...
- dajonker 2y agoSQLite is a great database but it's not suited for applications with frequent writes or write-transactions that take longer than a couple of milliseconds. I tried basically everything in this blog post with our Rails business application and none of it really works in practice. I expect the same to be true for Django. The reason is that with Rails, you usually end up with a bunch of callbacks on models which cause "long" running transactions. For example, the application writes a record to the database, then uses the newly generated primary key to create a bunch of related records, etc. With a reasonable amount of business logic, this means that a transaction can easily run over 100 milliseconds. Not because the database is slow (it's not), but because the application may do all kinds of slow stuff in between the different statements that run in a transaction. SQLite is single threaded when it comes to writes, so when an average write can take 100 ms, your throughput is already limited to about 10 transactions per second. IMMEDIATE transactions are indeed necessary, to wait dor the database to be ready before attwempting any transaction. Because at least it is easy to have a backoff/retry strategy before letting the framework run its transaction logic. However, even with just a handful of active users, I needed to make sure that waiting transactions were retried a large number of times, to prevent users from getting a 500 error and requiring them to perform the same action again. However, users were complaining that the application was slow, and I had the metrics that told me the same. Eventually I just switched to Postgres and all of the issues just disappeared. The users also immediately told me that the application became much more responsive. I did notice in the metrics that a lot of the read operations actually became slower, as SQLite is really efficient at doing lots of small reads as compared to a client/server model database. I do still think that SQLite is very suited for production applications, but only when it is read heavy or has very lightweight write transactions.
- anze3db 2y agoI agree 100% with everything you wrote. It was very surprising to me that every transaction and every write operation blocks the whole database and not only the table it's performed upon (it makes sense since it's all a single file, but still). The only workaround for this is splitting your main database into multiple databases, but this bleeds into your application logic and gets messy quickly. If you are in this position, it's best to switch to Postgres as you did!