2 ms·

Where I part ways a little with the recommendation is in the busy_timeout + BEGIN IMMEDIATE suggestion. On an embedded system, that's not really sufficient; I m

Where I part ways a little with the recommendation is in the busy_timeout + BEGIN IMMEDIATE suggestion. On an embedded system, that's not really sufficient; I maintain a caching service that is used by thousands of clients. It ingests data over MQTT and has a web interface for queries. In my design, there is a MQTT thread and pruning thread. The MQTT ingestion thread and the pruning thread can both end up "contending" for the write lock, and increasing the busy_timeout only makes them wait longer before noticing that contention and moving on to the next task. What we ended up needing to do was add an application-level lock that only allows a single writer transaction to be in flight at a time; BEGIN IMMEDIATE is nice for preventing other readers from blocking, but it doesn't help much with other writers. Meanwhile, you might be surprised how often “single-tenant edge deployment” comes up in embedded contexts; a Pi with an SD card is a common configuration for an embedded database, and it's right at the intersection of these problems.

hinkley 3mo ago
One of the lessons I picked up from my very brief foray into the world of realtime software is the idea of amortization. If you have one activity that is possibly time-unbounded, then the solution is to trade away a bit of throughput for certainty by taxing the frequent task with making incremental progress on the cleanup. The first real-time garbage collectors worked this way. Every allocation had to do a bit of sweeping work, which generally keeps all but the largest allocations from every being in danger of hitting a pause (in real time systems large allocations are typically done at bootstrapping time to avoid this issue entirely) > Meanwhile, you might be surprised how often “single-tenant edge deployment” comes up in embedded contexts; a Pi with an SD card is a common configuration for an embedded database, and it's right at the intersection of these problems. Probably shouldn't be. One of the more high profile use cases for SQLite is per-customer sharding. Particularly with services where at any given time a small number of customers are generating the majority of the traffic. The data for customers that you haven't seen logged in for weeks tends to have a logarithmic effect on the query time for active customers when you use a single shared database for all of the data. Unloading that data reduces the cost of interactive workloads handily, even though the cost of backups may still be either the same or still have a logarithmic cost (eg, rsync detects deltas).

hinkley 3mo ago

One of the lessons I picked up from my very brief foray into the world of realtime software is the idea of amortization. If you have one activity that is possibly time-unbounded, then the solution is to trade away a bit of throughput for certainty by taxing the frequent task with making incremental progress on the cleanup. The first real-time garbage collectors worked this way. Every allocation had to do a bit of sweeping work, which generally keeps all but the largest allocations from every being in danger of hitting a pause (in real time systems large allocations are typically done at bootstrapping time to avoid this issue entirely) > Meanwhile, you might be surprised how often “single-tenant edge deployment” comes up in embedded contexts; a Pi with an SD card is a common configuration for an embedded database, and it's right at the intersection of these problems. Probably shouldn't be. One of the more high profile use cases for SQLite is per-customer sharding. Particularly with services where at any given time a small number of customers are generating the majority of the traffic. The data for customers that you haven't seen logged in for weeks tends to have a logarithmic effect on the query time for active customers when you use a single shared database for all of the data. Unloading that data reduces the cost of interactive workloads handily, even though the cost of backups may still be either the same or still have a logarithmic cost (eg, rsync detects deltas).