9 ms·
SQLite in Production: Lessons from Running a Store on a Single File
- ryguz 7mo ago[dead]
- sgbeal 7mo ago> json_extract returns native types. json_extract(data, '$.id') returns an integer if the value was stored as a number. Comparing it to a string silently fails. Always CAST(json_extract(...) AS TEXT) when you need string comparison. More simply: sqlite> select typeof('{a:1}'->>'a') ; ╭──────────────────────╮ │ typeof('{a:1}'->>... │ ╞══════════════════════╡ │ integer │ ╰──────────────────────╯ vs: sqlite> select typeof('{a:1}'->'a') ; ╭──────────────────────╮ │ typeof('{a:1}'->'a') │ ╞══════════════════════╡ │ text │ ╰──────────────────────╯
- yokuze 7mo ago> The technical fix was embarrassingly simple: stop pushing to main every ten minutes. Wait, you push straight to main? > We added a rule — batch related changes, avoid rapid-fire pushes. It's in our CLAUDE.md (the governance file that all our AI agents follow): > Avoid rapid-fire pushes to main — 11 pushes in 2h caused overlapping Kamal deploys with concurrent SQLite access. Wait, you let _Claude_ push your e-commerce code straight to main which immediately results in a production deploy?
- crabmusket 7mo agoPatient: doctor, my app loses data when I deploy twice during a 10 minute interval! Doctor: simply do not do that
- pavel_lishin 7mo agoDoctor: solution is simple, stop letting that stupid clown Pagliacci define how you do your work! Patient: but doctor,
- pjc50 7mo agopAIgliacci: as a large language model, I am unable to experience live comedy.
- rcakebread 7mo agoBob Newhart did it best https://www.youtube.com/watch?v=LhQGzeiYS_Q https://www.youtube.com/watch?v=LhQGzeiYS_Q
- bombcar 7mo agoHey, Apple still takes their store down during product launches!
- pstuart 7mo agoI assumed that it was to ensure that the announced products were revealed in a controlled manner rather than because they aren't able to do updates to their product listings as a regular thing.
- bombcar 7mo agoMy reading of the tea leaves is it started out as the latter and continues as the former as part of the “mystique”.
- tensegrist 7mo agoi hate to be so blunt but look around the site and then tell me you're surprised
- xnorswap 7mo agoI'm fairly confident they let it write the blog post too.
- simonw 7mo ago"Not as a proof of concept. Not for a side project with three users. A real store" - suggestion for human writers, don't use "not X, not Y" - it carries that LLM smell whether or not you used an LLM.
- xnorswap 7mo agoAnd that's just the opening paragraph, the full text is rounded off with: "The constraint is real: one server, and careful deploy pacing." Another strong LLM smell, "The <X> is real", nicely bookends an obviously generated blog-post.
- These335 7mo agoYou're absolutely right, this was an AI post
- yokuze 7mo agoI see what you did there XD
- chasil 7mo agoThis is the actual problem: "Kamal runs blue-green deploys — it starts a new container, health-checks it, then stops the old one. During the switchover, both containers are running. Both mount ultrathink_storage. Both have the SQLite files open." WAL mode requires shared access to System V IPC mapped memory. This is unlikely to work across containers. In case anybody needs a refresher: https://en.wikipedia.org/wiki/Shared_memory https://en.wikipedia.org/wiki/Shared_memory https://en.wikipedia.org/wiki/CB_UNIX https://en.wikipedia.org/wiki/CB_UNIX https://www.ibm.com/docs/en/aix/7.1.0?topic=operations-system-v-interprocess-communication-ipc https://www.ibm.com/docs/en/aix/7.1.0?topic=operations-syste...
- Retr0id 7mo ago> This is unlikely to work across containers. Why not?
- simonw 7mo agoThanks for this, the anecdote with the lost data was very concerning to me. I think you're exactly right about the WAL shared memory not crossing the container boundary. EDIT: It looks like WAL works fine across Docker boundaries, see https://news.ycombinator.com/item?id=47637353#47677163 https://news.ycombinator.com/item?id=47637353#47677163 I don't know much about Kamal but I'd look into ways of "pausing" traffic during a deploy - the trick where a proxy pretends that a request is taking another second to finish when it's actually held in the proxy while the two containers switch over. From https://kamal-deploy.org/docs/upgrading/proxy-changes/ https://kamal-deploy.org/docs/upgrading/proxy-changes/ it looks like Kamal 2's new proxy doesn't have this yet, they list "Pausing requests" as "coming soon".
- chasil 7mo agoYou might consider taking the database(s) out of WAL mode during a migration. That would eliminate the need for shared memory.
- Retr0id 7mo ago> I think you're exactly right about the WAL shared memory not crossing the container boundary. I don't, fwiw (so long as all containers are bind mounting the same underlying fs).
- littlestymaar 7mo ago> Wait, you let _Claude_ push your e-commerce code straight to main which immediately results in a production deploy? Yikes. Thank you I'm not going to read “Lessons learned” by someone this careless.
- 66yatman 7mo agoThe issue wasn’t done by the ai but their lack of architectural knowledge
- okkdev 7mo agoGoes hand in hand
- whateveracct 7mo agostupid is as stupid does
- burnt-resistor 7mo agoI suspect they don't wear helmets or seatbelts either. Sigh. The "I'm so proud and ignorant of unnecessarily risky behaviors" meme is tiring. The Meta dev model of diff reviews merge into main (rebase style) after automated tests run is pretty good. Also, staging and canary, gradual, exponential prod deployment/rollback approaches help derisk change too. Finally, have real, tested backups and restore processes (not replicated copies) and ability to rollback.
- leosanchez 7mo ago> Backups are cp production.sqlite3 backup.sqlite3 I use gobackup[0] as another container in compose.yml file which can backup to multiple locations. [0]: https://gobackup.github.io/ https://gobackup.github.io/
- hedora 7mo agoDoes cp actually work on live sqlite files? I wouldn’t expect it to, since cp does not create a crash-consistent snapshot.
- sgbeal 7mo ago> Does cp actually work on live sqlite files? I wouldn’t expect it to, since cp does not create a crash-consistent snapshot. cp "works" but it has a very strong possibility of creating a corrupt copy (the more active the db, the higher the chance of corruption). Anyone using "cp" for that purpose does not have a reliable backup. sqlite3_rsync and SQLite's "vacuum into" exist to safely create backups of live databases.
- 66yatman 7mo agoMaybe if the system is idle
- faangguyindia 7mo agoI've a busy app, i just deploy to canary. And use loadbalancer to move 5% traffic to it, i observe how it reacts and then rollout the canary changes to all. how hard and complex is it to roll out postgres?
- pezh0re 7mo agoNot hard at all - geerlingguy has a great Ansible role and there are a metric crapton of guides pre-AI/2022 that cover gardening.
- cadamsdotcom 7mo agoThe fix appears to nicely asking the forgetful unreliable agent to please (very closely pretty please!) follow the deploy instructions (and also please never hallucinate or mess up, because statistics tells us an entity with no long term memory and no incentive to get everything right will do the job right 99.99999999% of the time, which is good enough to run an eshop) not deploy too often per hour. With one simple instruction the system (99.9999% of the time) gains the handy property that “only” two processes end up with the database files open at once. Thanks for the vibes!
- devmor 7mo agoI have to work with agents as a part of my job and the very first thing I did when writing MCP tools for my workflow was to ensure they were read only or had a deterministic, hardcoded stopgap that evaluates the output. I do not understand the level of carelessness and lack of thinking displayed in the OP.
- mywittyname 7mo agoEven just having the agent write scripts to disk and run those works wonders. It keeps the agent from having to rebuild a script for the same tasks, etc.
- devmor 7mo agoThat too! Every time the agent does something I didn't intend, I end up making a tool or process guidance to prevent it from happening again. Not just add "don't do that" to the context.
- politelemon 7mo ago> embarrassingly simple This is becoming the new overused LLM goto expression for describing basic concepts.
- Natfan 7mo agollm generated article. please consider writing it yourself. quirks in human writing is infinitely more interesting than a next-token-predicted 500 word piece
- NewsaHackO 7mo agoBut then how would they get people to buy their $99 AI CEO package?
- pullshark91 7mo agoHuh, and here I thought it was a joke...
- NewsaHackO 7mo agoMaybe it is, didn't really look into it.
- jmull 7mo agoRedis, four dbs, container orchestration for a site of this modest scope… generated blog posts. Our AI future is a lot less grand than I expected.
- ramon156 7mo agoHow else will you get all those resume entries ! (/j)
- add-sub-mul-div 7mo agoIronically, AI de-skilling results in a robust-sounding resume.
- jszymborski 7mo agoThe LLM prose are grating read. I promise, you'd do a better job yourself.
- littlestymaar 7mo agoGiven how dumb their workflow is (let Claude Code push directly to production without supervision) I'm not so sure.
- infamia 7mo agoSQLite has a ".backup" command that you should always use to backup a SQLite DB. You're risking data loss/corruption using "cp" to backup your database as prescribed in the article. https://sqlite.org/cli.html#special_commands_to_sqlite3_dot_commands_ https://sqlite.org/cli.html#special_commands_to_sqlite3_dot_...
- anonzzzies 7mo agoYeah, using cp to backup sqlite is a very bad idea. And yet, unless you know this, this is what Claude etc will implement for you. Every friggin' time.
- chasil 7mo agoIt's fine if you run the equivalent of "init 1" first. Does your OS have a single-user mode?
- hombre_fatal 7mo agoWell, humans also default to 'cp' until they learn the better pattern or find out their backup is missing data. Also, my n=1 is that I told Claude to create a `make backup` task and it used .backup. I don't understand the double standard though. Why do we pretend us humans are immaculate in these AI convos? If you had the prescience to be the guy who looked up how to properly back up an sqlite db, you'd have the prescience to get Claude to read docs. It's the same corner cut. There's this weird contradiction where we both expect and don't expect AI to do anything well. We expect it to yolo the correct solution without docs since that's what we tried to make it do. And if it makes the error a human would make without docs, of course it did, it's just AI. Or, it shouldn't have to read docs, it's AI.
- refulgentis 7mo agoYou're confusing a workman's winking complaint about their tool, with, being unfair by not treating AI like a human.
- 7mo ago
- pgideas 7mo ago[dead]
- crazygringo 7mo ago> Would We Choose SQLite Again? Yes. For a single-server deployment with moderate write volume, SQLite eliminates an entire category of infrastructure complexity. No connection pool tuning. No database server upgrades. No replication lag. These are weird reasons. You can just install Postgres or MySQL locally too. Connection pool tuning certainly isn't anything you have to worry about for a moderate write volume. You don't ever need to upgrade the database if you don't want to, since you're not publicly exposing it. There's obviously no replication lag if you're not replicating, which you wouldn't be with a single server. The reason you don't usually choose SQLite for the web is future-proofing. If you're totally sure you'll always stay single-server forever, then sure, go for it. But if there's even a tiny chance you'll ever need to expand to multiple web servers, then you'll wish you'd chosen a client-server database from the start. And again, you can run Postgres/MySQL locally, on even the tiniest cheapest VPS, basically just as easily as using SQLite.
- xnorswap 7mo agoYeah, it's weird "they" don't consider any middle ground between SQLite and replicated postgres cluster. Locally running database servers are massively underrated as a working technology for smaller sites. You can even easily replicate it to another server for resiliency while keeping the local performance.
- talkingtab 7mo agoThis. Spinning up Postgresql is easy once you know how. Just as SQLITE3 is easy once you know how. But I can find no benefit from not just learning postgres the first time around.
- kaibee 7mo agoThey're using AI Agents to do it in either case and using docker. There was no reason to choose SQLite.
- kaibee 7mo agoYeah a PG Docker container is basically magic. I too went down a rabbit-hole of trying to setup a write-heavy SQLite thing because my job is still using CentOS6 on their AWS cluster (don't ask). Once I finally got enough political capital to get my own EC2 box I could put a PG docker container on, so much nonsense I was doing just evaporated.
- nop_slide 7mo agoI still haven't figured out a good way to due blue/green sqlite deploys on fly.io. Is this just a limitation of using sqlite or using Fly? I've been very happy with sqlite otherwise, rather unsure how to do a cutover to a new instance. Anyone have some docs on how to cutover gracefully with sqlite on other providers?
- wolttam 7mo agoYou accept downtime. That's the limitation of SQLite. Or you use some distributed SQLite tool like rqlite, etc
- nop_slide 7mo agoI'm personally fine with a little bit of downtime for my particular small app. I'm just surprised there's not a more detailed story around deploying sqlite in a high availability prod environment given it's increased popularity and coverage over the last few years. Especially surprising with Rails' (my stack) going full "sqlite-first".
- wolttam 7mo agoThe "sqlite-first" folks have accepted that a bit of downtime is better than engineering wildly complex systems that avoid it, for non-mission-critical apps (if your mission is a low volume e-commerce shop.. it's not critical)
- jmull 7mo agoI don't know if it's just me, but this whole post seems to have time traveled forward from about 3-4 days ago. It's not just a repost. The thread includes a comment I made at the time which now from "1 hour ago". Makes me wonder if it's an honest bug or someone has hacked the hacker news front page to sell their t-shits, mugs, and AI starter kits.
- Retr0id 7mo agoIt's an artefact of the "second chance pool" mechanism.
- worksonmine 7mo agoInteresting choice to change the time of the comment, a deja-vu can be weird enough without staring at a comment with a recent timestamp.
- trelliumD 7mo agocould have used firebird embedded, also a simple deployment such as sqllite, but better concurrency and more complete system, also a tad faster
- NicoJuicy 7mo agoIf Nico send him an email. The AI CEO should take his offer.
- mt42or 7mo agoNIH syndrome, almost mental health issues.
- minutesmith 7mo ago[flagged]
- jp0001 7mo agoI took three weeks off from tech, read books from last century, and travelled Europe. Coming back, reading LLM generated content and code feels like nails on a chalkboard. Taste, it does not have taste.
- PunchyHamster 7mo agoIt is so tiring...
- literallyroy 7mo agoIt’s strange how easy it is to spot.
- 66yatman 7mo agoJust use a 4gb server and install Postgres
- MagicMoonlight 7mo agoSlopcoded article for a Slopcoded website
- PunchyHamster 7mo ago> Yes. For a single-server deployment with moderate write volume, SQLite eliminates an entire category of infrastructure complexity. No connection pool tuning. No database server upgrades. No replication lag. None of these is needed if you run sqlite sized workloads... I like SQLite but right tools for right jobs... tho data loss is most likely code bug
- deleted 7mo ago[deleted]
- heikkilevanto 7mo agoI use SqLite for a small hobby project, fine for that. Wanted to read the article to see why I should not, but it attacked me with a "subscribe" popup, so I stopped there. The comments here seem to be based on daydreaming on scaling to a lot of users who need 24/7 uptime, which is not always the case.
- kristiandupont 7mo agoSQLite is a rock solid piece of software that offers a great value prop: in-process database. For locally running apps (desktop or mobile), this makes perfect sense. However, I genuinely don't see the appeal when you are in a client/server environment. Spinning up Postgres via a container is a one-liner and equally simple for tests (via testcontainers or pglite). The "simple" type system of SQLite feels like nothing but a limitation to me.
- adobrawy 7mo agoIf the problem is excessive deployments via GitHub Actions, why not use concurrency control on GitHub Actions ( https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-workflow-concurrency https://docs.github.com/en/actions/how-tos/write-workflows/c... ) instead of relying on agent randomness and the hope that it won't make the same mistake again? Am I missing something?
- rienbdj 7mo agoA well designed system shouldn’t drop orders? If you perform at least once processing then use Stripe idempotency keys you avoid such issues?
- mattrighetti 7mo agoI see tons of articles like this, and I have no doubt sqlite proved to be a great piece of software in production environments, but what I rarely find discussed is that we lack tools that enable you to access and _maintain_ SQLite databases. It's so convenient to just open Datagrip and have a look at all my PostgreSQL instances; that's not possible with sqlite AFAIK (not even SSH tunnelling?). If something goes wrong, you have to SSH into the machine and use raw SQL. I know there are some cool front-end interfaces to inspect the db but it requires more setup than you'd expect. I think that most people give up on sqlite for this reason and not because of its performance.
- simonw 7mo agoI have a project to help with that: uvx datasette data.db That starts a web app on port 8001 that looks like this: https://latest.datasette.io/fixtures https://latest.datasette.io/fixtures
- philkrylov 7mo agoHi, thanks for your project! Do you accept pull requests? https://github.com/simonw/datasette/pull/2616 https://github.com/simonw/datasette/pull/2616 https://github.com/simonw/datasette/pull/2617 https://github.com/simonw/datasette/pull/2617
- siruwastaken 7mo agoAm I the only one finding this article highly suspect? It seems like the errors made are so basic, i.e. using the wrong SQL dialect for the db system in use, and there orders were apparently only at 17?
- sgarland 7mo ago> The sqlite_sequence table is the most underappreciated debugging tool in SQLite. It tracks the highest auto-increment value ever assigned for each table — even if that row was subsequently lost. > WorkQueueTask.count returns ~300 (current rows). The sequence shows 3,700+ (every task ever created). If those numbers diverge unexpectedly, something deleted rows it shouldn't have. Or it means that SQLite is exhibiting some of its "maybe I will, maybe I won't" behavior [0]: > Note that "monotonically increasing" does not imply that the ROWID always increases by exactly one. One is the usual increment. However, if an insert fails due to (for example) a uniqueness constraint, the ROWID of the failed insertion attempt might not be reused on subsequent inserts, resulting in gaps in the ROWID sequence. AUTOINCREMENT guarantees that automatically chosen ROWIDs will be increasing but not that they will be sequential. > No ILIKE. PostgreSQL developers reach for WHERE name ILIKE '%term%' instinctively. SQLite throws a syntax error. Use WHERE LOWER(name) LIKE '%term%' instead. You should not be reaching for ILIKE, functions on predicates, or leading wildcards unless you're aware of the impacts those have on indexing. > json_extract returns native types. json_extract(data, '$.id') returns an integer if the value was stored as a number. Comparing it to a string silently fails. Always CAST(json_extract(...) AS TEXT) when you need string comparison. If you're using strings embedded in JSON as predicates, you're going to have a very bad time when you get more than a trivial number of rows in the table. 0: https://sqlite.org/autoinc.html https://sqlite.org/autoinc.html
- fbuilesv 7mo ago> The Fix: Stop Deploying So Fast I don't know how Ultrathink works, and I have no "real world" experience with Kamal, but I find it intriguing to see someone consider 11 deployments in 2 hours to be "fast". Instead of handicapping yourself, fix your deployment pipeline, 10 min deploys are not OK for an online store.
- nonameiguess 7mo agoHow the hell did this get so much engagement, let alone as a repost? Is "SQLite" in the title really all it takes? This site was registered 8 months ago, the whole blog started in February, the first post declares its writer to be an AI CEO, most posts are about hiring and managing other AI agents, it claims to sell everything from coffee mugs to services from itself to also be the CEO of your business. This feels more like performance art than a business. There's no evidence they've sold anything or even have any actual inventory. AI agents can't build you a warehouse or manufacture physical goods. You guys are arguing with a bot, in a way almost arguing with yourselves, as it may very well not have actually done any of this, is definitely not running a "real store," and is seemingly publishing posts that are a parody of Hacker News style founder journeys but if the founders were bots.
- stock-parrot 7mo agoYou are absolutely right! Everything feels off and done for engagement Is this what we can expect in the near future?
- elliot07 7mo agoYou should use Shopify and focus on the store part of the business, not the SQLite part :)
- bob1029 7mo agoI think SQLite is fantastic but it does start to fall apart at the edges sometimes. What is more interesting to me is the fact that everyone seems to think of Postgres as the obvious alternative to SQLite. It is certainly an alternative. For me, the most opposite thing of SQLite is something like Oracle or MSSQL. The complexity being relatively constant is the part I care about most here. Running a paid, COTS database engine on a blessed OS tends to be a little bit easier than an OSS solution that can run on toasters and drones. Especially, if you are using replication, high availability, etc. The business liability coverage seems to track proportionally with how much money you spend on the solution. SQLite offers zero guarantees accordingly. You don't have a support contract or an account manager you can get upset with. Depending on the nature of the business this could be preferable or adverse. It really depends. For serious regulated business with oppressive audit cycles, SQLite trends toward liability more than asset if it's being used as a system of record. That it merely works and performs well is often not sufficient for acceptance. I'm not saying that Postgres isn't capable of passing an intense audit, but I am saying that it might be easier to pass it if you used MSSQL. The cost of having your staff tied up with compliance should be considered when making technology choices in relevant businesses.
- throwaway173738 7mo agoMost of us don’t work in businesses where indemnity coverage is more important than licensing costs.
- bob1029 7mo agoI agree these exist but I don't know about "most". Licensing costs are almost always a drop in the bucket compared to things like your salary.
- mergisi 7mo ago[dead]
- talkfold 7mo agoLost two orders, diagnosed it with sqlite_sequence, fixed it by telling the AI agents to slow down. That's a new genre of postmortem.