3 ms·
I’ve worked on several projects with sqlite, both read and write heavy, all with high concurrency, with databases in the few hundred MB with 400k server clients
by matthewaveryusa 4y ago
I’ve worked on several projects with sqlite, both read and write heavy, all with high concurrency, with databases in the few hundred MB with 400k server clients, and 100 bare-metal servers running at capacity. The sqlite part of our system is never the problem. In our case sqlite has been an alternative to custom files on disk or replacing a spaghetti of hashmaps in memory. we also replaced a single postgresql instance with all customers into many sqlites per customer. Performance and reliability is why I always reach for it first. At this point I’m a zealot and would argue your first ‘data structure’ of choice should be an sqlite database :)
- Mertax 4y agoWhat pain points have you experienced with "many sqlites per customer". I'm considering transitioning to something similar but would love to know what pitfalls I might not be considering.
- matthewaveryusa 4y agoNone really, but we have a fairly simple design where all the shared databases we attach are read-only (think big static lookup tables that a separate process takes care of updating.) I would probably avoid having databases attach contextually -- seems complicated and error-prone.
- giantrobot 4y agoI'm also in the SQLite is a data structure camp. You get fast access and persistence for free. You can also fix a lot of issues just exploring the database offline.
- ricg 4y agoDid you ever run into the problem of the system running out of file descriptors? If each client gets its own database file, each connection will require a file descriptor. For example, Linux limits the number of file descriptors to 1024 per process to each process can open up to 1024 files at any point in time. I ran into this problem in the past in another project (the system ran out of file descriptors for another reason).