4 ms·
I recently wrote a production system that uses SQLite as the main backend. SQLite is in memory in this case and its entire state gets rebuilt from Kafka on sta
by meitham 3y ago
I recently wrote a production system that uses SQLite as the main backend. SQLite is in memory in this case and its entire state gets rebuilt from Kafka on start. The DB receives about 2 updates a second, wrapped with rest api aiohttp and odata filters. It has been able to handle close to 9k requests/second ands it’s a primary system in a financial institution. So yes SQLite is fully capable prod db.
- endisneigh 3y agoYou’re using SQLite and Kafka? Very ironic.
- meitham 3y agoIn large organisations you often have no choice of the type of queue between your team and other teams. That being said there’s nothing wrong with Kafka and the ability to seek back to the earliest timestamp since midnight and being able to rebuild our state from that is a godsend feature, in comparison to other queues. This means we can make our application stateless or at least afford to lose the state and be able to build it quickly from Kafka.
- endisneigh 3y agoKafka is just fine. I just thought it was funny that SQLite would be involved at all.
- morelisp 3y agoKafka for consistency/durability and local storage for speed/structure is a common architecture, and a really good one any time you can tolerate async writes.
- paulryanrogers 3y agoSo your source of truth is ... Kafka?
- jerrygenser 3y agoIf you don't need SQL (relational data), but maybe have a schema per topic, I've used rocksdb as a cache for latest in tombstones topic. It has high write throughput for rebuilding state when playing forward a stream