3 ms·
Outside of the DB: some kind of queue. Could be a SQS or an in memory queue in your app (hand waves about mutiple nodes but that can be dealt with or use redis
by aqueueaqueue 2y ago
Outside of the DB: some kind of queue.
Could be a SQS or an in memory queue in your app (hand waves about mutiple nodes but that can be dealt with or use redis). Have a single processor that aggregates and does writes. You can use DLQ if something goes wrong to avoid data loss.
Inside the DB maybe just append rows for each add to a table without indexes. That way the engine probably just needs to write out a journal record and append but doesn't spend time on updating indexes. Then periodically read all from the table and aggregate into another. You could cycle between 2 tables.
These is eventually consistent in aggregations but transactional in source of truth. I.e. if you write you can be sure the data is stored, but may take time to aggregate.
Or Use a more exotic DB. That is distributed but still supports transactions.