4 ms·
Can you elaborate on what it is that slows down your app performance? Is it bottle-necking on writing to the database? Does the database have read activity? Doe
by storyinmemo 10y ago
Can you elaborate on what it is that slows down your app performance? Is it bottle-necking on writing to the database? Does the database have read activity? Does the application slow down when there is read activity?
With some basic assumptions on my part including that you can have a delay in writing data to the database (since it's archival and analysis), but you don't want the application to be delayed, putting a fast queue like Kafka or RabbitMQ in place could help. It'll buffer when the database is under load, isolating your application from that.
Offline analysis might also skip the database entirely: columnar data formats (parquet) or log structured merge / sorted string tables (rocksdb) could work very well for archival or offline analysis in MR type environments.
- cocoflunchy 10y agoYes, we only have one DB for everything and it is bottle-necking on writes. Queuing the writes is probably the easiest solution since we already have RabbitMQ setup... thanks for your answer.
- saosebastiao 10y agoThere are a lot of good suggestions here, but if this is the route you're going down and you are still using the same database, do everything you can to batch your inserts. In your queue consumer, aggregate as much as you can tolerate in memory, and then insert them all and commit the inserts in a single transaction. This eliminates a huge amount of MVCC overhead.