4 ms·
disclaimer: I work for Citus Data The bottleneck is usually not I/O, but computing aggregates over data that continuously gets updated. This is quite CPU inten
by mslot 11y ago
disclaimer: I work for Citus Data
The bottleneck is usually not I/O, but computing aggregates over data that continuously gets updated. This is quite CPU intensive even for smaller data sizes.
You might want to consider PostgreSQL, with Citus to shard tables and parallelise queries across many PostgreSQL servers. There's another big advertising platform that I helped move from MySQL to PostgreSQL+Citus recently and they're pretty happy with it. They ingest several TB of data per day and a dashboard runs group-by queries, with 99.5% of queries taking under 1 second. The data are also rolled up into daily aggregates inside the database.
There are inherent limitations to any distributed database. That's why there are so many. In Citus, not every SQL query works on distributed tables, but since every server is PostgreSQL 9.5, you do have a lot of possibilities.
Looking at your username, are you based in the Netherlands by any chance? :)
Some pointers:
- How CloudFlare uses Citus: https://blog.cloudflare.com/scaling-out-postgresql-for-cloudflare-analytics-using-citusdb/ https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...
- Overview of Citus: https://citus-conferences.s3.amazonaws.com/pgconf.ru-2016/CitusDB.pdf https://citus-conferences.s3.amazonaws.com/pgconf.ru-2016/Ci...
- Documentation: https://www.citusdata.com/documentation/citusdb-documentation/ https://www.citusdata.com/documentation/citusdb-documentatio...