3 ms·
While we're all here talking about databases I've got a question for the DBA/well versed software engineers. What database or database design strategies to be
by _kyran 7y ago
While we're all here talking about databases I've got a question for the DBA/well versed software engineers.
What database or database design strategies to be used for an analytics dashboard that can be queries real-time similar to Google Analytics?
The first real scaling issue I've come across (as a junior dev working on a side project) is a table with 38 million rows and rapidly growing.
Short of just adding indexes and more RAM to the problem, is there another way to design such a table to be effectively queried for Counts and Group Bys over a date range?
Iimagine things can't really be pre-comuted too easily due to the different filters?
- slindz 7y agoI'm not sure how much flexibility you have, but have you taken a look at Prometheus (for time series data) + Grafana (for dashboards) I'm certainly not a DB expert, but this is where a similar need led me a few months ago.
- kthejoker2 7y agoYou want an OLAP database, something like Clickhouse.
- resolaibohp 7y agoI currently run analytic queries on a page view dataset around ~600 million rows on postgres. You can start by looking into partitioning by date. Also take a look at your IO on your queries. IO was the biggest bottleneck for me. Make sure you have hardware that is not limiting it.