4 ms·
SQL in my experience is not designed for workloads where you want to compute together/return a ton of values from a ton of rows at once. Its good where you need
by sparkling 7y ago
SQL in my experience is not designed for workloads where you want to compute together/return a ton of values from a ton of rows at once. Its good where you need to cherry-pick just a few rows belonging to one user from a databses of millions of row (e.g. CRUD operations).
tl;dr:
simple CRUD operations on few rows scale fine
data-science/analytics stuff involving millions of in one query scale rather badly
- felixge 7y agoSQL isn't the problem IMO. The problem is that the implementation we're using (PostgreSQL) is a row-store which is indeed more optimal for the kind of operations you mention. However, SQL isn't limited to row-stores. There are column-store implementations that are quite amazing for aggregate queries, e.g. Clickhouse. Using one of those would very likely work for us, but my understanding is that loading data into them in real-time is problematic.
- kwillets 7y agoHow close is real time? Seconds? Minutes? Also, what type of aggregate?