5 ms·

this is a really good point. there's a really uncomfortable space where your production database is just a wee bit bigger than you can comfortably work on with

this is a really good point. there's a really uncomfortable space where your production database is just a wee bit bigger than you can comfortably work on with a single machine, and there is a case to be made for using some level of parallelism, but you don't yet need to go full monty on it with a compute cluster and a distributed datastore. so you start looking at stuff like sharding or vertical scaling and you keep on doing things more or less the way you have been but with steadily degrading performance on every new insert.

justinsaccount 9y ago
Exactly.. I used to say that I didn't have "big data", but I have "annoying data". clickhouse turns my annoying data back into something that I can query in 30 seconds. I just ran a random query to find what day had the most connections: select day,count() as c from conn group by day order by c desc limit 1 And that took all of: 1 rows in set. Elapsed: 16.412 sec. Processed 1.43 billion rows, 2.87 GB (87.33 million rows/s., 174.65 MB/s.)