3 ms·
> A trillion rows of indexed data will fit on a single cloud VM and be reasonably performant. create and subsequently update index on trillion rows is very unt
by riku_iki 2y ago
> A trillion rows of indexed data will fit on a single cloud VM and be reasonably performant.
create and subsequently update index on trillion rows is very untrivial task performance wise.. Which DB you would suggest to use for this?
- winrid 2y agoYou just have to plan and do it in the background at those data sizes.
- riku_iki 2y ago> just have to plan and do it in the background this can mean building some untrivial infra, so the task is much more complicated than just using cloud vm. My point is that bottle neck and why you need cluster and not single VM is likely CPU and iops, and not data storage.
- winrid 2y ago> this can mean building some untrivial infra As someone that does it all the time on 10b+ record tables, not really? if you don't have extra resources to build the occasional index your DB is under-provisioned and you're close to falling over, cluster or not.
- riku_iki 2y ago10b+ record table and trillions indexed rows on single machine are very different tasks.
- winrid 2y agoah I forgot grandparent comment said trillion. yeah that's an order of magnitude I would distribute across a cluster if we were in active development and indexes were changing etc. if you're storing that much data you should be able to afford it :)