3 ms·
No. Not true at all. At around a billion records, you start getting into distributed/scale-out systems, and distributed indexes are performance death whether fo
by quizotic 10y ago
No. Not true at all. At around a billion records, you start getting into distributed/scale-out systems, and distributed indexes are performance death whether for OLTP or analytics. Even before you cross into distributed query processing, the random access nature of indexes levies a heavy tax. Indexes are a tiny piece of this issue. They address the problem of finding a single needle in a haystack, but if you need to aggregate lots of data, indexes are largely worthless.
What about different compression schemes for different sets of "column" values - with the idea that you execute against the compressed values, rarely ever decompressing. Even column store systems that allow you to specify a compression scheme aren't adaptive. Why the hell not? This is almost 2020.
- needusername 10y ago> At around a billion records, you start getting into distributed/scale-out systems, and distributed indexes are performance death whether for OLTP or analytics. Even before you cross into distributed query processing, the random access nature of indexes levies a heavy tax. I assume you mean once process/aggregate a billion records. Having a billion records in a single table on a single, non-distributed system is not an issue at all.
- gaius 10y agoAt around a billion records, you start getting into distributed/scale-out systems Uhh, a billion-row table you can handle on a laptop easily. Probably in SQLite.
- orf 10y ago> At around a billion records, you start getting into distributed/scale-out systems Yeahhh.... no. I remember running a 250-300,000,000 row postgres table of Wikipedia links on my 2010 Dell XPS and it functioning fine, if not a bit slow (it had a huge index on an array column). Good machine, but ancient now.