3 ms·
Postgres can handle a lot more data than that without sharding and still perform well, especially for predictable table-scan-and-filter kinds of queries (which
by Jweb_Guru 5y ago
Postgres can handle a lot more data than that without sharding and still perform well, especially for predictable table-scan-and-filter kinds of queries (which are pretty much what you want to run if you're doing the kind of denormalization the post you replied to mentions). Of course a column store would perform even better in many of those cases due to better compression / vectorization opportunities, but Postgres is really no slouch compared to other fully featured databases (research databases can blow it away though).
- winrid 5y agoSo, 20TB on a single node? Am I understanding this correctly? I guess it would be mostly for ETL stuff, as you would be swapping indexes in/out of memory too much without a lot of care.
- Jweb_Guru 5y agoDepends on the kind of index (BRIN are quite effective for large tables), how many rows we're talking about and how the column data are distributed, and how much of the data is actually hot, but yes.
- phibz 5y agoYes you understood correctly, 20 TB on a single node. And this was in 2007. The data was email and direct mail campaign and response data. Typically only the more recent data was analyzed. We retained all source data sou that reprocessing could happen when adding facts or dimensions. Analysis, scheduled or ad hoc, only ever happened against the data cubes. Their partial aggregate nature along with partitioning was part of how the performance was so good.