4 ms·
"Big" is apparently still limited by the 32TB table size limit in PostegreSQL. My last job was running into this. Sadly we didn't get rigorous metrics, but the
by brianolson 11y ago
"Big" is apparently still limited by the 32TB table size limit in PostegreSQL.
My last job was running into this. Sadly we didn't get rigorous metrics, but the database 'felt' slower when it got over .5 TB. The process pumping data into the database was several percent slower in items per second than when it started.
Still, PostgreSQL is great and I'm using it right now on the new project and I hope that someday I am blessed with tables too big for it.
- Cieplak 11y agoInsert performance degrades as indexes grow large. Indexing in batch is faster than indexing with each insert. http://stackoverflow.com/questions/12206600/how-to-speed-up-insertion-performance-in-postgresql http://stackoverflow.com/questions/12206600/how-to-speed-up-... Depending on the nature of your project, you might consider a column store engine for postgres, such as https://github.com/citusdata/cstore_fdw https://github.com/citusdata/cstore_fdw
- nabla9 11y agoIf 32TB table size limit and degrading performance worries you, I suggest that you delve into Postgress documentation and discover table partitioning. (you can partition table into chunks that are together larger than 32TB)
- collyw 11y agoThat's orders of magnitude bigger than what most people who claim to be working with "big data" need.