3 ms·
The common solution is to, you know, not fill your database. Old data should be archived whenever possible. If you really need to keep around more data than c
by yellowapple 8y ago
The common solution is to, you know, not fill your database. Old data should be archived whenever possible.
If you really need to keep around more data than can fit in a single server, the PostgreSQL wiki has some documentation to that effect [1], with PostgreSQL-XL [2] and Citus [3] being specific examples of systems which can shard a Postgres database across multiple nodes.
A more DIY alternative to the above is to leverage PostgreSQL's support for foreign data wrappers [4] to include results from other servers. The cool thing about this approach is that the other servers don't even have to be PostgreSQL servers; hell, they don't even have to be SQL, nor do they even have to be servers.
The great thing about most/all of these solutions is that you don't have to worry about them upfront; you can roll them out when you're closer to actually hitting practical/theoretical limits. No need to prematurely optimize :)
[1]: https://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling https://wiki.postgresql.org/wiki/Replication,_Clustering,_an...
[2]: https://www.postgres-xl.org/overview/ https://www.postgres-xl.org/overview/
[3]: https://www.citusdata.com/product/community#scale-out https://www.citusdata.com/product/community#scale-out
[4]: https://wiki.postgresql.org/wiki/Foreign_data_wrappers https://wiki.postgresql.org/wiki/Foreign_data_wrappers
- F_J_H 8y agoGood point re: PostgresSQL foreign data wrappers (aka "linked tables"). I've used this with data in an Oracle database, and it worked well.
- riku_iki 8y agoAll those solutions have much more complexity than adapting NoSQL/NewSQL solution from the beginning, which supposed to provide much easier scale out story. Also FDW is probably limited in terms of transaction support and push down predicates.
- yellowapple 8y agoRight, but even with the added complexity (not that much for something like Postgres-XL), you still end up with all the benefits of a robust and performant relational database. Meanwhile, for the FDW-based DIY approach, it should be at least theoretically (if not practically) possible to support both transactions and predicate pushdowns, especially if the foreign database is another Postgres database (or SQL at all, for that matter). More to the point, though: the idea is that you shouldn't have to prematurely optimize. The vast majority of websites don't need a full-blown distributed database. In the rare event that your site grows to that point, it ain't really all that difficult to switch to Postgres-XL or CockroachDB or something that gets you the best of both worlds.