3 ms·

WRT exposing Postgres, you're right that you can't simply expose a database and then expect the DB to scale under random selects, inserts, etc. (RAC is another

by ianso 17y ago

WRT exposing Postgres, you're right that you can't simply expose a database and then expect the DB to scale under random selects, inserts, etc. (RAC is another matter I guess.) However, as I said in the first comment to the blog post, if you define an interface using stored procedures (a pattern familiar to many Oracle DBAs), then PL/Proxy (http://pgfoundry.org/projects/plproxy/ http://pgfoundry.org/projects/plproxy/) lets you do hash-based partitioning in a way that's more or less transparent to the end-user of a DB, again assuming that it has a defined interface. The PL/Proxy installs form a 'bus' between the DB and the user. Self-promotion: I'm currently working on hacking PL/Proxy into something that can be used to auto-scale a Postgres cluster on-demand, which has interfaces as a pre-requisite, along with some other things. The end-goal is to do exactly what you say: to scale-out Postgres like any other internet service on an ideal horizontal. Link: http://code.google.com/p/hotrepart http://code.google.com/p/hotrepart (Hum, I'm gonna get accused of spamming for repeating myself so often :-)

bayareaguy 17y ago
While Skype's PL/Proxy is a great way to make PostgreSQL more scalable, it doesn't do much to refute the basic argument that SQL databases aren't scalable since the SQL it helps you scale is limited to short RPC style operations.
- ianso 17y ago
  That's a good point, and it brings to light an unspoken assumption underlying my post, which is that the use-case is a web-based, read-heavy OLTP-style system. If you were thinking of things like OLAP and data warehousing, then I'd agree with you absolutely. However, under my re-qualified assertion :-) for large, complicated commits, the logic would either have to be at the database level for it to be in the same transaction, or a solution using temporary tables could be put together for more convoluted calls. Neither of these are elegant, I'll grant you, but the two basic approaches - longer transactions over logic closely coupled to the datastore, or staged writes - are what most DBAs on high-end databases end up doing anyway, and would probably be reproduced in some form or another in any ACID system, no? Either way you'd still have a database that scales.

by ianso 17y ago

bayareaguy 17y ago

While Skype's PL/Proxy is a great way to make PostgreSQL more scalable, it doesn't do much to refute the basic argument that SQL databases aren't scalable since the SQL it helps you scale is limited to short RPC style operations.

ianso 17y ago
That's a good point, and it brings to light an unspoken assumption underlying my post, which is that the use-case is a web-based, read-heavy OLTP-style system. If you were thinking of things like OLAP and data warehousing, then I'd agree with you absolutely. However, under my re-qualified assertion :-) for large, complicated commits, the logic would either have to be at the database level for it to be in the same transaction, or a solution using temporary tables could be put together for more convoluted calls. Neither of these are elegant, I'll grant you, but the two basic approaches - longer transactions over logic closely coupled to the datastore, or staged writes - are what most DBAs on high-end databases end up doing anyway, and would probably be reproduced in some form or another in any ACID system, no? Either way you'd still have a database that scales.