4 ms·
This is very interesting, but not as a CPU saving optimization. I'm sure it does that very well, but that's not why it's Important. Rather, this seems to me l
by adamtj 12y ago
This is very interesting, but not as a CPU saving optimization. I'm sure it does that very well, but that's not why it's Important. Rather, this seems to me like the next step toward the inevitable future of PostgreSQL as arbitrarily scalable, and as _the_ general query engine that ties together whatever physical data stores you happen to use.
It seems obvious to me that pushing the Foreign Data Wrapper layer with work like this is how we eventually break through the RDBMS scalability barrier of the individual host. In the future, I'm sure you'll see similar work where _the_ GPU won't be the GPU and the PCI bus won't be the PCI bus. Rather, they'll be _a_ host and the network. A database service (database cluster in Posgres's nomenclature) will eventually run not just on a single machine, but on a single cluster of machines. Instead of a cluster of machines for redundancy, you'll have a cluster of clusters.
Postgres is really two things in one: a physical layer of bytes in pages in files, and a logical layer of queries on tables of records. The most important piece in the future will be the logical layer. The FDW layer will naturally be extended and generalized until it is fully as powerful as the current physical layer. At that point, it can be made THE api through which the logical layer accesses data. The current physical layer will then be nothing more than the default implementation of that general API.
At that point, we can move whole or partial tables to other hosts. Perhaps the autovacuum daemon will gain a sibling in the autosharding daemon. The query optimizer will need to care not just about disk IO, but network IO and will need to start considering the non-uniform performance characteristics of different tables. Some tables will be driven by Postgres's default physical storage engine. Others will be driven by other RDBMSs, or by NoSQL key/value or document stores, or other data stores. They may be on the same machine or a different one.
Postgres will transform into a query engine on top of whatever data stores best fit your workload. I expect the query engine will learn about columnar stores and be able to mix those in a single query with the row stores, key/value stores and document stores that it already understands. PostgreSQL will be a central point through which you can aggregate, analyze, and manipulate any and all of your data. It needn't be intrusive or disruptive: you can still use a normal redis client for your redis store, but you can also use Postgres to manipulate that data with SQL and to combine it with other tables, whole RDBMSs, other NoSQL stores, spreadsheets, web services, or anything else. Maybe it will even make things like Map/Reduce frameworks redundant.
I don't typically follow Postgres's internal discussions, so maybe this is already being discussed and planned. Or, maybe it's so obvious that nobody even needs to talk about it. Or, perhaps I'm just some wide-eyed idealist who doesn't understand the fundamental problems preventing such a thing from ever being practical.