3 ms·
There are a bunch of commercial engines built on Postgres that implement sharding and parallel querying at huge scale: Amazon Redshift, Truviso, Netezza, ParAcc
by lobster_johnson 11y ago
There are a bunch of commercial engines built on Postgres that implement sharding and parallel querying at huge scale: Amazon Redshift, Truviso, Netezza, ParAccel, Aster Data and CitusDB come to mind. It's a shame that none of this has tricked down into the open-source version.
There's Postgres-X2 (formerly Postgres-XC), sponsored by NTT, which implements fully consistent multimaster replication and partitioning. After many, many years apparently still isn't production-ready nor particularly scalable, and the likelihood that it will ever be merged into mainline Postgres is zero (its planner changes alone are apparently several hundred thousand lines of code).
More interestingly, there's Postgres-XL, which is apparently a merger of Postgres-XC and a different implementation called StormDB that was bought by a company called TransLattice. (TransLattice also sponsored or acquired an earlier project that went nowhere, Postgres-R.) Unlike XC/X2, they say they aim to contribute changes back to the mainline, and they also claim their distributed query model is superior. With the commerical support behind it (it's used as the basis of a commercial product), it's possible that this is something that will be usable. Unfortunately, it seems very quiet and not very open-sourcy; most of the development seems to be by just one guy [3], and nobody seems to be using it in production at this point.
[1] https://github.com/postgres-x2/postgres-x2 https://github.com/postgres-x2/postgres-x2
[2] http://www.postgres-xl.org/ http://www.postgres-xl.org/
[3] http://git.postgresql.org/gitweb/?p=postgres-xl.git;a=summary http://git.postgresql.org/gitweb/?p=postgres-xl.git;a=summar...
- jacques_chester 11y ago> It's a shame that none of this has tricked down into the open-source version. Greenplum is an MPP database forked from the Postgres 8.x codebase. It'll be opensourced soon.
- cmrdporcupine 11y agoOh wow, I had no idea Greenplum was being open sourced, that's great news. I fiddled with it 6 years agocomparing it and Vertica for some large scale analytics work and it was pretty impressive.. I wonder how hard to merge Postgres head features back into Greenplum though, or vice versa. Probably not gonna happen?
- jacques_chester 11y agoIt's hard for me to answer you because I work for Pivotal Labs and I've heard stuff in confidence about the current thinking of what to do next. But once it's opensource I think all sorts of possibilities open up legally, technically and economically.
- dorfsmay 11y agoAre the newer PostgreSQL development been merged back in greenplum, or is it based on an older version? Is the recent Dell acquisition putting the open sourcing plans at risk?
- jacques_chester 11y agoI'm afraid I couldn't answer these questions even if I knew. Blah blah a lot of it I don't know rhubarb doctrine of confidence murmer cough professional ethics mumble mutter SEC regulations yadda yadda.
- lobster_johnson 11y agoThat's interesting. 8.x is pretty antique, though; has anything from the 9.x mainline been merged back in? I'd love to hear about a release date, too.
- jacques_chester 11y agoWe announced our opensource plans in February[1]. I don't know when Greenplum will be ready to be opened up, as I work in Labs, which is a different division. So far we have opened Apache HAWQ (incubating)[2], which is an SQL query engine for Hadoop based on the MPP query planner in Greenplum. And Apache Geode (incubating)[3], which is based on Gemfire, which is the most successful in-memory grid database you've never heard of. Edit: I suppose I should mention, entirely in my own self-interest, that we are hiring in our Big Data division across all our products. You can look at our webpage (https://pivotal.io/careers https://pivotal.io/careers) or email me (jchester@pivotal.io). [1] http://pivotal.io/big-data/press-release/pivotal-introduces-first-open-source-enterprise-grade-big-data-product-suite http://pivotal.io/big-data/press-release/pivotal-introduces-... [2] http://hawq.incubator.apache.org/ http://hawq.incubator.apache.org/ [3] http://geode.incubator.apache.org/ http://geode.incubator.apache.org/
- lobster_johnson 11y agoLooks like you got GemFire from Gemstone, which is also owned by VMware? That explains why it's written in Java, whereas I was happily surprised that HAWQ is written in C++. It's great that you're releasing stuff as open source. Perhaps you could persuade a colleague at your other division to talk about Greenplum in this thread?
- jacques_chester 11y agoWe (Pivotal) had a bunch of EMC and VMWare companies, technologies and technologies spun into us when the company was founded a few years ago. Including: Cloud Foundry, Greenplum, Spring, Gemfire and of course the namesake -- Pivotal Labs. Email me and I will connect you with our director of engineering for Big Data.