14 ms·

Pivotal Greenplum Database has been open sourced

195 points by snaga 11y ago

nickpeterson 11y ago
I don't see any information on the site about limitations for enterprise use, is this free as in beer?
- thristian 11y ago
  It says it's Apache 2 licensed, so it should be free-as-in-freedom, even for enterprise use.
- lobster_johnson 11y ago
  Apache license, so yes, free to use and modify for whatever purpose.
  - james2vegas 11y ago
    except putting the code back into mainline postgres
- younata 11y ago
  It's Apache 2.0, so free as in beer and free as in speech. https://github.com/greenplum-db/gpdb/blob/master/LICENSE https://github.com/greenplum-db/gpdb/blob/master/LICENSE
- owyn 11y ago
  It's released under the Apache2 license. So, IANAL but yes. They do have some kind of a commercial/support offering on top of it.
djokkataja 11y ago
"Greenplum Database is based on PostgreSQL 8.2 with a few features added in from the 8.3 release. To support the distributed nature and typical workload of a Greenplum Database system, some SQL commands have been added or modified, and there are a few PostgreSQL features that are not supported. Greenplum has also added features not found in PostgreSQL, such as physical data distribution, parallel query optimization, external tables, resource queues for workload management and enhanced table partitioning." http://gpdb.docs.pivotal.io/4360/ref_guide/feature_summary.html#topic8 http://gpdb.docs.pivotal.io/4360/ref_guide/feature_summary.h... Are there plans to eventually have full feature parity with ongoing PostgreSQL development (as in adding features from versions of PostgreSQL newer than 8.2), or is Greenplum going to mostly be taking its own trajectory?
- jacques_chester 11y ago
  > Are there plans to eventually have full feature parity with ongoing PostgreSQL development, or is Greenplum going to mostly be taking its own trajectory? I asked about this today while I was at work. The gist was: it will depend on a lot of factors, so it'd be unwise to nail any documents to any doors. But watch this space for further developments. In the mean time, Greenplum speaks the PostgreSQL wire protocol. All the tools that can speak to PostgreSQL can speak seamlessly to Greenplum. Disclaimer: While I work for Pivotal, I work in Pivotal Labs. I'm just an engineer and I don't have any input into product direction for our data products, which is done in another division. If pain persists, consult your doctor.
fl0wenol 11y ago
This is exciting news. Greenplum's is one of the few instances of a Postgres-based MPP that I've had good experiences with. The experiences and tuning in the query plan re-writer for MPP might be helpful in rolling in parallel operations support to the base. Not necessarily the "how" since the GP strategy won't necessarily mesh, but the "what".
jacques_chester 11y ago
Edit: There's a shiny website too -- http://greenplum.org/ http://greenplum.org/ We (Pivotal, for whom I don't speak in any official capacity) have also opensourced Apache HAWQ (incubating)[1], which is an SQL front-end for Hadoop that was extracted from Greenplum, as well as Apache Geode (incubating)[2] which was based on GemFire. This was part of a general announcement we made in February that our intention was to opensource our data products as it became possible to do so. Incidentally, we are hiring engineers across our entire data suite, including PostgreSQL specialists. The Data division solves seriously heavyweight distributed problems. You can look up the listings on http://pivotal.io/careers http://pivotal.io/careers, or if you like helping a fellow engineer collect a referral, email me: jchester@pivotal.io and I'll work out which office and division to send you to. [1] http://hawq.incubator.apache.org/ http://hawq.incubator.apache.org/ [2] http://geode.incubator.apache.org/ http://geode.incubator.apache.org/
- iheartmemcache 11y ago
  Oh man this is huge. Are you guys opening Chorus and the Pivotal HD stuff too? I'd be shorting HP stock right now, because Vertica just lost all of its appeal. Teradata and it's Hadoop H-SQL or whatever must be shaking in their boots too. Are you guys going to attempt to upstream this or is it forked to the point of no return? I'd love to work on a project like this but I've got no PGSQL experience (though I have worked on HFT both with KDB+ and writing a fairly decent functional knock-off operating in production and I bet the distribution and data set problems were similar). How is EMC to work for as a parent company?
  - jacques_chester 11y ago
    > Are you guys opening Chorus and the Pivotal HD stuff too? I'm not sure where Chorus and PHD fit into the picture -- the latter is I believe a distribution of Hadoop that we curate. > Are you guys going to attempt to upstream this or is it forked to the point of no return? Too early to say. > How is EMC to work for as a parent company? EMC is entirely hands off, in my experience. Pivotal runs itself. Best job I've ever had by a wide margin.
  - 11y ago
lobster_johnson 11y ago
Look like a worthy competitor to Elasticsearch for analytics: it has custom partitioning strategies, true parallel querying, and support for columnar table storage, and has much of Postgres' rich SQL implementation. The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving the master? The other concern is that this isn't just "horizontally partitioned Postgres". It's forked from a very old version (8.2) of Postgres, and so doesn't have things like hot standby, streaming replication, the JSON datatype, GIN indexes (does it have GiST?), arrays, etc. It looks like it's optimized for parallel workloads, not for general use. Anyone here with any experience with Greenplum who can perhaps speak about their work and the things that Greenplum is good at?
- jacques_chester 11y ago
  > The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving the master? Yes, this is what gpfdist[1] is for. Used properly, data can be bulk loaded in parallel across worker nodes without a master bottleneck. I've never used it, but our field engineers tell stories of heroism and victory that typically starred gpfdist as the magic sword. In terms of what it's good at: really big queries on really big data sets. Note in particular the integration with the Apache MADlib (incubating) machine learning toolkit[2], which was opensourced and donated to the ASF earlier this year. [1] http://gpdb.docs.pivotal.io/4330/utility_guide/admin_utilities/gpfdist.html http://gpdb.docs.pivotal.io/4330/utility_guide/admin_utiliti... [2] http://madlib.net/ http://madlib.net/
  - lobster_johnson 11y ago
    Thanks, useful. I guess I was too subtle in my comment; what I mean is, this looks like geared towards long, periodic bulk loads and not granular OLTP/webapp-type workloads where lots of clients write small transactions. We're using ElasticSearch for analytics, and being able to write to any node is really nice. We stream events in real time, and only do bulk loads when we need to change the schema or reprocess the data. But ES has its share of problems, and somethings like Greenplum would be preferable, if it can handle the write pattern.
parasubvert 11y ago
It's been a long time coming for this database niche to reach open source, which is the tech behind the big analytical prowess of countless companies (before and after Hadoop). Teradata, the first massively parallel database, was released in 1984. A brief overview on MPP databases / Greenplum here: https://dwarehouse.wordpress.com/2012/12/28/introduction-to-massively-parallel-processing-mpp-database/ https://dwarehouse.wordpress.com/2012/12/28/introduction-to-...
piggybox 11y ago
would like to see some benchmark vs Redshift, though the latter is a blackbox
- LittlePeter 11y ago
  What do you mean blackbox? It has explain analyze, it has rich query meta info, it has a web interface to query stats. You even know on what hardware it runs. Genuinely not sure what's blackbox about Redshift.
  - fidget 11y ago
    We had a cluster last week spend 7 hours in a DB health 'unknown' state. Response from AWS? Spin up a new cluster from snapshot.
    - vgt 11y ago
      I suggest you check out BigQuery. Redshift is not truly "fully managed", and it's not really HA/Durable, as your experience indicates.
  - ogrisel 11y ago
    It's a hosted service. You don't have access to the source code.
polskibus 11y ago
What's the story behind greenplum? Is it an old startup that has been bought by pivotal some time ago? I found an article about greenplum raising 20 mln $ in 2008 [1] I was wondering whether anyone used greenplum in production and with what kind of loads. With today's in-memory fad, I am also interested in whether gp model supports loading everything into memory in a MOLAP fashion. [1] http://techcrunch.com/2008/01/21/greenplum-takes-27-million-series-c/ http://techcrunch.com/2008/01/21/greenplum-takes-27-million-...
- Herald_MJ 11y ago
  Greenplum was a startup that was acquired by EMC in 2010. Pivotal is partially an EMC venture, so I guess that's how it ended up in their hands.
__david__ 11y ago
What's up with just a single commit since 2006? https://github.com/greenplum-db/gpdb/commits/master https://github.com/greenplum-db/gpdb/commits/master Why include all the really old commits, while squashing the most recent 10 years of commits into a single commit (6b0e52bead)?
- tlrobinson 11y ago
  The old source history is Postgres itself, the latest single commit is Greenplum. I wonder why it's based on Postgres 8.2 rather than a newer version?
  - petepete 11y ago
    That's probably why it was open-sourced. It'd take a huge effort to 'move' something as entrenched as Greenplum onto PostgreSQL 9.x and the fact that so many features and enhancements are missing from 8.2, it's putting off potential users.
  - jacques_chester 11y ago
    > I wonder why it's based on Postgres 8.2 rather than a newer version? That's what it was originally forked from.
- moretti 11y ago
  Reminds me of this tweet: https://twitter.com/capotej/status/647213485294616576 https://twitter.com/capotej/status/647213485294616576
tlrobinson 11y ago
This might pair nicely with Metabase, our open source BI tool: http://www.metabase.com/ http://www.metabase.com/
- kfk 11y ago
  I am looking for a new report solution for my company, but it seems you guys have a different set of users in mind. But your dashboard building solution looks great, I wish BI suites out there would be that easy
mhw 11y ago
Hmm; this might make it more attractive to investigate Chorus, the Rails-based analytics application that Pivotal also open-sourced (https://github.com/Chorus/chorus https://github.com/Chorus/chorus). It was always awkward to figure out what components you needed to build a development environment for it. It depended on having a virtual machine with a specific version of Greenplum Database running in it, but that version of the database wasn't easy to find. An open source Greenplum Database might make it easier to get started.
uberneo 11y ago
looks very similar to citus data https://www.citusdata.com/ https://www.citusdata.com/ which has also MPP architecture and based on PostgreSql
snaga 11y ago
FYI. [HACKERS] Patent warning about the Greenplum source code https://www.mail-archive.com/pgsql-hackers@postgresql.org/msg272048.html https://www.mail-archive.com/pgsql-hackers@postgresql.org/ms... You may have to check this message before diving into the code.

195 points by snaga 11y ago

nickpeterson 11y ago

I don't see any information on the site about limitations for enterprise use, is this free as in beer?

thristian 11y ago
It says it's Apache 2 licensed, so it should be free-as-in-freedom, even for enterprise use.
lobster_johnson 11y ago
Apache license, so yes, free to use and modify for whatever purpose.
- james2vegas 11y ago
  except putting the code back into mainline postgres
younata 11y ago
It's Apache 2.0, so free as in beer and free as in speech. https://github.com/greenplum-db/gpdb/blob/master/LICENSE https://github.com/greenplum-db/gpdb/blob/master/LICENSE
owyn 11y ago
It's released under the Apache2 license. So, IANAL but yes. They do have some kind of a commercial/support offering on top of it.

djokkataja 11y ago

"Greenplum Database is based on PostgreSQL 8.2 with a few features added in from the 8.3 release. To support the distributed nature and typical workload of a Greenplum Database system, some SQL commands have been added or modified, and there are a few PostgreSQL features that are not supported. Greenplum has also added features not found in PostgreSQL, such as physical data distribution, parallel query optimization, external tables, resource queues for workload management and enhanced table partitioning." http://gpdb.docs.pivotal.io/4360/ref_guide/feature_summary.html#topic8 http://gpdb.docs.pivotal.io/4360/ref_guide/feature_summary.h... Are there plans to eventually have full feature parity with ongoing PostgreSQL development (as in adding features from versions of PostgreSQL newer than 8.2), or is Greenplum going to mostly be taking its own trajectory?

jacques_chester 11y ago
> Are there plans to eventually have full feature parity with ongoing PostgreSQL development, or is Greenplum going to mostly be taking its own trajectory? I asked about this today while I was at work. The gist was: it will depend on a lot of factors, so it'd be unwise to nail any documents to any doors. But watch this space for further developments. In the mean time, Greenplum speaks the PostgreSQL wire protocol. All the tools that can speak to PostgreSQL can speak seamlessly to Greenplum. Disclaimer: While I work for Pivotal, I work in Pivotal Labs. I'm just an engineer and I don't have any input into product direction for our data products, which is done in another division. If pain persists, consult your doctor.

fl0wenol 11y ago

This is exciting news. Greenplum's is one of the few instances of a Postgres-based MPP that I've had good experiences with. The experiences and tuning in the query plan re-writer for MPP might be helpful in rolling in parallel operations support to the base. Not necessarily the "how" since the GP strategy won't necessarily mesh, but the "what".

jacques_chester 11y ago

Edit: There's a shiny website too -- http://greenplum.org/ http://greenplum.org/ We (Pivotal, for whom I don't speak in any official capacity) have also opensourced Apache HAWQ (incubating)[1], which is an SQL front-end for Hadoop that was extracted from Greenplum, as well as Apache Geode (incubating)[2] which was based on GemFire. This was part of a general announcement we made in February that our intention was to opensource our data products as it became possible to do so. Incidentally, we are hiring engineers across our entire data suite, including PostgreSQL specialists. The Data division solves seriously heavyweight distributed problems. You can look up the listings on http://pivotal.io/careers http://pivotal.io/careers, or if you like helping a fellow engineer collect a referral, email me: jchester@pivotal.io and I'll work out which office and division to send you to. [1] http://hawq.incubator.apache.org/ http://hawq.incubator.apache.org/ [2] http://geode.incubator.apache.org/ http://geode.incubator.apache.org/

iheartmemcache 11y ago
Oh man this is huge. Are you guys opening Chorus and the Pivotal HD stuff too? I'd be shorting HP stock right now, because Vertica just lost all of its appeal. Teradata and it's Hadoop H-SQL or whatever must be shaking in their boots too. Are you guys going to attempt to upstream this or is it forked to the point of no return? I'd love to work on a project like this but I've got no PGSQL experience (though I have worked on HFT both with KDB+ and writing a fairly decent functional knock-off operating in production and I bet the distribution and data set problems were similar). How is EMC to work for as a parent company?
- jacques_chester 11y ago
  > Are you guys opening Chorus and the Pivotal HD stuff too? I'm not sure where Chorus and PHD fit into the picture -- the latter is I believe a distribution of Hadoop that we curate. > Are you guys going to attempt to upstream this or is it forked to the point of no return? Too early to say. > How is EMC to work for as a parent company? EMC is entirely hands off, in my experience. Pivotal runs itself. Best job I've ever had by a wide margin.
- 11y ago

lobster_johnson 11y ago

Look like a worthy competitor to Elasticsearch for analytics: it has custom partitioning strategies, true parallel querying, and support for columnar table storage, and has much of Postgres' rich SQL implementation. The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving the master? The other concern is that this isn't just "horizontally partitioned Postgres". It's forked from a very old version (8.2) of Postgres, and so doesn't have things like hot standby, streaming replication, the JSON datatype, GIN indexes (does it have GiST?), arrays, etc. It looks like it's optimized for parallel workloads, not for general use. Anyone here with any experience with Greenplum who can perhaps speak about their work and the things that Greenplum is good at?

jacques_chester 11y ago
> The parallel loading looks good, but I'm concerned that the single master means it's a bottleneck for writes -- is this the case, or is there a way to distribute writes across segments without involving the master? Yes, this is what gpfdist[1] is for. Used properly, data can be bulk loaded in parallel across worker nodes without a master bottleneck. I've never used it, but our field engineers tell stories of heroism and victory that typically starred gpfdist as the magic sword. In terms of what it's good at: really big queries on really big data sets. Note in particular the integration with the Apache MADlib (incubating) machine learning toolkit[2], which was opensourced and donated to the ASF earlier this year. [1] http://gpdb.docs.pivotal.io/4330/utility_guide/admin_utilities/gpfdist.html http://gpdb.docs.pivotal.io/4330/utility_guide/admin_utiliti... [2] http://madlib.net/ http://madlib.net/
- lobster_johnson 11y ago
  Thanks, useful. I guess I was too subtle in my comment; what I mean is, this looks like geared towards long, periodic bulk loads and not granular OLTP/webapp-type workloads where lots of clients write small transactions. We're using ElasticSearch for analytics, and being able to write to any node is really nice. We stream events in real time, and only do bulk loads when we need to change the schema or reprocess the data. But ES has its share of problems, and somethings like Greenplum would be preferable, if it can handle the write pattern.

parasubvert 11y ago

It's been a long time coming for this database niche to reach open source, which is the tech behind the big analytical prowess of countless companies (before and after Hadoop). Teradata, the first massively parallel database, was released in 1984. A brief overview on MPP databases / Greenplum here: https://dwarehouse.wordpress.com/2012/12/28/introduction-to-massively-parallel-processing-mpp-database/ https://dwarehouse.wordpress.com/2012/12/28/introduction-to-...

piggybox 11y ago

would like to see some benchmark vs Redshift, though the latter is a blackbox

LittlePeter 11y ago
What do you mean blackbox? It has explain analyze, it has rich query meta info, it has a web interface to query stats. You even know on what hardware it runs. Genuinely not sure what's blackbox about Redshift.
- fidget 11y ago
  We had a cluster last week spend 7 hours in a DB health 'unknown' state. Response from AWS? Spin up a new cluster from snapshot.
  - vgt 11y ago
    I suggest you check out BigQuery. Redshift is not truly "fully managed", and it's not really HA/Durable, as your experience indicates.
- ogrisel 11y ago
  It's a hosted service. You don't have access to the source code.

polskibus 11y ago

What's the story behind greenplum? Is it an old startup that has been bought by pivotal some time ago? I found an article about greenplum raising 20 mln $ in 2008 [1] I was wondering whether anyone used greenplum in production and with what kind of loads. With today's in-memory fad, I am also interested in whether gp model supports loading everything into memory in a MOLAP fashion. [1] http://techcrunch.com/2008/01/21/greenplum-takes-27-million-series-c/ http://techcrunch.com/2008/01/21/greenplum-takes-27-million-...

Herald_MJ 11y ago
Greenplum was a startup that was acquired by EMC in 2010. Pivotal is partially an EMC venture, so I guess that's how it ended up in their hands.

__david__ 11y ago

What's up with just a single commit since 2006? https://github.com/greenplum-db/gpdb/commits/master https://github.com/greenplum-db/gpdb/commits/master Why include all the really old commits, while squashing the most recent 10 years of commits into a single commit (6b0e52bead)?

tlrobinson 11y ago
The old source history is Postgres itself, the latest single commit is Greenplum. I wonder why it's based on Postgres 8.2 rather than a newer version?
- petepete 11y ago
  That's probably why it was open-sourced. It'd take a huge effort to 'move' something as entrenched as Greenplum onto PostgreSQL 9.x and the fact that so many features and enhancements are missing from 8.2, it's putting off potential users.
- jacques_chester 11y ago
  > I wonder why it's based on Postgres 8.2 rather than a newer version? That's what it was originally forked from.
moretti 11y ago
Reminds me of this tweet: https://twitter.com/capotej/status/647213485294616576 https://twitter.com/capotej/status/647213485294616576

tlrobinson 11y ago

This might pair nicely with Metabase, our open source BI tool: http://www.metabase.com/ http://www.metabase.com/

kfk 11y ago
I am looking for a new report solution for my company, but it seems you guys have a different set of users in mind. But your dashboard building solution looks great, I wish BI suites out there would be that easy

mhw 11y ago

Hmm; this might make it more attractive to investigate Chorus, the Rails-based analytics application that Pivotal also open-sourced (https://github.com/Chorus/chorus https://github.com/Chorus/chorus). It was always awkward to figure out what components you needed to build a development environment for it. It depended on having a virtual machine with a specific version of Greenplum Database running in it, but that version of the database wasn't easy to find. An open source Greenplum Database might make it easier to get started.

uberneo 11y ago

looks very similar to citus data https://www.citusdata.com/ https://www.citusdata.com/ which has also MPP architecture and based on PostgreSql

snaga 11y ago

FYI. [HACKERS] Patent warning about the Greenplum source code https://www.mail-archive.com/pgsql-hackers@postgresql.org/msg272048.html https://www.mail-archive.com/pgsql-hackers@postgresql.org/ms... You may have to check this message before diving into the code.