4 ms·

Redshift isn't even developed by Amazon — it's a commercial product called ParAccel, which they license (and modify, presumably). Another commercial MPP databa

Redshift isn't even developed by Amazon — it's a commercial product called ParAccel, which they license (and modify, presumably). Another commercial MPP database based on Postgres 8.x, GreenplumDB, was open-sourced a few years back. The changes are so extensive that there's little hope of catching up with the current Postgres codebase. Given the focus on OLAP and analytics over OLTP, there might not even be a strong motivation to catch up, either.

anarazel 8y ago
Worthwhile to note that Greeplum is being moved forward. While it initially seemed, from the outside, fairly slow-going, it seems that later versions were done more quickly. Apparently largely because of fixing some technical debt. They're catching up to 9.4 in their development branch, I believe. For years they were on 8.2...
manigandham 8y ago
The original codebase came from ParAccel which itself was later acquired by Actian, but Redshift is definitely owned and developed by Amazon. But yes, the changes are not viable to upstream without so many modifications that fundamentally change the database. Pluggable storage in mainline would be a good first step.
- atombender 8y ago
  Looks like Amazon actually bought the source code and forked it. This Quora thread has some replies by people from Actian: https://www.quora.com/Amazon-redshift-uses-Actians-ParaAccel-MPP-database-so-what-does-Amazon-redshift-team-work-on-Only-integrating-ParaAccel-in-amazon-ecosystem-Do-some-people-from-Redshift-team-work-on-ParaAccel-database-as-well https://www.quora.com/Amazon-redshift-uses-Actians-ParaAccel....
  - scapecast 8y ago
    I'm the guy who wrote the reply that pops up first when you read that Quora thread. Here's the short story (and I know all of this because the guy who invented the core engine for ParAccel's MPP columnar tech, that is the foundation for Redshift, is one of our early advisors). - ParAccel developed the tech for a columnar storage database. I believe it was called "matrix" - Amazon AWS bought the source code from ParAccel, limited for use as a cloud service, i.e. they couldn't create another on-premise version that would compete with ParAccel - ParAccel then sold to Actian, and a few years ago Actian shelved the product as clearly the on-premise world had lost to cloud warehouses. The reason AWS bought the source code was time-to-market. It would have taken too long to build a product from scratch, and customers were asking for a cloud warehouse. Back then, ParAccel had by far the best and fastest MPP / columnar tech, plus it's very attractive since it's based on Postgres. So Actian and Amazon AWS essentially had the same tech, just different distribution models. One is on-premise (Actian), the other one a managed cloud service (AWS). We all know who won. there's very interesting paper by the Amazon RDS team (where Redshift rolls up). It's not only about "faster, better, cheaper" - it really is about simplicity and that's what Redshift delivered on. https://event.cwi.nl/lsde/papers/p1917-gupta.pdf https://event.cwi.nl/lsde/papers/p1917-gupta.pdf Spin up a cluster in less than 5 minutes and get results within 15 min. Keep in mind, this was all in late 2012, so what appears "normal" today was pure magic back then. but ever since the "fork", i.e. when AWS purchased a snapshot in time of the code base, the products have obviously diverged. There's some 8 years of development now in Amazon Redshift.
    - throwaway12iii 8y ago
      In 2012 neither a column store or spinning up a cluster was magic or state of the art. Redshift delivered on sales and marketing. Amazon made their fortune on the backs of open source contributors many times, and this is just another one of those times.
gfody 8y ago
ParAccel also came from Postgres, https://wiki.postgresql.org/wiki/PostgreSQL_derived_databases https://wiki.postgresql.org/wiki/PostgreSQL_derived_database...
- atombender 8y ago
  Yes, we already covered that. :-)
jacques_chester 8y ago
It is an explicit goal of the Greenplum team to merge up to the PostgreSQL mainline. It is a heroic effort to apply tens of thousands of patches, consolidate them with heavily forked and modified subsystems, while maintaining reliability and performance. The biggest hurdle was that after 8.2, the on-disk storage format changed. The traditional way you upgrade PostgreSQL is to dump the data and re-import it. This is basically a non-starter with MPP, for the simple reason that there is just too much data. Given the available engineering bandwidth, Greenplum for a long time didn't try to cross that bridge. When the decision was made to merge up to mainline, coming up with a safe format migration was the major obstacle. Disclosure: I work for Pivotal, the main contributor to Greenplum, though in a different group.
- atombender 8y ago
  Very interesting, thanks. I can't begin to imagine how you'd catch up a decade-old codebase patch by patch. During Greenplum's development, did the team try to avoid modifying core parts of Postgres so that keeping the codebase in sync would be easier? For example, I imagine that you wouldn't need to touch most of the storage engine (page/row management, indices etc.), but you'd have to modify the planner quite a bit.
  - jacques_chester 8y ago
    I think that basically the disk format change was too far a bridge to cross. You can't tell customers that they should quadruple their cluster so that they can dump their data and reimport it to perform the upgrade. The planner is basically entirely different. It was later extracted into a standalone module to build HAWQ[0]. Since then there has been work to build a new shared query planner called GPORCA[1]. [0] https://hawq.apache.org/ https://hawq.apache.org/ [1] http://engineering.pivotal.io/post/gporca-open-source/ http://engineering.pivotal.io/post/gporca-open-source/

anarazel 8y ago

Worthwhile to note that Greeplum is being moved forward. While it initially seemed, from the outside, fairly slow-going, it seems that later versions were done more quickly. Apparently largely because of fixing some technical debt. They're catching up to 9.4 in their development branch, I believe. For years they were on 8.2...

manigandham 8y ago

The original codebase came from ParAccel which itself was later acquired by Actian, but Redshift is definitely owned and developed by Amazon. But yes, the changes are not viable to upstream without so many modifications that fundamentally change the database. Pluggable storage in mainline would be a good first step.

atombender 8y ago
Looks like Amazon actually bought the source code and forked it. This Quora thread has some replies by people from Actian: https://www.quora.com/Amazon-redshift-uses-Actians-ParaAccel-MPP-database-so-what-does-Amazon-redshift-team-work-on-Only-integrating-ParaAccel-in-amazon-ecosystem-Do-some-people-from-Redshift-team-work-on-ParaAccel-database-as-well https://www.quora.com/Amazon-redshift-uses-Actians-ParaAccel....
- scapecast 8y ago
  I'm the guy who wrote the reply that pops up first when you read that Quora thread. Here's the short story (and I know all of this because the guy who invented the core engine for ParAccel's MPP columnar tech, that is the foundation for Redshift, is one of our early advisors). - ParAccel developed the tech for a columnar storage database. I believe it was called "matrix" - Amazon AWS bought the source code from ParAccel, limited for use as a cloud service, i.e. they couldn't create another on-premise version that would compete with ParAccel - ParAccel then sold to Actian, and a few years ago Actian shelved the product as clearly the on-premise world had lost to cloud warehouses. The reason AWS bought the source code was time-to-market. It would have taken too long to build a product from scratch, and customers were asking for a cloud warehouse. Back then, ParAccel had by far the best and fastest MPP / columnar tech, plus it's very attractive since it's based on Postgres. So Actian and Amazon AWS essentially had the same tech, just different distribution models. One is on-premise (Actian), the other one a managed cloud service (AWS). We all know who won. there's very interesting paper by the Amazon RDS team (where Redshift rolls up). It's not only about "faster, better, cheaper" - it really is about simplicity and that's what Redshift delivered on. https://event.cwi.nl/lsde/papers/p1917-gupta.pdf https://event.cwi.nl/lsde/papers/p1917-gupta.pdf Spin up a cluster in less than 5 minutes and get results within 15 min. Keep in mind, this was all in late 2012, so what appears "normal" today was pure magic back then. but ever since the "fork", i.e. when AWS purchased a snapshot in time of the code base, the products have obviously diverged. There's some 8 years of development now in Amazon Redshift.
  - throwaway12iii 8y ago
    In 2012 neither a column store or spinning up a cluster was magic or state of the art. Redshift delivered on sales and marketing. Amazon made their fortune on the backs of open source contributors many times, and this is just another one of those times.

gfody 8y ago

ParAccel also came from Postgres, https://wiki.postgresql.org/wiki/PostgreSQL_derived_databases https://wiki.postgresql.org/wiki/PostgreSQL_derived_database...

atombender 8y ago
Yes, we already covered that. :-)

jacques_chester 8y ago

It is an explicit goal of the Greenplum team to merge up to the PostgreSQL mainline. It is a heroic effort to apply tens of thousands of patches, consolidate them with heavily forked and modified subsystems, while maintaining reliability and performance. The biggest hurdle was that after 8.2, the on-disk storage format changed. The traditional way you upgrade PostgreSQL is to dump the data and re-import it. This is basically a non-starter with MPP, for the simple reason that there is just too much data. Given the available engineering bandwidth, Greenplum for a long time didn't try to cross that bridge. When the decision was made to merge up to mainline, coming up with a safe format migration was the major obstacle. Disclosure: I work for Pivotal, the main contributor to Greenplum, though in a different group.

atombender 8y ago
Very interesting, thanks. I can't begin to imagine how you'd catch up a decade-old codebase patch by patch. During Greenplum's development, did the team try to avoid modifying core parts of Postgres so that keeping the codebase in sync would be easier? For example, I imagine that you wouldn't need to touch most of the storage engine (page/row management, indices etc.), but you'd have to modify the planner quite a bit.
- jacques_chester 8y ago
  I think that basically the disk format change was too far a bridge to cross. You can't tell customers that they should quadruple their cluster so that they can dump their data and reimport it to perform the upgrade. The planner is basically entirely different. It was later extracted into a standalone module to build HAWQ[0]. Since then there has been work to build a new shared query planner called GPORCA[1]. [0] https://hawq.apache.org/ https://hawq.apache.org/ [1] http://engineering.pivotal.io/post/gporca-open-source/ http://engineering.pivotal.io/post/gporca-open-source/