4 ms·

It is an explicit goal of the Greenplum team to merge up to the PostgreSQL mainline. It is a heroic effort to apply tens of thousands of patches, consolidate th

It is an explicit goal of the Greenplum team to merge up to the PostgreSQL mainline. It is a heroic effort to apply tens of thousands of patches, consolidate them with heavily forked and modified subsystems, while maintaining reliability and performance. The biggest hurdle was that after 8.2, the on-disk storage format changed. The traditional way you upgrade PostgreSQL is to dump the data and re-import it. This is basically a non-starter with MPP, for the simple reason that there is just too much data. Given the available engineering bandwidth, Greenplum for a long time didn't try to cross that bridge. When the decision was made to merge up to mainline, coming up with a safe format migration was the major obstacle. Disclosure: I work for Pivotal, the main contributor to Greenplum, though in a different group.

atombender 8y ago
Very interesting, thanks. I can't begin to imagine how you'd catch up a decade-old codebase patch by patch. During Greenplum's development, did the team try to avoid modifying core parts of Postgres so that keeping the codebase in sync would be easier? For example, I imagine that you wouldn't need to touch most of the storage engine (page/row management, indices etc.), but you'd have to modify the planner quite a bit.
- jacques_chester 8y ago
  I think that basically the disk format change was too far a bridge to cross. You can't tell customers that they should quadruple their cluster so that they can dump their data and reimport it to perform the upgrade. The planner is basically entirely different. It was later extracted into a standalone module to build HAWQ[0]. Since then there has been work to build a new shared query planner called GPORCA[1]. [0] https://hawq.apache.org/ https://hawq.apache.org/ [1] http://engineering.pivotal.io/post/gporca-open-source/ http://engineering.pivotal.io/post/gporca-open-source/

atombender 8y ago

Very interesting, thanks. I can't begin to imagine how you'd catch up a decade-old codebase patch by patch. During Greenplum's development, did the team try to avoid modifying core parts of Postgres so that keeping the codebase in sync would be easier? For example, I imagine that you wouldn't need to touch most of the storage engine (page/row management, indices etc.), but you'd have to modify the planner quite a bit.

jacques_chester 8y ago
I think that basically the disk format change was too far a bridge to cross. You can't tell customers that they should quadruple their cluster so that they can dump their data and reimport it to perform the upgrade. The planner is basically entirely different. It was later extracted into a standalone module to build HAWQ[0]. Since then there has been work to build a new shared query planner called GPORCA[1]. [0] https://hawq.apache.org/ https://hawq.apache.org/ [1] http://engineering.pivotal.io/post/gporca-open-source/ http://engineering.pivotal.io/post/gporca-open-source/