5 ms·

The elephant in the room is that AWS Babelfish could be a viable alternative to PostgreSQL. Babelfish is currently a fork of PostgreSQL that is compatible with

The elephant in the room is that AWS Babelfish could be a viable alternative to PostgreSQL. Babelfish is currently a fork of PostgreSQL that is compatible with Microsoft SQL Server’s TDS wire protocol and T-SQL language. The article proposes adding hooks for wire protocol extensions eliminating the need for separate forks. Adding new languages is already supported (EDIT: incorrect, see mattashii's comment).

mattashii 6y ago
> Adding new languages is already supported. This is true only to a degree: Adding new procedural languages is supported, but support for new query dialects (i.e. the main contact surface of most applications with an SQL RDBMS) is not currently a supported feature. As mentioned in the psql-hackers thread from the article, currently the postgresql-specific parsed syntax tree is passed through many components, and to replace the syntax you'd have to alter and recompile the core application.
ahachete 6y ago
OP. One of my fears. That unless we, as the PostgreSQL Community, do something about it, and embrace the efforts that AWS started doing and I hope will continue, it would effectively end up being a fork. And a fork with a collaboration on GitHub, which many would prefer to contribute to rather than current's Postgres contribution mechanism. And if that's the case, I wonder if it could become the MariaDB, or the LibreOffice, of Postgres. I hope we will be able to do the best thing and have the contribution that Postgres is receiving integrated and merged into Postgres.
- _msw_ 6y ago
  Disclosure: I work for Amazon where I build AWS infrastructure, but these opinions are mine alone. Personally, I think that the word "fork" has a lot of historical meaning in the world of Free and Open Source Software. To me, "fork" means that there is a "vote of no confidence" in the direction of the people who are making decisions about open source software. That clearly is not the case here. The PostgreSQL core team has a very good reputation of being collaborative, and thoughtful, in their work to make PostgreSQL the best database it can be for the community at large, and not build it to benefit any particular commercial interest. Often software development happens on branches. These are experiments, not forks. Over time, good ideas are propagated, and grafted, between the branches. These are not "forks" to me.
  - ahachete 6y ago
    Indeed. I see this as a very open and collaborative approach. To announce such a big contribution to Postgres, and be it open sourced when published, is very welcome, and I believe one of the boosts, as I mentioned in the post, that Postgres needs. I'm also convinced that Babelfish and PostgreSQL will be properly integrated, and this will be a great win. I consider it too a development branch more than a fork, which certainly has "no confidence" connotations. But there's still some worry that it would become one, one day, if there's no sufficient effort towards its integration --as the conversation that I wanted to spark here was almost non-existent up until recently. This actually brings the topic of more explicitly branch development in Postgres, but it's a different story ;)
    - petergeoghegan 6y ago
      > To announce such a big contribution to Postgres, and be it open sourced when published, is very welcome But the source code hasn't been published. As I said to you today on pgsq-hackers, that's why it hasn't been taken seriously - that's just how it works. > But there's still some worry that it would become one, one day Who are you attributing that concern to, exactly? You're basically claiming that it will be a great opportunity. But you're also claiming that it might be an existential crisis. Do I have that right? Have you considered the possibility that it might actually be neither of those two things?
      - ahachete 6y ago
        > that's just how it works. And maybe that's just how it shouldn't. Technology moves fast, very fast. Markets do so. Users, you get it. Competing technologies, undeniably. To delay a discussion for months just because the source is not published is in my opinion only reflecting on part of the opportunity/problem. I'm calling for an strategic discussion, which is higher level than a code-only, limited, partial analysis. > Have you considered the possibility that it might actually be neither of those two things? Sure thing, like anything in life, it may be neither. But what I know for sure is that a) we as the PostgreSQL Community need to mitigate risks and plan for them; b) leverage (or transform into!) opportunities as soon as we identify them.
- linuxhiker 6y ago
  This indeed is the concern. I don't think most people are going to care if it is PostgreSQL.Org PostgreSQL, or Babelfish for PostgreSQL in the long run. What they will care about is modernity of the development process and who has the "cool" features. When you look at something that has nothing to do with the Babelfish functionality but would absolutely help PostgreSQL, consider PHOT. PHOT (Partial HOT) is a great opportunity for PostgreSQL that Amazon is trying to give back. However, I wouldn't be surprised if the .Org community were to not accept the patch, the feature would end up in Babelfish. If that happens, we now have a viable fork which has 100% compatibility++.
  - aargh_aargh 6y ago
    This is the only reference I found for the exact query "partial heap only tuple". It's hard to find anything using "partial HOT". https://www.talkend.net/post/268380.html https://www.talkend.net/post/268380.html
    - Tostino 6y ago
      here is a link to the thread on the mailing list: https://www.postgresql-archive.org/partial-heap-only-tuples-td6178310.html https://www.postgresql-archive.org/partial-heap-only-tuples-...
- random5634 6y ago
  Agreed - a concern for me as well. AWS has enough scale to have its own gravity and will likely run development forward regardless of .org, it would be GREAT if .org could actively work to integrate where possible the open source options here. I remember some early efforts of google to contribute wakelocks and other features to the linux kernel - so annoying when that was shot down and created a pretty significant split, and the commercial entity wasn't going to hold off on wakelocks because a subsystem maintainer didn't like it - so android starting going off into its own land. Wakelocks also ended up proving a reasonable power management approach (despite complaints they didn't use existing linux mechanisms etc). There was a big push ultimately to get wakelocks into mainline and drive some convergence.
_msw_ 6y ago
NB: it is not "AWS Babelfish". It is (currently) "Babelfish for PostgreSQL". https://babelfish-for-postgresql.github.io/babelfish-for-postgresql/ https://babelfish-for-postgresql.github.io/babelfish-for-pos... Can the title of this post be changed to remove "AWS" please?
- dang 6y ago
  The opening paragraph of the article says " Amazon AWS announced Babelfish" and links to https://aws.amazon.com/blogs/opensource/want-more-postgresql-you-just-might-like-babelfish/ https://aws.amazon.com/blogs/opensource/want-more-postgresql... from the AWS blog. This all seems accurate to me. Is there some other reason why we should take AWS out of the title? Branding is not a reason that HN readers would support, and I think if we took AWS out for that reason, the community would accuse us of trying to obscure the origins of the project. Internet people are trigger-happy about this kind of thing, and too quick to accuse us of serving $BigCo agenda as it is.
  - _msw_ 6y ago
    I think we are trying to establish the brand of this project in these early days, and to have that brand distinct from AWS. I see this like "Mozilla" was separate from "Netscape" in their early days [1] (note, this link will not work when the referrer is from HN. Archive link provided for convenience [2]). It is a project that AWS is sponsoring and stewarding, but from what I currently know, I think the intent is for it to be community-centric in the long term. If successful, Babelfish will grow a community, and work harmoniously as an extension of PostgreSQL. And including "AWS" in the brand of the project may hamper the efforts to build a community around it (in addition to complications of using a brand / trademark like "AWS" in an open-source project). The TL;DR is: "AWS" is not part of the branding of Babelfish for PostgreSQL, and that is intentional. The reasons for this aren't about obscuring the origins or initial sponsorship of the work, but are rather about establishing a separate identity. [1] https://www.jwz.org/blog/2016/10/they-live-and-the-secret-history-of-the-mozilla-logo/#post-13248461:~:text=With%20the%20launch%20of%20Mozilla.org%2C%20I,to%20some%20degree%20from%20Netscape%20itself https://www.jwz.org/blog/2016/10/they-live-and-the-secret-hi... [2] http://archive.is/UMMCx http://archive.is/UMMCx

mattashii 6y ago

> Adding new languages is already supported. This is true only to a degree: Adding new procedural languages is supported, but support for new query dialects (i.e. the main contact surface of most applications with an SQL RDBMS) is not currently a supported feature. As mentioned in the psql-hackers thread from the article, currently the postgresql-specific parsed syntax tree is passed through many components, and to replace the syntax you'd have to alter and recompile the core application.

ahachete 6y ago

OP. One of my fears. That unless we, as the PostgreSQL Community, do something about it, and embrace the efforts that AWS started doing and I hope will continue, it would effectively end up being a fork. And a fork with a collaboration on GitHub, which many would prefer to contribute to rather than current's Postgres contribution mechanism. And if that's the case, I wonder if it could become the MariaDB, or the LibreOffice, of Postgres. I hope we will be able to do the best thing and have the contribution that Postgres is receiving integrated and merged into Postgres.

_msw_ 6y ago
Disclosure: I work for Amazon where I build AWS infrastructure, but these opinions are mine alone. Personally, I think that the word "fork" has a lot of historical meaning in the world of Free and Open Source Software. To me, "fork" means that there is a "vote of no confidence" in the direction of the people who are making decisions about open source software. That clearly is not the case here. The PostgreSQL core team has a very good reputation of being collaborative, and thoughtful, in their work to make PostgreSQL the best database it can be for the community at large, and not build it to benefit any particular commercial interest. Often software development happens on branches. These are experiments, not forks. Over time, good ideas are propagated, and grafted, between the branches. These are not "forks" to me.
- ahachete 6y ago
  Indeed. I see this as a very open and collaborative approach. To announce such a big contribution to Postgres, and be it open sourced when published, is very welcome, and I believe one of the boosts, as I mentioned in the post, that Postgres needs. I'm also convinced that Babelfish and PostgreSQL will be properly integrated, and this will be a great win. I consider it too a development branch more than a fork, which certainly has "no confidence" connotations. But there's still some worry that it would become one, one day, if there's no sufficient effort towards its integration --as the conversation that I wanted to spark here was almost non-existent up until recently. This actually brings the topic of more explicitly branch development in Postgres, but it's a different story ;)
  - petergeoghegan 6y ago
    > To announce such a big contribution to Postgres, and be it open sourced when published, is very welcome But the source code hasn't been published. As I said to you today on pgsq-hackers, that's why it hasn't been taken seriously - that's just how it works. > But there's still some worry that it would become one, one day Who are you attributing that concern to, exactly? You're basically claiming that it will be a great opportunity. But you're also claiming that it might be an existential crisis. Do I have that right? Have you considered the possibility that it might actually be neither of those two things?
    - ahachete 6y ago
      > that's just how it works. And maybe that's just how it shouldn't. Technology moves fast, very fast. Markets do so. Users, you get it. Competing technologies, undeniably. To delay a discussion for months just because the source is not published is in my opinion only reflecting on part of the opportunity/problem. I'm calling for an strategic discussion, which is higher level than a code-only, limited, partial analysis. > Have you considered the possibility that it might actually be neither of those two things? Sure thing, like anything in life, it may be neither. But what I know for sure is that a) we as the PostgreSQL Community need to mitigate risks and plan for them; b) leverage (or transform into!) opportunities as soon as we identify them.
linuxhiker 6y ago
This indeed is the concern. I don't think most people are going to care if it is PostgreSQL.Org PostgreSQL, or Babelfish for PostgreSQL in the long run. What they will care about is modernity of the development process and who has the "cool" features. When you look at something that has nothing to do with the Babelfish functionality but would absolutely help PostgreSQL, consider PHOT. PHOT (Partial HOT) is a great opportunity for PostgreSQL that Amazon is trying to give back. However, I wouldn't be surprised if the .Org community were to not accept the patch, the feature would end up in Babelfish. If that happens, we now have a viable fork which has 100% compatibility++.
- aargh_aargh 6y ago
  This is the only reference I found for the exact query "partial heap only tuple". It's hard to find anything using "partial HOT". https://www.talkend.net/post/268380.html https://www.talkend.net/post/268380.html
  - Tostino 6y ago
    here is a link to the thread on the mailing list: https://www.postgresql-archive.org/partial-heap-only-tuples-td6178310.html https://www.postgresql-archive.org/partial-heap-only-tuples-...
random5634 6y ago
Agreed - a concern for me as well. AWS has enough scale to have its own gravity and will likely run development forward regardless of .org, it would be GREAT if .org could actively work to integrate where possible the open source options here. I remember some early efforts of google to contribute wakelocks and other features to the linux kernel - so annoying when that was shot down and created a pretty significant split, and the commercial entity wasn't going to hold off on wakelocks because a subsystem maintainer didn't like it - so android starting going off into its own land. Wakelocks also ended up proving a reasonable power management approach (despite complaints they didn't use existing linux mechanisms etc). There was a big push ultimately to get wakelocks into mainline and drive some convergence.

_msw_ 6y ago

NB: it is not "AWS Babelfish". It is (currently) "Babelfish for PostgreSQL". https://babelfish-for-postgresql.github.io/babelfish-for-postgresql/ https://babelfish-for-postgresql.github.io/babelfish-for-pos... Can the title of this post be changed to remove "AWS" please?

dang 6y ago
The opening paragraph of the article says " Amazon AWS announced Babelfish" and links to https://aws.amazon.com/blogs/opensource/want-more-postgresql-you-just-might-like-babelfish/ https://aws.amazon.com/blogs/opensource/want-more-postgresql... from the AWS blog. This all seems accurate to me. Is there some other reason why we should take AWS out of the title? Branding is not a reason that HN readers would support, and I think if we took AWS out for that reason, the community would accuse us of trying to obscure the origins of the project. Internet people are trigger-happy about this kind of thing, and too quick to accuse us of serving $BigCo agenda as it is.
- _msw_ 6y ago
  I think we are trying to establish the brand of this project in these early days, and to have that brand distinct from AWS. I see this like "Mozilla" was separate from "Netscape" in their early days [1] (note, this link will not work when the referrer is from HN. Archive link provided for convenience [2]). It is a project that AWS is sponsoring and stewarding, but from what I currently know, I think the intent is for it to be community-centric in the long term. If successful, Babelfish will grow a community, and work harmoniously as an extension of PostgreSQL. And including "AWS" in the brand of the project may hamper the efforts to build a community around it (in addition to complications of using a brand / trademark like "AWS" in an open-source project). The TL;DR is: "AWS" is not part of the branding of Babelfish for PostgreSQL, and that is intentional. The reasons for this aren't about obscuring the origins or initial sponsorship of the work, but are rather about establishing a separate identity. [1] https://www.jwz.org/blog/2016/10/they-live-and-the-secret-history-of-the-mozilla-logo/#post-13248461:~:text=With%20the%20launch%20of%20Mozilla.org%2C%20I,to%20some%20degree%20from%20Netscape%20itself https://www.jwz.org/blog/2016/10/they-live-and-the-secret-hi... [2] http://archive.is/UMMCx http://archive.is/UMMCx