5 ms·

N00b question: is Amazon aurora a reimplementation of Postgres, or just something that allows you to use Postgres?

by syspec 4y ago

js4ever 4y ago
It's not open source so we don't really know. But from what I read online it's a fork of postgres with a custom storage layer
- melony 4y ago
  Dynamo underneath amirite?
  - nkozyra 4y ago
    Pardon my ignorance but isn't dynamo just a document store itself? Most storage engines are designed and optimized for RDBs, so I'd expect inefficiency here unless there's a layer between them.
    - randtrain34 4y ago
      I think GP was being humorous
  - smalltalks 4y ago
    From one their talk the CTO hinted it was regular EBS. But for many other database such as "Amazon DocumentDB" , "Time Series" humour suggest it's Dynamo.
whirlwin 4y ago
Aurora is a managed Postgres offering, similar to Amazon RDS, but with clustering capabilities. We use Aurora for all apps. It costs more, but needs little to no maintenance from our side
- wahnfrieden 4y ago
  it's way better than vanilla RDS even without any clustering. it is far more stable under load / near capacity. the storage layer is amazing. people here who are underselling it relative to vanilla RDS simply lack operational experience under load and have a checklist-level understanding
  - leesalminen 4y ago
    +1 Aurora was the secret sauce for us to scale a db-heavy SaaS very quickly with no dedicated ops team. Got us an extra 9 in uptime, too. The cost difference from vanilla RDS was negligible for our revenue model. YMMV, of course on that.
  - gcbirzan 4y ago
    Except... That's not always true. We were told the same, but we found the performance to be underwhelming, at best. Concurrent writes to the same row are orders of magnitude worse, certain read patterns are similarly slow, and you are severely limited in how many transactions you can have before the storage daemon gives up and you end up with double digit write latency for any write. Yeah, for the most part it's better, but you lose the page cache buffer to stop you from running oom if you miscalculated your shared buffers, your cpu usage will go up significantly and sometimes things will be slower. While on paper the max IOPS are way higher, you do get more IOPS from migrating and then you run into other issues. If you are or close to the 80k IOPS limit, or 65TB storage limit, there's no alternative, but the "aurora is always faster" line that AWS tries to sell you is bullshit.
    - wahnfrieden 4y ago
      thx for the contrary perspective from your experience
gurjeet 4y ago
Aurora Postgres is a fork of Postgres code, with modifications that allow it to leverage AWS' Aurora storage, and a few other things (replicas, etc.). The goal is to remain 100% Postgres compatible. So the network protocol, SQL queries, and tools, all can be used without any change. There are (were?) some (read: very few) caveats/restrictions, though, because of the heritage of, and need to be compatible with, Amazon RDS Postgres. Disclosure: Member of the founding-team of Aurora Postgres; left the team quite a while back, so my knowledge is quite outdated. Edit: There's a lot of literature, and public information (articles, conference talks, etc.) published by the Aurora team over the years, which all allude to these facts, and you can use that material to build your confidence that it is in fact Postgres compatible.
- metadat 4y ago
  Interesting, thanks for sharing this tidbit. For no particularly good reason, I was under the impression Postgres Aurora was just a Postgres compatibility interface layer on top of a common substrate shared with the MySQL Aurora implementation.
- gedy 4y ago
  Is this the same situation that came up with Elasticsearch?
  - dhd415 4y ago
    It's a long story, but not exactly. Elasticsearch was developed independently by Shay Banon in 2010 and a couple others who are the founders of Elastic (http://www.elastic.co http://www.elastic.co) under the Apache2 open source license. In March of 2015, Elastic acquired Found, a company offering cloud-hosted managed Elasticsearch clusters which became their "Elastic Cloud" product. In October of 2015, AWS launched their own managed Elasticsearch service called "AWS Elasticsearch Service". Werner Vogels announced it in a now-deleted tweet saying that it was a "great partnership between @elastic and #AWS" when, in fact, there was no partnership at all (thing https://www.theregister.com/2021/01/21/aws_not_ok_says_elastic https://www.theregister.com/2021/01/21/aws_not_ok_says_elast...). In January of 2021, Elastic changed the license under which Elasticsearch was developed from Apache2 to the "Elastic" license which allowed essentially all the same things as Apache2 (free use, modification, etc) so long as it was not used to provide a managed offering of Elasticsearch (https://www.elastic.co/blog/licensing-change https://www.elastic.co/blog/licensing-change). This was done because Elastic viewed itself as developing the Elasticsearch product that AWS immediately took and used for its own "AWS Elasticsearch Service" in competition against Elastic's own Elastic Cloud service which was a significant source of revenue for Elastic. Additionally, Elastic sued AWS for trademark infringement for the use of the "Elasticsearch" trademark in the "AWS Elasticsearch Service" name. In response, AWS forked the last Apache2 release of Elasticsearch and named their fork "OpenSearch". They also changed the name of their managed offering to the "AWS OpenSearch Service". During the same timeframe, many other companies have seen the same risk of AWS taking their open source products and providing competing managed offerings and have chosen various methods of protecting themselves from AWS competition. Many of them (including MongoDB, MariaDB, Confluent, CockroachDB, Sentry.io, Apollo, Graylog, Couchbase) have adopted licenses such as the SSPL, BSL, and even the Elastic license that prevent or strongly discourage AWS from using their products in a managed offering. Others such as Grafana Labs have partnered with AWS (under undisclosed terms) for a managed offering. It's an ongoing tension between companies that want to offer an open source product with a managed offering that monetizes it.
everfrustrated 4y ago
There's some deep dives on the aws YouTube channel that talk about their storage layer if you want some detail. AWS happily share quite a lot more under NDA as well, if that's an option for you. The short answer is the proprietary changes are very well done in a way that provides no meaningful vendor lock-in.

N00b question: is Amazon aurora a reimplementation of Postgres, or just something that allows you to use Postgres?

by syspec 4y ago

N00b question: is Amazon aurora a reimplementation of Postgres, or just something that allows you to use Postgres?

It's not open source so we don't really know. But from what I read online it's a fork of postgres with a custom storage layer

melony 4y ago
Dynamo underneath amirite?
- nkozyra 4y ago
  Pardon my ignorance but isn't dynamo just a document store itself? Most storage engines are designed and optimized for RDBs, so I'd expect inefficiency here unless there's a layer between them.
  - randtrain34 4y ago
    I think GP was being humorous
- smalltalks 4y ago
  From one their talk the CTO hinted it was regular EBS. But for many other database such as "Amazon DocumentDB" , "Time Series" humour suggest it's Dynamo.

whirlwin 4y ago

Aurora is a managed Postgres offering, similar to Amazon RDS, but with clustering capabilities. We use Aurora for all apps. It costs more, but needs little to no maintenance from our side

wahnfrieden 4y ago
it's way better than vanilla RDS even without any clustering. it is far more stable under load / near capacity. the storage layer is amazing. people here who are underselling it relative to vanilla RDS simply lack operational experience under load and have a checklist-level understanding
- leesalminen 4y ago
  +1 Aurora was the secret sauce for us to scale a db-heavy SaaS very quickly with no dedicated ops team. Got us an extra 9 in uptime, too. The cost difference from vanilla RDS was negligible for our revenue model. YMMV, of course on that.
- gcbirzan 4y ago
  Except... That's not always true. We were told the same, but we found the performance to be underwhelming, at best. Concurrent writes to the same row are orders of magnitude worse, certain read patterns are similarly slow, and you are severely limited in how many transactions you can have before the storage daemon gives up and you end up with double digit write latency for any write. Yeah, for the most part it's better, but you lose the page cache buffer to stop you from running oom if you miscalculated your shared buffers, your cpu usage will go up significantly and sometimes things will be slower. While on paper the max IOPS are way higher, you do get more IOPS from migrating and then you run into other issues. If you are or close to the 80k IOPS limit, or 65TB storage limit, there's no alternative, but the "aurora is always faster" line that AWS tries to sell you is bullshit.
  - wahnfrieden 4y ago
    thx for the contrary perspective from your experience

gurjeet 4y ago

Aurora Postgres is a fork of Postgres code, with modifications that allow it to leverage AWS' Aurora storage, and a few other things (replicas, etc.). The goal is to remain 100% Postgres compatible. So the network protocol, SQL queries, and tools, all can be used without any change. There are (were?) some (read: very few) caveats/restrictions, though, because of the heritage of, and need to be compatible with, Amazon RDS Postgres. Disclosure: Member of the founding-team of Aurora Postgres; left the team quite a while back, so my knowledge is quite outdated. Edit: There's a lot of literature, and public information (articles, conference talks, etc.) published by the Aurora team over the years, which all allude to these facts, and you can use that material to build your confidence that it is in fact Postgres compatible.

metadat 4y ago
Interesting, thanks for sharing this tidbit. For no particularly good reason, I was under the impression Postgres Aurora was just a Postgres compatibility interface layer on top of a common substrate shared with the MySQL Aurora implementation.
gedy 4y ago
Is this the same situation that came up with Elasticsearch?
- dhd415 4y ago
  It's a long story, but not exactly. Elasticsearch was developed independently by Shay Banon in 2010 and a couple others who are the founders of Elastic (http://www.elastic.co http://www.elastic.co) under the Apache2 open source license. In March of 2015, Elastic acquired Found, a company offering cloud-hosted managed Elasticsearch clusters which became their "Elastic Cloud" product. In October of 2015, AWS launched their own managed Elasticsearch service called "AWS Elasticsearch Service". Werner Vogels announced it in a now-deleted tweet saying that it was a "great partnership between @elastic and #AWS" when, in fact, there was no partnership at all (thing https://www.theregister.com/2021/01/21/aws_not_ok_says_elastic https://www.theregister.com/2021/01/21/aws_not_ok_says_elast...). In January of 2021, Elastic changed the license under which Elasticsearch was developed from Apache2 to the "Elastic" license which allowed essentially all the same things as Apache2 (free use, modification, etc) so long as it was not used to provide a managed offering of Elasticsearch (https://www.elastic.co/blog/licensing-change https://www.elastic.co/blog/licensing-change). This was done because Elastic viewed itself as developing the Elasticsearch product that AWS immediately took and used for its own "AWS Elasticsearch Service" in competition against Elastic's own Elastic Cloud service which was a significant source of revenue for Elastic. Additionally, Elastic sued AWS for trademark infringement for the use of the "Elasticsearch" trademark in the "AWS Elasticsearch Service" name. In response, AWS forked the last Apache2 release of Elasticsearch and named their fork "OpenSearch". They also changed the name of their managed offering to the "AWS OpenSearch Service". During the same timeframe, many other companies have seen the same risk of AWS taking their open source products and providing competing managed offerings and have chosen various methods of protecting themselves from AWS competition. Many of them (including MongoDB, MariaDB, Confluent, CockroachDB, Sentry.io, Apollo, Graylog, Couchbase) have adopted licenses such as the SSPL, BSL, and even the Elastic license that prevent or strongly discourage AWS from using their products in a managed offering. Others such as Grafana Labs have partnered with AWS (under undisclosed terms) for a managed offering. It's an ongoing tension between companies that want to offer an open source product with a managed offering that monetizes it.

everfrustrated 4y ago

There's some deep dives on the aws YouTube channel that talk about their storage layer if you want some detail. AWS happily share quite a lot more under NDA as well, if that's an option for you. The short answer is the proprietary changes are very well done in a way that provides no meaningful vendor lock-in.