13 ms·

My bet is that it is built on top of Aurora PostgreSQL. By looking at the "Limits" section (https://docs.aws.amazon.com/documentdb/latest/developerguide/limits.

My bet is that it is built on top of Aurora PostgreSQL. By looking at the "Limits" section (https://docs.aws.amazon.com/documentdb/latest/developerguide/limits.html https://docs.aws.amazon.com/documentdb/latest/developerguide...), identifiers are limited to 63 characters and the same characters that PostgreSQL limits identifiers to; and a collection size limit of 32TB, coincidentally maximum PostgreSQL table size. Edit: I can confirm: does not allow the UTF-8 null character in strings: https://docs.aws.amazon.com/documentdb/latest/developerguide/functional-differences.html https://docs.aws.amazon.com/documentdb/latest/developerguide... ... It is written on top of PostgreSQL.

deleted 8y ago
[deleted]
misframer 8y ago
I think they're built on a common storage system, just like the MySQL compatible version too.
erikig 8y ago
It would be nice if Amazon provided an API to access the data via SQL alongside the MongoDB API; I've seen quite a number of organizations migrate from mongo to Postgres once they get out of the rapid development phase. This would make that transition butter smooth.
- ahachete 8y ago
  That would make the internal representation used an "API" and thus won't be able to change it in the future. Apparently, they are using a 1:1 mapping between a collection and a table. Either by flattening the document or by using jsonb or equivalent. I'm not a big believer this is good for performance reasons, at least compared to a more normalized approach like the one we did for https://www.torodb.com https://www.torodb.com But they may change it in the future --if they don't expose the SQL API to their internal representation.
- scarface74 8y ago
  Anecdote: I led a C# project where we could seamlessly switch back and forth between Mongo and SQL Server without changing the underlying LINQ expressions. We sent the expressions to the Mongo driver and they got translated to MongoQuery we sent the expressions to Entity Framework and they got translated to Sql Server.
  - deleted 8y ago
    [deleted]
  - manigandham 8y ago
    C# is ahead of the game with LINQ, expression syntax, and the entire Rosyln platform. Passing an IQueryable<> around that can be interpreted and transformed for multiple backends is a incredibly productive. I wish more people knew about this, and .NET in general.
    - scarface74 8y ago
      And I’ve seen a few Java and Javascript libraries that purport to “implement LINQ” and they don’t get the power of LINQ is not the syntax, it’s that LINQ translates your code to expressions that can be parsed and translated to any back end query - it’s not just an ORM. I’ve seen a LINQ to REST API provider.
- talawahdotnet 8y ago
  I doubt that they actually built this on top of Postgres. They probably just integrated the WiredTiger[1] storage engine used by Mongo with their Aurora storage subsystem. I am however really hoping Amazon provides a MySQL 8.0 compatible version of Aurora with full support for its new hybrid SQL and Document Store interfaces[2] courtesy of the X DevAPI[3] and lightweight "serverless" friendly connections courtesy of the new X Protocol. That way your don't have to choose just one approach, and you can have your data in one place with high reliability and durability. My ultimate pipe dream would be that they also provided a redis compatible key/value interface that allows you to fetch simple values directly from the underlying innodb storage engine without going thru the SQL layer, similar to how the memcached plugin currently works[4] 1. https://github.com/wiredtiger/wiredtiger https://github.com/wiredtiger/wiredtiger 2. https://mysqlserverteam.com/mysql-8-0-announcing-ga-of-the-mysql-document-store/ https://mysqlserverteam.com/mysql-8-0-announcing-ga-of-the-m... 3. https://dev.mysql.com/doc/x-devapi-userguide/en/devapi-users-introduction.html https://dev.mysql.com/doc/x-devapi-userguide/en/devapi-users... 4. https://dev.mysql.com/doc/refman/8.0/en/innodb-memcached.html https://dev.mysql.com/doc/refman/8.0/en/innodb-memcached.htm...
  - mrep 8y ago
    > I doubt that they actually built this on top of Postgres. Maybe not (but OP makes a lot of good points for why it is), but it is still based on the aurora limits, 64TB of size, 15 low latency read replicas in minutes, and presumably 1 write capacity which makes it a laughable nosql system since it cannot scale past 1 servers write capacity.
    - sorenbs 8y ago
      Are you aware that they are working on multi-master for Aurora? https://aws.amazon.com/about-aws/whats-new/2017/11/sign-up-for-the-preview-of-amazon-aurora-multi-master/ https://aws.amazon.com/about-aws/whats-new/2017/11/sign-up-f...
  - UlfWendel 8y ago
    What's the motivation for a faster access path to InnoDB: performance? X DevAPI and X Protocol/X Plugin could team up and map K/V style access to the server internal InnoDB API instead of using a SQL service as it is currently done. They could try to do it "transparently" or let you set hints. Whatever is desired from an application standpoint.
- StreamBright 8y ago
  And there are organizations who can do rapid development in Postgres.
ahachete 8y ago
Corollary: PostgreSQL is also web-scale! ;P
Scarbutt 8y ago
Aurora Postgres isn't really Postgres(only compatible), or is it?
- idunno246 8y ago
  The storage engine is different, but the frontend is actual Postgres
  - deleted 8y ago
    [deleted]
moltar 8y ago
Wasn’t latest Mongo built on Postgres backend too?
- manigandham 8y ago
  No, MongoDB has its own storage engines, it's not built on top of anything else.
- sbr464 8y ago
  I think your thinking of the BI Connector for analytics/SQL compatibility. From the docs: Changed in version 2.0: Version 2.0 of the MongoDB Connector for BI introduces a new architecture that replaces the previous PostgreSQL foreign data wrapper with the new mongosqld.
talawahtech 8y ago
It sounds like it is built on top of the Aurora storage subsystem that is used by both Aurora MySQl and Aurora Postgres[1]. I kinda expected them to build it on top of DynamoDb's backend and provide the same kind of "Serverless" on demand experience, but I guess the architecture didn't fit, or maybe this was just faster. 1. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Overview.StorageReliability.html https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...
- mrep 8y ago
  Definitely because it was faster. Amazon's strategy is to launch new features ASAP and then rely on everyone having to be on-call to fix shit when it inevitably breaks in prod because they rushed to launch. I will admit that while their "operational excellence" is shit, the security engineers do have quite a bit of power to block launches so their security isn't as bad as the reliability. However, the fact that writes aren't horizontally scalable makes it a laughable nosql database but it probably satisfies the checkmark for enough of their enterprise customers that it will be a mild success and they'll keep it on life support like simpledb forever until they implement a proper solution assuming there is enough demand for it.
  - drb91 8y ago
    MongoDB is not horizontally scalable either is it?
    - mrep 8y ago
      Yes it is: https://docs.mongodb.com/manual/sharding/ https://docs.mongodb.com/manual/sharding/
  - senderista 8y ago
    I was there for the launch of a major AWS service where they had an entire separate team working on the next iteration since well before launch (because the initial design wasn’t even intended to be sustainable). They are happy to incur technical risk (and in this case, to eat major losses in hardware costs) in order to be first to market.
garyclarke27 8y ago
Interesting- Reminds me, I wish Postgres would increase this default identifier limit to 255 - or make it easily user configurable. It can be done by a sophisticated user, but only via special compilation and only when first installed, which is a right pain. I find Long identifier names useful for constraint names and foreign key names auto generated by code.

deleted 8y ago

[deleted]

misframer 8y ago

I think they're built on a common storage system, just like the MySQL compatible version too.

erikig 8y ago

It would be nice if Amazon provided an API to access the data via SQL alongside the MongoDB API; I've seen quite a number of organizations migrate from mongo to Postgres once they get out of the rapid development phase. This would make that transition butter smooth.

ahachete 8y ago
That would make the internal representation used an "API" and thus won't be able to change it in the future. Apparently, they are using a 1:1 mapping between a collection and a table. Either by flattening the document or by using jsonb or equivalent. I'm not a big believer this is good for performance reasons, at least compared to a more normalized approach like the one we did for https://www.torodb.com https://www.torodb.com But they may change it in the future --if they don't expose the SQL API to their internal representation.
scarface74 8y ago
Anecdote: I led a C# project where we could seamlessly switch back and forth between Mongo and SQL Server without changing the underlying LINQ expressions. We sent the expressions to the Mongo driver and they got translated to MongoQuery we sent the expressions to Entity Framework and they got translated to Sql Server.
- deleted 8y ago
  [deleted]
- manigandham 8y ago
  C# is ahead of the game with LINQ, expression syntax, and the entire Rosyln platform. Passing an IQueryable<> around that can be interpreted and transformed for multiple backends is a incredibly productive. I wish more people knew about this, and .NET in general.
  - scarface74 8y ago
    And I’ve seen a few Java and Javascript libraries that purport to “implement LINQ” and they don’t get the power of LINQ is not the syntax, it’s that LINQ translates your code to expressions that can be parsed and translated to any back end query - it’s not just an ORM. I’ve seen a LINQ to REST API provider.
talawahdotnet 8y ago
I doubt that they actually built this on top of Postgres. They probably just integrated the WiredTiger[1] storage engine used by Mongo with their Aurora storage subsystem. I am however really hoping Amazon provides a MySQL 8.0 compatible version of Aurora with full support for its new hybrid SQL and Document Store interfaces[2] courtesy of the X DevAPI[3] and lightweight "serverless" friendly connections courtesy of the new X Protocol. That way your don't have to choose just one approach, and you can have your data in one place with high reliability and durability. My ultimate pipe dream would be that they also provided a redis compatible key/value interface that allows you to fetch simple values directly from the underlying innodb storage engine without going thru the SQL layer, similar to how the memcached plugin currently works[4] 1. https://github.com/wiredtiger/wiredtiger https://github.com/wiredtiger/wiredtiger 2. https://mysqlserverteam.com/mysql-8-0-announcing-ga-of-the-mysql-document-store/ https://mysqlserverteam.com/mysql-8-0-announcing-ga-of-the-m... 3. https://dev.mysql.com/doc/x-devapi-userguide/en/devapi-users-introduction.html https://dev.mysql.com/doc/x-devapi-userguide/en/devapi-users... 4. https://dev.mysql.com/doc/refman/8.0/en/innodb-memcached.html https://dev.mysql.com/doc/refman/8.0/en/innodb-memcached.htm...
- mrep 8y ago
  > I doubt that they actually built this on top of Postgres. Maybe not (but OP makes a lot of good points for why it is), but it is still based on the aurora limits, 64TB of size, 15 low latency read replicas in minutes, and presumably 1 write capacity which makes it a laughable nosql system since it cannot scale past 1 servers write capacity.
  - sorenbs 8y ago
    Are you aware that they are working on multi-master for Aurora? https://aws.amazon.com/about-aws/whats-new/2017/11/sign-up-for-the-preview-of-amazon-aurora-multi-master/ https://aws.amazon.com/about-aws/whats-new/2017/11/sign-up-f...
- UlfWendel 8y ago
  What's the motivation for a faster access path to InnoDB: performance? X DevAPI and X Protocol/X Plugin could team up and map K/V style access to the server internal InnoDB API instead of using a SQL service as it is currently done. They could try to do it "transparently" or let you set hints. Whatever is desired from an application standpoint.
StreamBright 8y ago
And there are organizations who can do rapid development in Postgres.

ahachete 8y ago

Corollary: PostgreSQL is also web-scale! ;P

Scarbutt 8y ago

Aurora Postgres isn't really Postgres(only compatible), or is it?

idunno246 8y ago
The storage engine is different, but the frontend is actual Postgres
- deleted 8y ago
  [deleted]

moltar 8y ago

Wasn’t latest Mongo built on Postgres backend too?

manigandham 8y ago
No, MongoDB has its own storage engines, it's not built on top of anything else.
sbr464 8y ago
I think your thinking of the BI Connector for analytics/SQL compatibility. From the docs: Changed in version 2.0: Version 2.0 of the MongoDB Connector for BI introduces a new architecture that replaces the previous PostgreSQL foreign data wrapper with the new mongosqld.

talawahtech 8y ago

It sounds like it is built on top of the Aurora storage subsystem that is used by both Aurora MySQl and Aurora Postgres[1]. I kinda expected them to build it on top of DynamoDb's backend and provide the same kind of "Serverless" on demand experience, but I guess the architecture didn't fit, or maybe this was just faster. 1. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Overview.StorageReliability.html https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

mrep 8y ago
Definitely because it was faster. Amazon's strategy is to launch new features ASAP and then rely on everyone having to be on-call to fix shit when it inevitably breaks in prod because they rushed to launch. I will admit that while their "operational excellence" is shit, the security engineers do have quite a bit of power to block launches so their security isn't as bad as the reliability. However, the fact that writes aren't horizontally scalable makes it a laughable nosql database but it probably satisfies the checkmark for enough of their enterprise customers that it will be a mild success and they'll keep it on life support like simpledb forever until they implement a proper solution assuming there is enough demand for it.
- drb91 8y ago
  MongoDB is not horizontally scalable either is it?
  - mrep 8y ago
    Yes it is: https://docs.mongodb.com/manual/sharding/ https://docs.mongodb.com/manual/sharding/
- senderista 8y ago
  I was there for the launch of a major AWS service where they had an entire separate team working on the next iteration since well before launch (because the initial design wasn’t even intended to be sustainable). They are happy to incur technical risk (and in this case, to eat major losses in hardware costs) in order to be first to market.

garyclarke27 8y ago

Interesting- Reminds me, I wish Postgres would increase this default identifier limit to 255 - or make it easily user configurable. It can be done by a sophisticated user, but only via special compilation and only when first installed, which is a right pain. I find Long identifier names useful for constraint names and foreign key names auto generated by code.