6 ms·

It sounds like it is built on top of the Aurora storage subsystem that is used by both Aurora MySQl and Aurora Postgres[1]. I kinda expected them to build it o

It sounds like it is built on top of the Aurora storage subsystem that is used by both Aurora MySQl and Aurora Postgres[1]. I kinda expected them to build it on top of DynamoDb's backend and provide the same kind of "Serverless" on demand experience, but I guess the architecture didn't fit, or maybe this was just faster. 1. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Overview.StorageReliability.html https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

mrep 8y ago
Definitely because it was faster. Amazon's strategy is to launch new features ASAP and then rely on everyone having to be on-call to fix shit when it inevitably breaks in prod because they rushed to launch. I will admit that while their "operational excellence" is shit, the security engineers do have quite a bit of power to block launches so their security isn't as bad as the reliability. However, the fact that writes aren't horizontally scalable makes it a laughable nosql database but it probably satisfies the checkmark for enough of their enterprise customers that it will be a mild success and they'll keep it on life support like simpledb forever until they implement a proper solution assuming there is enough demand for it.
- drb91 8y ago
  MongoDB is not horizontally scalable either is it?
  - mrep 8y ago
    Yes it is: https://docs.mongodb.com/manual/sharding/ https://docs.mongodb.com/manual/sharding/
- senderista 8y ago
  I was there for the launch of a major AWS service where they had an entire separate team working on the next iteration since well before launch (because the initial design wasn’t even intended to be sustainable). They are happy to incur technical risk (and in this case, to eat major losses in hardware costs) in order to be first to market.
  - Corrado 8y ago
    We used MongoDB in my last job and I just want to say that I would have given up management of that beast in a heartbeat. We didn't stress MongoDB nearly enough to warrant all the effort required to construct it, monitor it, back it up, etc. Even if the performance was crappy, I would have lobbied hard to change to DocumentDB ASAP.
    - halbritt 8y ago
      While I don't love MongoDB, I don't find it to be especially difficult to run. I'm running ~200 instances of MongoDB with a small team and it consumes very little of my attention. ElasticSearch on the other hand...
      - bloomburger 8y ago
        Why is Elastic so difficult to run for you?
  - adaptiveValleys 8y ago
    I'm sure they are, so they a pass those lovely negative externalities onto the customer because they know it's in demand and only they provide that service. If only they had a competitor that could launch the same products a few months later but offered higher reliability off the bat, that could eventually force Amazon to improve their reliability or risk losing customers long term. Being first to market doesn't ensure eventual market dominance. Sure, it could give you important feedback. But if your product is subpar, the feedback will have a ton of noise and possibly be useless. Plus it's not worth creating negative externalities and earning the reputation.
    - nailer 8y ago
      > If only they had a competitor that could launch the same products a few months later but offered higher reliability off the bat Doesn't Azure Cosmos DB do this? From https://docs.microsoft.com/en-us/azure/cosmos-db/introduction https://docs.microsoft.com/en-us/azure/cosmos-db/introductio... > You can elastically scale throughput and storage, and take advantage of fast, single-digit-millisecond data access using your favorite API among SQL, MongoDB, Cassandra, Tables, or Gremlin. Haven't used it though, so would welcome some real world experience.
- tanilama 8y ago
  AWS prioritizes launch above EVERYTHING. It is their strategy, to have market tells them what to build. I think it works, and AWS has yet been brought down by this horizontal complexity. Quite an achievement, but might not be a satisfying experience for the engineers work there.
  - talawahdotnet 8y ago
    It makes sense in terms of feeling out the market as well. If this version of the service takes off it validates the decision to proceed with a more complex/scalable version and it gives them more customer feedback. Standard MVP best practices. The downside is that a lot of their products lack polish which sucks. On the flip side even when they are launched with minimal features, they do tend to be reliable, durable and secure, which is important when it comes to data related services.
  - danpalmer 8y ago
    This is one of the main reasons why I don't like AWS services, everything just seems so half-finished. There's not a lot in AWS that I would trust enough to use in production. I wonder how widespread this view is. I suspect it's more widespread than Amazon realise. They may have optimised into a local maximum where they get a lot of value from being first to market, but could potentially get more by being first to "viable to trust a business on".
    - talawahdotnet 8y ago
      I certainly agree that they seem half finished in terms of features and developer experience, but from the point of view of security and data durability they have an excellent reputation. They typically have a pretty good reliability story as well, but it relies on the customer architecture their solution to take advantage of multiple AZs/Regions, which is often not trivial. As far as being "viable to trust a business on" the numbers don't lie, AWS is number one because customers are running their businesses on AWS. The fact that DocumentDB launched with supporting quotes from Capital One, Dow Jones and WaPo shows that customers were clamoring to use it even before GA. Remember a lot of these customers are coming to AWS because they tried doing themselves and stuggled. When it comes to data, customers trust AWS more than they trust themselves, and rightly so.
- int_19h 8y ago
  I suspect that most MongoDB users never get to the point where they need to horizontally scale (i.e. it gets chosen for fad reasons, not because they actually have something big enough to scale). And the nice thing about this hypothesis, you can test it by looking how successful DocumentDB will turn out to be. ~
- manigandham 8y ago
  Apparently SimpleDB is still used quite a lot internally. As for their market tactics, there's no denying it works as their pace is accelerating and leaving everyone else in the dust. Most customers just want to pay some money and have a solution ready to go, they don't need infinite scaling from day 1, if ever. This focus on actually meeting needs today is what keeps AWS on top while the others take 2 years to launch minor service upgrades.
- talawahdotnet 8y ago
  It definitely sounds like it sucks from the perspective of an internal AWS developer or SRE, but if the AWS systems are architected such that these internal failures aren't seen by end users then AWS's reliability reputation remains fully intact. Customers are paying AWS so that their SREs don't get called, they don't care if the AWS SREs do as long as the system keeps running. Based on the supporting quotes at launch from Capital One, Dow Jones and WaPo it sound like enough customers are ok with vertical write scalability and (pretty awesome) horizontal read scalability for now because it fits their use case and is better than what they had before. Also consider that since the cluster management overhead has been removed from the customer, they can essentially "shard" by using a separate cluster for each sufficiently large service/org/dept, which might actually work out better for them in some respects. Perfect is the enemy of good enough, the architecture might be laughable to you, but it is probably miles ahead of what the customer was using before.
evil-olive 8y ago
DynamoDB is written on top of MySQL (more specifically, MySQL's storage engine, not the query engine) so using Aurora which has a newer design would make sense.
- talawahtech 8y ago
  Saying DynamoDB is built on top of InnoDB is a pretty big oversimplification of a much more complex distributed system[1] and for all we know they could have switched out the low level the storage engine on the backend to something like RocksDB or WiredTiger. The Aurora storage subsystem is much more limited in terms of horizontal scalability and performance, they probably chose it because it was a better/quicker fit. 1. https://youtu.be/yvBR71D0nAQ https://youtu.be/yvBR71D0nAQ
  - evil-olive 8y ago
    Yeah, I used to work on DynamoDB, I know it's more complicated (much more complicated than that video makes out - their code quality was atrocious, like 2000-5000 line Java classes in 3 or 4 deep inheritance hierarchies; no unit tests, only "smoke tests" that took 2 hours to run and were so prone to race conditions that common advice was to close everything else on your machine, run them, then leave them alone while you went to meetings) There was work underway at the time I left to replace InnoDB with WiredTiger. It seemed to be very slow going, and I suspect WiredTiger being acquired by 10gen had a part in it. They also had only 1-2 engineers on the project of ripping out MySQL and replacing it, in a long-lived branch that constantly dealt with merge conflicts from more active feature development happening on mainline. Aurora, simply by virtue of being newer and learning from DDB's mistakes (in the same way DDB learned from SimpleDB and the original Dynamo) probably has better extension points for supporting (MySQL, Postgres, Mongo) in a sane way.
    - talawahdotnet 8y ago
      Interesting, how long ago was that? I would be curious to know if the WiredTiger switch ever happened, and what that support relationship looks like not given the contentious relationship between MongoDB and AWS. The old Wired Tiger Inc website[1] still lists AWS as a customer. Then again, the relationship between AWS and Oracle is even more contentious and Aurora MySQL is one of AWS's most popular products so I don't think they are terribly worried about building on competitor's technologies. 1. http://www.wiredtiger.com/ http://www.wiredtiger.com/

mrep 8y ago

Definitely because it was faster. Amazon's strategy is to launch new features ASAP and then rely on everyone having to be on-call to fix shit when it inevitably breaks in prod because they rushed to launch. I will admit that while their "operational excellence" is shit, the security engineers do have quite a bit of power to block launches so their security isn't as bad as the reliability. However, the fact that writes aren't horizontally scalable makes it a laughable nosql database but it probably satisfies the checkmark for enough of their enterprise customers that it will be a mild success and they'll keep it on life support like simpledb forever until they implement a proper solution assuming there is enough demand for it.

drb91 8y ago
MongoDB is not horizontally scalable either is it?
- mrep 8y ago
  Yes it is: https://docs.mongodb.com/manual/sharding/ https://docs.mongodb.com/manual/sharding/
senderista 8y ago
I was there for the launch of a major AWS service where they had an entire separate team working on the next iteration since well before launch (because the initial design wasn’t even intended to be sustainable). They are happy to incur technical risk (and in this case, to eat major losses in hardware costs) in order to be first to market.
- Corrado 8y ago
  We used MongoDB in my last job and I just want to say that I would have given up management of that beast in a heartbeat. We didn't stress MongoDB nearly enough to warrant all the effort required to construct it, monitor it, back it up, etc. Even if the performance was crappy, I would have lobbied hard to change to DocumentDB ASAP.
  - halbritt 8y ago
    While I don't love MongoDB, I don't find it to be especially difficult to run. I'm running ~200 instances of MongoDB with a small team and it consumes very little of my attention. ElasticSearch on the other hand...
    - bloomburger 8y ago
      Why is Elastic so difficult to run for you?
- adaptiveValleys 8y ago
  I'm sure they are, so they a pass those lovely negative externalities onto the customer because they know it's in demand and only they provide that service. If only they had a competitor that could launch the same products a few months later but offered higher reliability off the bat, that could eventually force Amazon to improve their reliability or risk losing customers long term. Being first to market doesn't ensure eventual market dominance. Sure, it could give you important feedback. But if your product is subpar, the feedback will have a ton of noise and possibly be useless. Plus it's not worth creating negative externalities and earning the reputation.
  - nailer 8y ago
    > If only they had a competitor that could launch the same products a few months later but offered higher reliability off the bat Doesn't Azure Cosmos DB do this? From https://docs.microsoft.com/en-us/azure/cosmos-db/introduction https://docs.microsoft.com/en-us/azure/cosmos-db/introductio... > You can elastically scale throughput and storage, and take advantage of fast, single-digit-millisecond data access using your favorite API among SQL, MongoDB, Cassandra, Tables, or Gremlin. Haven't used it though, so would welcome some real world experience.
tanilama 8y ago
AWS prioritizes launch above EVERYTHING. It is their strategy, to have market tells them what to build. I think it works, and AWS has yet been brought down by this horizontal complexity. Quite an achievement, but might not be a satisfying experience for the engineers work there.
- talawahdotnet 8y ago
  It makes sense in terms of feeling out the market as well. If this version of the service takes off it validates the decision to proceed with a more complex/scalable version and it gives them more customer feedback. Standard MVP best practices. The downside is that a lot of their products lack polish which sucks. On the flip side even when they are launched with minimal features, they do tend to be reliable, durable and secure, which is important when it comes to data related services.
- danpalmer 8y ago
  This is one of the main reasons why I don't like AWS services, everything just seems so half-finished. There's not a lot in AWS that I would trust enough to use in production. I wonder how widespread this view is. I suspect it's more widespread than Amazon realise. They may have optimised into a local maximum where they get a lot of value from being first to market, but could potentially get more by being first to "viable to trust a business on".
  - talawahdotnet 8y ago
    I certainly agree that they seem half finished in terms of features and developer experience, but from the point of view of security and data durability they have an excellent reputation. They typically have a pretty good reliability story as well, but it relies on the customer architecture their solution to take advantage of multiple AZs/Regions, which is often not trivial. As far as being "viable to trust a business on" the numbers don't lie, AWS is number one because customers are running their businesses on AWS. The fact that DocumentDB launched with supporting quotes from Capital One, Dow Jones and WaPo shows that customers were clamoring to use it even before GA. Remember a lot of these customers are coming to AWS because they tried doing themselves and stuggled. When it comes to data, customers trust AWS more than they trust themselves, and rightly so.
int_19h 8y ago
I suspect that most MongoDB users never get to the point where they need to horizontally scale (i.e. it gets chosen for fad reasons, not because they actually have something big enough to scale). And the nice thing about this hypothesis, you can test it by looking how successful DocumentDB will turn out to be. ~
manigandham 8y ago
Apparently SimpleDB is still used quite a lot internally. As for their market tactics, there's no denying it works as their pace is accelerating and leaving everyone else in the dust. Most customers just want to pay some money and have a solution ready to go, they don't need infinite scaling from day 1, if ever. This focus on actually meeting needs today is what keeps AWS on top while the others take 2 years to launch minor service upgrades.
talawahdotnet 8y ago
It definitely sounds like it sucks from the perspective of an internal AWS developer or SRE, but if the AWS systems are architected such that these internal failures aren't seen by end users then AWS's reliability reputation remains fully intact. Customers are paying AWS so that their SREs don't get called, they don't care if the AWS SREs do as long as the system keeps running. Based on the supporting quotes at launch from Capital One, Dow Jones and WaPo it sound like enough customers are ok with vertical write scalability and (pretty awesome) horizontal read scalability for now because it fits their use case and is better than what they had before. Also consider that since the cluster management overhead has been removed from the customer, they can essentially "shard" by using a separate cluster for each sufficiently large service/org/dept, which might actually work out better for them in some respects. Perfect is the enemy of good enough, the architecture might be laughable to you, but it is probably miles ahead of what the customer was using before.

evil-olive 8y ago

DynamoDB is written on top of MySQL (more specifically, MySQL's storage engine, not the query engine) so using Aurora which has a newer design would make sense.

talawahtech 8y ago
Saying DynamoDB is built on top of InnoDB is a pretty big oversimplification of a much more complex distributed system[1] and for all we know they could have switched out the low level the storage engine on the backend to something like RocksDB or WiredTiger. The Aurora storage subsystem is much more limited in terms of horizontal scalability and performance, they probably chose it because it was a better/quicker fit. 1. https://youtu.be/yvBR71D0nAQ https://youtu.be/yvBR71D0nAQ
- evil-olive 8y ago
  Yeah, I used to work on DynamoDB, I know it's more complicated (much more complicated than that video makes out - their code quality was atrocious, like 2000-5000 line Java classes in 3 or 4 deep inheritance hierarchies; no unit tests, only "smoke tests" that took 2 hours to run and were so prone to race conditions that common advice was to close everything else on your machine, run them, then leave them alone while you went to meetings) There was work underway at the time I left to replace InnoDB with WiredTiger. It seemed to be very slow going, and I suspect WiredTiger being acquired by 10gen had a part in it. They also had only 1-2 engineers on the project of ripping out MySQL and replacing it, in a long-lived branch that constantly dealt with merge conflicts from more active feature development happening on mainline. Aurora, simply by virtue of being newer and learning from DDB's mistakes (in the same way DDB learned from SimpleDB and the original Dynamo) probably has better extension points for supporting (MySQL, Postgres, Mongo) in a sane way.
  - talawahdotnet 8y ago
    Interesting, how long ago was that? I would be curious to know if the WiredTiger switch ever happened, and what that support relationship looks like not given the contentious relationship between MongoDB and AWS. The old Wired Tiger Inc website[1] still lists AWS as a customer. Then again, the relationship between AWS and Oracle is even more contentious and Aurora MySQL is one of AWS's most popular products so I don't think they are terribly worried about building on competitor's technologies. 1. http://www.wiredtiger.com/ http://www.wiredtiger.com/