7 ms·

I can safely say that the team members working in DynamoDB are very skilled and they care deeply about the product. They really work hard and think of interesti

I can safely say that the team members working in DynamoDB are very skilled and they care deeply about the product. They really work hard and think of interesting solutions to a lot of problems that their biggest customers face which is great from a product standpoint. There are some pretty smart people working there. Engineering, however, was a disaster story. Code is horribly written and very few tests are maintained to make sure deployments go without issues. There was too much emphasis on deployment and getting fixes/features out over making sure it won't break anything else. It was a common scenario to release a new feature and put duct tape all around it to make sure it "works". And way too many operational issues. There are a lot of ways to break DynamoDB :) Overall, though, the product is very solid and it's one of the few database that you can say "just works" when it comes to scalability and reliability (as most AWS services are) I worked at DynamoDB for over 2 years.

vasili111 5y ago
>Engineering, however, was a disaster story. Code is horribly written and very few tests are maintained to make sure deployments go without issues. There was too much emphasis on deployment and getting fixes/features out over making sure it won't break anything else. It was a common scenario to release a new feature and put duct tape all around it to make sure it "works". And way too many operational issues. There are a lot of ways to break DynamoDB :) >Overall, though, the product is very solid and it's one of the few database that you can say "just works" when it comes to scalability and reliability (as most AWS services are) How those two can coexist?
- badhombres 5y ago
  I mean eventually enough duct tape can be solid like a tank :)
  - jeffreygoesto 5y ago
    Or a bridge... https://www.popularmechanics.com/science/a5732/mythbusters-build-duct-tape-bridge/ https://www.popularmechanics.com/science/a5732/mythbusters-b...
    - bastardoperator 5y ago
      Or a boat... https://flexsealproducts.com/products/flex-tape https://flexsealproducts.com/products/flex-tape
- meepmorp 5y ago
  You need the really good duct tape.
- indogooner 5y ago
  It sounds incredulous but I have heard similar things about Oracle. May be a large dev team can duct tape enough so that the product is solid.
  - eternalban 5y ago
    They both likely have solid 80% solutions (design) and incrementally cover the 20% gap as need arises. This in turn adds to operational complexity. Alternative would be to attempt a near 'perfect' solution for the product requirements and that may either hit an impossibility wall or may require substantial long term effort that would impede product development cycles. So likely the former approach is the smarter choice.
  - gfd 5y ago
    You're probably thinking of this comment, oracle and 25 million lines of c code: https://news.ycombinator.com/item?id=18442941 https://news.ycombinator.com/item?id=18442941
- 0xbadcafebee 5y ago
  You throw bodies at it. A small bunch of people will be overworked, stressed, constantly fighting fires and struggling to fight technical debt, implement features, and keep the thing afloat. Production is always a hair away from falling over but luck and grit keeps it running. To the team it's a nightmare, to the business everything is fine.
  - wnolens 5y ago
    this is the answer. source: currently being burned out on an adjacent aws team..
    - 0xbadcafebee 5y ago
      That sucks, man. If they won't move you to another team, just get out of there. We don't benefit by suffering for them, and they're not gonna change.
      - wnolens 5y ago
        Yea. I can probably move to a more chill team, but I wouldn't work on anything nearly as cutting edge. I mentally check out for weeks at a time, then get back into it and deliver something large. I'm low key job hunting, but don't entirely trust that it'll be different anywhere else (previous jobs were like this too)
  - dustingetz 5y ago
    literally every co if you want to know why capitalism causes this, start a startup and prioritize quality, do not get to market, do not raise money, do not pass go, watch dumpster fires with millions of betrayed and angry users raise their series d
- deleted 5y ago
  [deleted]
- StreamBright 5y ago
  AWS level of disaster is very different to an average disaster. :)
vp8989 5y ago
"And way too many operational issues." I've seen this kind of thing mentioned many times, pretty baffling TBH based on Dynamo's pretty good reputation in industry. Are these mostly to the stateless components of the product, or do they see data loss?
- salil999 5y ago
  I can't say in risk of violating some NDA but a lot of it is internal stuff that customers will never even be aware of or it would require too much effort for them to break. There are times when bad deployments happen and customers were impacted.
  - haimez 5y ago
    Careful about running your systems in us-east-1, folks
otterley 5y ago
Customers care about the outcome, not the internal process. Besides, I’ve never worked at any sizable company in my 20+-year-long career where I didn’t conclude, “it’s a miracle this garbage works at all.” Enjoy the sausage, but if you have a weak stomach, don’t watch how it’s made. (I work for AWS but not on the DynamoDB team and I have no first-hand knowledge of the above claim. Opinions are my own and not those of my employer.)
- salil999 5y ago
  Exactly this. I was too young at the time to grasp this idea.
- chrisfosterelli 5y ago
  > Customers care about the outcome, not the internal process. This is true though there's only so much technical debt and internal process chaos you can create before it affects the outcome. It's a leading indicator, so by the time customers are feeling that pain you've got a lot of work in front of you before you can turn it around, if at all, and customers are not going to be happy for that duration. Technical debt is not something to completely defeat or completely ignore, instead it's a tradeoff to manage.
  - jsdalton 5y ago
    This article from Martin Fowler explores your point in greater depth. It's a good read: https://martinfowler.com/articles/is-quality-worth-cost.html https://martinfowler.com/articles/is-quality-worth-cost.html One concrete problem with technical debt the article highlights is it that negatively impacts the time to deliver new features. Customers today usually expect not only a great initial feature set from a product, but also a steady stream of improvements and growth, along with responsiveness to feedback and pain points.
- wnolens 5y ago
  > Customers care about the outcome, not the internal process Additionally, the business cares about the outcome, not the internal process. Ostensibly, the business should care about process but it actually doesn't matter as long as the product is just good enough to obtain/retain customers, and the people spending the money (managers) aren't incentivized to make costs any lower than previously promised (status quo).
geodel 5y ago
Very interesting! To me the takeaway is large/interesting/challenging engineering projects are pretty close to disasters generally. Some time they do become disaster actually. On the other hand if a project looks like straight up designed, neatly put into JIRA stories, and developers deliver code consistently week after week then it may be a successfully planned and delivered project. But it would mostly be doing stuff that has already been many times over and likely by same people on team. At least this has been my experience while working on standardized / templated projects vs something new.
- notyourwork 5y ago
  Challenging the cutting edge of your product domain is what I get from this. Easy things are easy and predictable. Hard things and unpredictable evolving requirements are a tension against the initial system design which is the foundation of your code base. Over time the larger projects get the perhaps further they deviate from the original design. If you could predict it up front in many cases its not all that interesting or challenging of a problem. Duct tape is fine to use as long as you understand when you've gone too far and might want to re-design from scratch based on prior learnings.
- marcosdumay 5y ago
  On the other hand, if you don't go solder things and remove the duct tape from time to time, you will always come closer to a disaster, never further away. Some projects are run like the Doomsday Clock, and nobody can get anything done. Other ones increase and decrease on complexity all the time, and those tend to catch-up to the first set quite quickly.
0xbadcafebee 5y ago
I worked at a company who re-implemented the entire Dynamo paper and API, and it was exactly the same story. Completely eliminated all my illusions about the supposed superiority of distributed systems. It was a mound of tires held together with duct tape, with a tiki torch in each tire.
- yftsui 5y ago
  Dynamo paper and DynamoDB are two very different things…
  - 0xbadcafebee 5y ago
    Err you're right, they re-used something that implemented the Dynamo paper (Riak) and implemented on top of it the DynamoDB API.
- AtlasBarfed 5y ago
  Did they have a spare 100 million hanging around to burn? That seems pretty ridiculous. Why did they not just run cassandra?
  - 0xbadcafebee 5y ago
    They did have 100 million to burn, but my mostly-wild-guess is it was closer to $1.5M/yr. But that gives you an in-house SaaS DB used across a hundred other teams/products/services, so it actually saved money (and nothing else matched its performance/CAP/functionality). Cassandra is too opinionated and its CAP behavior wasn't great for a service like this, so they built on top of Riak. (This also eliminated any thoughts I had about Erlang being some uber-language for distributed systems, as there were (are?) tons of bugs and missing edge cases in Riak)
    - staticassertion 5y ago
      Erlang gives you great primitives for building reliable protocols, but they're just primitives, and there are tons of footguns since building protocols is hard.
AtlasBarfed 5y ago
So they just rolled out global replication, and I can't for the life of me figure out how they resolve write conflicts without cell timestamps or any other obvious CRDT measures. Questions were handwaved away, and the usual Amazon black box non-answers which always smells like they are hiding problems. Any ideas how this is working? It seems bolt-on and not well thought out, and I doubt they'll ever pay for Aphyr to put it through his torture tests.
- greiskul 5y ago
  From: https://aws.amazon.com/dynamodb/global-tables/ https://aws.amazon.com/dynamodb/global-tables/ Consistency and conflict resolution Any changes made to any item in any replica table are replicated to all the other replicas within the same global table. In a global table, a newly written item is usually propagated to all replica tables within a second. With a global table, each replica table stores the same set of data items. DynamoDB does not support partial replication of only some of the items. If applications update the same item in different Regions at about the same time, conflicts can arise. To help ensure eventual consistency, DynamoDB global tables use a last-writer-wins reconciliation between concurrent updates, in which DynamoDB makes a best effort to determine the last writer. With this conflict resolution mechanism, all replicas agree on the latest update and converge toward a state in which they all have identical data.
  - WatchDog 5y ago
    So a write than doesn’t “win” just gets silently discarded?
- luhn 5y ago
  Honestly your expectations are too high. Conflict resolution is row-level last-write-wins. It's not a globally distributed database, it's just a pile of regional DynamoDB tables duct taped together... They're not going to hire Aphyr for testing because there's nothing for him to test.
deanCommie 5y ago
If the developers are happy about the code and testing quality of a project, then you waited too long to ship. If the customers don't have any feedback or missed feature asks at launch, you waited too long to ship. You know who has great internal code and test quality? Google. Which is why Google doesn't ship. They're a wealth distribution charity for talented engineers. And their competitive advantage is that they lure talented people away from other companies where they might actually ship something and compete with Google, to instead park them, distract them with toys, beer kegs, readability reviews, and monorepo upgrades.
sam0x17 5y ago
It's a shame they don't open source it. It's funny too, being AWS they really don't have to worry about AWS running a cheaper service, so at that point why not open source it.
- deleted 5y ago
  [deleted]
- nexuist 5y ago
  They probably view it as a competitive advantage that Azure or GCP would try to copy if they figured out the "secret sauce."
  - blowski 5y ago
    Azure has Cosmos which is arguably better than DynamoDB for a lot of use cases.
  - staticassertion 5y ago
    I kinda doubt it. It's probably just that open sourcing it won't provide much utility (I bet lots of code is aws specific) and just adds a new maintenance burden for them.
    - bitexploder 5y ago
      The way you need to write code for a massively scalable service is just different. And the things you need to operate a service are also just different.
  - sam0x17 5y ago
    Azure has Cosmos and Google has Datastore. They would never.
  - pclmulqdq 5y ago
    Azure has a better product (Cosmos is damn good) and Google engineers have too much hubris to import something from lowly Amazon engineers.
digitalgangsta 5y ago
I've worked at 5 different tech companies now - this is par for course. And every single one, wished they could go back and do it again, but at that point the product was too successful so they ran with it.

vasili111 5y ago

>Engineering, however, was a disaster story. Code is horribly written and very few tests are maintained to make sure deployments go without issues. There was too much emphasis on deployment and getting fixes/features out over making sure it won't break anything else. It was a common scenario to release a new feature and put duct tape all around it to make sure it "works". And way too many operational issues. There are a lot of ways to break DynamoDB :) >Overall, though, the product is very solid and it's one of the few database that you can say "just works" when it comes to scalability and reliability (as most AWS services are) How those two can coexist?

badhombres 5y ago
I mean eventually enough duct tape can be solid like a tank :)
- jeffreygoesto 5y ago
  Or a bridge... https://www.popularmechanics.com/science/a5732/mythbusters-build-duct-tape-bridge/ https://www.popularmechanics.com/science/a5732/mythbusters-b...
  - bastardoperator 5y ago
    Or a boat... https://flexsealproducts.com/products/flex-tape https://flexsealproducts.com/products/flex-tape
meepmorp 5y ago
You need the really good duct tape.
indogooner 5y ago
It sounds incredulous but I have heard similar things about Oracle. May be a large dev team can duct tape enough so that the product is solid.
- eternalban 5y ago
  They both likely have solid 80% solutions (design) and incrementally cover the 20% gap as need arises. This in turn adds to operational complexity. Alternative would be to attempt a near 'perfect' solution for the product requirements and that may either hit an impossibility wall or may require substantial long term effort that would impede product development cycles. So likely the former approach is the smarter choice.
- gfd 5y ago
  You're probably thinking of this comment, oracle and 25 million lines of c code: https://news.ycombinator.com/item?id=18442941 https://news.ycombinator.com/item?id=18442941
0xbadcafebee 5y ago
You throw bodies at it. A small bunch of people will be overworked, stressed, constantly fighting fires and struggling to fight technical debt, implement features, and keep the thing afloat. Production is always a hair away from falling over but luck and grit keeps it running. To the team it's a nightmare, to the business everything is fine.
- wnolens 5y ago
  this is the answer. source: currently being burned out on an adjacent aws team..
  - 0xbadcafebee 5y ago
    That sucks, man. If they won't move you to another team, just get out of there. We don't benefit by suffering for them, and they're not gonna change.
    - wnolens 5y ago
      Yea. I can probably move to a more chill team, but I wouldn't work on anything nearly as cutting edge. I mentally check out for weeks at a time, then get back into it and deliver something large. I'm low key job hunting, but don't entirely trust that it'll be different anywhere else (previous jobs were like this too)
- dustingetz 5y ago
  literally every co if you want to know why capitalism causes this, start a startup and prioritize quality, do not get to market, do not raise money, do not pass go, watch dumpster fires with millions of betrayed and angry users raise their series d
deleted 5y ago
[deleted]
StreamBright 5y ago
AWS level of disaster is very different to an average disaster. :)

vp8989 5y ago

"And way too many operational issues." I've seen this kind of thing mentioned many times, pretty baffling TBH based on Dynamo's pretty good reputation in industry. Are these mostly to the stateless components of the product, or do they see data loss?

salil999 5y ago
I can't say in risk of violating some NDA but a lot of it is internal stuff that customers will never even be aware of or it would require too much effort for them to break. There are times when bad deployments happen and customers were impacted.
- haimez 5y ago
  Careful about running your systems in us-east-1, folks

otterley 5y ago

Customers care about the outcome, not the internal process. Besides, I’ve never worked at any sizable company in my 20+-year-long career where I didn’t conclude, “it’s a miracle this garbage works at all.” Enjoy the sausage, but if you have a weak stomach, don’t watch how it’s made. (I work for AWS but not on the DynamoDB team and I have no first-hand knowledge of the above claim. Opinions are my own and not those of my employer.)

salil999 5y ago
Exactly this. I was too young at the time to grasp this idea.
chrisfosterelli 5y ago
> Customers care about the outcome, not the internal process. This is true though there's only so much technical debt and internal process chaos you can create before it affects the outcome. It's a leading indicator, so by the time customers are feeling that pain you've got a lot of work in front of you before you can turn it around, if at all, and customers are not going to be happy for that duration. Technical debt is not something to completely defeat or completely ignore, instead it's a tradeoff to manage.
- jsdalton 5y ago
  This article from Martin Fowler explores your point in greater depth. It's a good read: https://martinfowler.com/articles/is-quality-worth-cost.html https://martinfowler.com/articles/is-quality-worth-cost.html One concrete problem with technical debt the article highlights is it that negatively impacts the time to deliver new features. Customers today usually expect not only a great initial feature set from a product, but also a steady stream of improvements and growth, along with responsiveness to feedback and pain points.
wnolens 5y ago
> Customers care about the outcome, not the internal process Additionally, the business cares about the outcome, not the internal process. Ostensibly, the business should care about process but it actually doesn't matter as long as the product is just good enough to obtain/retain customers, and the people spending the money (managers) aren't incentivized to make costs any lower than previously promised (status quo).

geodel 5y ago

Very interesting! To me the takeaway is large/interesting/challenging engineering projects are pretty close to disasters generally. Some time they do become disaster actually. On the other hand if a project looks like straight up designed, neatly put into JIRA stories, and developers deliver code consistently week after week then it may be a successfully planned and delivered project. But it would mostly be doing stuff that has already been many times over and likely by same people on team. At least this has been my experience while working on standardized / templated projects vs something new.

notyourwork 5y ago
Challenging the cutting edge of your product domain is what I get from this. Easy things are easy and predictable. Hard things and unpredictable evolving requirements are a tension against the initial system design which is the foundation of your code base. Over time the larger projects get the perhaps further they deviate from the original design. If you could predict it up front in many cases its not all that interesting or challenging of a problem. Duct tape is fine to use as long as you understand when you've gone too far and might want to re-design from scratch based on prior learnings.
marcosdumay 5y ago
On the other hand, if you don't go solder things and remove the duct tape from time to time, you will always come closer to a disaster, never further away. Some projects are run like the Doomsday Clock, and nobody can get anything done. Other ones increase and decrease on complexity all the time, and those tend to catch-up to the first set quite quickly.

0xbadcafebee 5y ago

I worked at a company who re-implemented the entire Dynamo paper and API, and it was exactly the same story. Completely eliminated all my illusions about the supposed superiority of distributed systems. It was a mound of tires held together with duct tape, with a tiki torch in each tire.

yftsui 5y ago
Dynamo paper and DynamoDB are two very different things…
- 0xbadcafebee 5y ago
  Err you're right, they re-used something that implemented the Dynamo paper (Riak) and implemented on top of it the DynamoDB API.
AtlasBarfed 5y ago
Did they have a spare 100 million hanging around to burn? That seems pretty ridiculous. Why did they not just run cassandra?
- 0xbadcafebee 5y ago
  They did have 100 million to burn, but my mostly-wild-guess is it was closer to $1.5M/yr. But that gives you an in-house SaaS DB used across a hundred other teams/products/services, so it actually saved money (and nothing else matched its performance/CAP/functionality). Cassandra is too opinionated and its CAP behavior wasn't great for a service like this, so they built on top of Riak. (This also eliminated any thoughts I had about Erlang being some uber-language for distributed systems, as there were (are?) tons of bugs and missing edge cases in Riak)
  - staticassertion 5y ago
    Erlang gives you great primitives for building reliable protocols, but they're just primitives, and there are tons of footguns since building protocols is hard.

AtlasBarfed 5y ago

So they just rolled out global replication, and I can't for the life of me figure out how they resolve write conflicts without cell timestamps or any other obvious CRDT measures. Questions were handwaved away, and the usual Amazon black box non-answers which always smells like they are hiding problems. Any ideas how this is working? It seems bolt-on and not well thought out, and I doubt they'll ever pay for Aphyr to put it through his torture tests.

greiskul 5y ago
From: https://aws.amazon.com/dynamodb/global-tables/ https://aws.amazon.com/dynamodb/global-tables/ Consistency and conflict resolution Any changes made to any item in any replica table are replicated to all the other replicas within the same global table. In a global table, a newly written item is usually propagated to all replica tables within a second. With a global table, each replica table stores the same set of data items. DynamoDB does not support partial replication of only some of the items. If applications update the same item in different Regions at about the same time, conflicts can arise. To help ensure eventual consistency, DynamoDB global tables use a last-writer-wins reconciliation between concurrent updates, in which DynamoDB makes a best effort to determine the last writer. With this conflict resolution mechanism, all replicas agree on the latest update and converge toward a state in which they all have identical data.
- WatchDog 5y ago
  So a write than doesn’t “win” just gets silently discarded?
luhn 5y ago
Honestly your expectations are too high. Conflict resolution is row-level last-write-wins. It's not a globally distributed database, it's just a pile of regional DynamoDB tables duct taped together... They're not going to hire Aphyr for testing because there's nothing for him to test.

deanCommie 5y ago

If the developers are happy about the code and testing quality of a project, then you waited too long to ship. If the customers don't have any feedback or missed feature asks at launch, you waited too long to ship. You know who has great internal code and test quality? Google. Which is why Google doesn't ship. They're a wealth distribution charity for talented engineers. And their competitive advantage is that they lure talented people away from other companies where they might actually ship something and compete with Google, to instead park them, distract them with toys, beer kegs, readability reviews, and monorepo upgrades.

sam0x17 5y ago

It's a shame they don't open source it. It's funny too, being AWS they really don't have to worry about AWS running a cheaper service, so at that point why not open source it.

deleted 5y ago
[deleted]
nexuist 5y ago
They probably view it as a competitive advantage that Azure or GCP would try to copy if they figured out the "secret sauce."
- blowski 5y ago
  Azure has Cosmos which is arguably better than DynamoDB for a lot of use cases.
- staticassertion 5y ago
  I kinda doubt it. It's probably just that open sourcing it won't provide much utility (I bet lots of code is aws specific) and just adds a new maintenance burden for them.
  - bitexploder 5y ago
    The way you need to write code for a massively scalable service is just different. And the things you need to operate a service are also just different.
- sam0x17 5y ago
  Azure has Cosmos and Google has Datastore. They would never.
- pclmulqdq 5y ago
  Azure has a better product (Cosmos is damn good) and Google engineers have too much hubris to import something from lowly Amazon engineers.

digitalgangsta 5y ago

I've worked at 5 different tech companies now - this is par for course. And every single one, wished they could go back and do it again, but at that point the product was too successful so they ran with it.