18 ms·

DynamoDB 10 years later

252 points by mariuz 5y ago

tmitchel2 5y ago
DynamoDB for me is the perfect database for my serverless / graphql API. My only gripe is the limitation of items in a transaction of 25. I've had to resort to layering another transaction mgmt system on the top of it.
oriettaxx 5y ago
The only experience I had with dynamodb has been in aws: we set up a testing DB with defaults... then we left it there.... two months later we realized we've lost about 1500$: our mistake was to use the setup defaults (which had some sort of auto-scaling): I hope in the meanwhile AWS corrected this: we did let them know)
afandian 5y ago
We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for provisioning, considering node read/write ratios, data characteristics, etc. Basically, worrying about all the stuff you'd have to worry about with a traditional database. To be honest, I'd hoped that it could be a bit more 'magic', like S3, and it AWS would take care of provisioning, scaling, sharding etc. But it seemed disappointingly that you'd have to focus on proactively worrying about operations and provisioning. Is that sense correct? Is the dream of a self-managing, fire-and-forget key value database completely naive?
- aneil 5y ago
  Exactly my experience. I got sucked into using more than once, thinking it would be better next time, but there are just so many sharp edges. At one company, someone accidentally set the write rate rate high to transfer data into the db. This had the effect of permanently increasing the shard count to a huge number, basically making the DB useless.
- augustl 5y ago
  I have no direct experience with scaling DynamoDB in production, so take this with a grain of salt. But it seems to me that the on-demand scaling mode in DynamoDB has gotten _really_ good the last couple of years. For example, you used to have to manually set RCU/WCU to a high number when you expected a spike in traffic, since the ramp-up for on-demand scaling was pretty slow (could take up to 30 minutes). But these days, on-demand can handle spikes from 10s of requests a minute to 100s/1000s per second gracefully. The downside of on-demand is the pricing - it's more expensive if you have continuous load. But it can easily become _much_ cheaper if you have naturally spiky load patterns. Example: https://aws.amazon.com/blogs/database/running-spiky-workloads-and-optimizing-costs-by-more-than-90-using-amazon-dynamodb-on-demand-capacity-mode/ https://aws.amazon.com/blogs/database/running-spiky-workload...
  - PaywallBuster 5y ago
    Indeed We've some regular jobs that require scaling up dynamodb in advance few times per day, but then dynamo is only able to scale down 4x per day, so we're probably paying for over capacity unnecessarily (10x or more) for a couple hours a day Now we just moved ondemand and let them handle it, works fine
  - moduspol 5y ago
    > The downside of on-demand is the pricing - it's more expensive if you have continuous load. True, although you don't have to make that choice permanently. You can switch from provisioned to on demand once every 24 hours. And you can also set up application autoscaling in provisioned mode, which'll allow you to set parameters under which it'll scale your provisioned capacity up or down for you. This doesn't require any code and works pretty well if you can accept autoscaling adjustments being made in the timeframe of a minute or two.
    - PaywallBuster 5y ago
      scaling down is limited to 4x a day
      - 5y ago
- jpgvm 5y ago
  It is for now but it doesn't have to be. Dynamo's design isn't particularly amenable to dynamic and heterogenous shard topologies however. There could exist a fantasy database where you still tell it your hash and range keys, which are roughly how you tell the database which data isn't closely related to each other and which data is (and which you may want to scan) but instead of hard provisioning shard capacity it automagically splits shards when they hotspot and doesn't rely consistent hashing so that every shard can be sized differently depending on how hot it is. Right now such a database doesn't exist AFAICT as most places that need something the scales big enough also generally have the skill to avoid most of the pitfalls that cause problems on simple databases like Dynamo.
- eknkc 5y ago
  I believe it used to be static provisioning, you'd set the read and limit capacity beforehand. Then obviously there is autoscaling of those but it is still steps of capacity being provisioned. They now have a dynamic provisioning scheme, you simply don't care but it is more expensive so if you have predictible requirements it is still better to use static capacity provisioning. There is an option though. DynamoDB also requires the developer to know about its data storage model. While this is generally a good practice for any data storage solution, I feel like Dynamo requires a lot more careful planning. I also think that most of the best practices, articles etc apply to giant datasets with huge scale issues etc. If you are running a moderately active app, you probably can get away with a lot of stupid design decisions.
  - paulgb 5y ago
    My experience with dynamic provisioning has been that it is pretty inelastic, at least at the lower range of capacity. E.g. if you have a few read units and then try to export the data using AWS's cli client, you can pretty quickly hit the capacity limit and have to start the export over again. Last time, I ended up manually bumping the capacity way up, waiting a few minutes for the new capacity to kick in, and then exporting. Not what I had in mind when I wanted a serverless database!
    - moduspol 5y ago
      I understand it's not really your point, but if you're actually looking to export all the data from the table, they've got an API call you can give to have DynamoDB write the whole table to S3. This doesn't use any of your available capacity. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DataExport.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg... Beyond that, though, it's really not designed for that kind of use case.
      - paulgb 5y ago
        Ah, fair point. Somehow I didn't encounter that when I was trying to export, even though it existed at the time. But it would have solved my problem.
- k__ 5y ago
  After looking into solutions like Fauna, Upstash, and Planetscale I don't understand why anyone is bothering with DDB anymore. I read "the dynamodb book" and almost got a stroke. So much idiosyncrasies, for what?!
- manigandham 5y ago
  Plenty of options already exist. DynamoDB has both autoscaling and serverless modes. AWS also has managed Cassandra (runs on top of DynamoDB) which doesn't need instance management. Azure has CosmosDB, GCP has Cloud Datastore/Firestore, and there are many DB vendors like Planetscale (mysql), CockroachDB (postgres), FaunaDB (custom document/relational) that have "serverless" options.
- gonzo41 5y ago
  There's not really magic with s3, you still need to name things with coherrent prefixes to spread around the load. DynamoDB is almost simple enough to learn in a day. And if you're doing nothing with it, you're only really paying for storage. Good luck with your decisions.
  - jopsen 5y ago
    And S3 won't scale instantly... If your load is big enough :) Everything has limits, but S3 is remarkably hard to break if used right.
  - PaywallBuster 5y ago
    Prefixes are not needed 90% of use cases
    - brodouevencode 5y ago
      I'm not going to speculate on the accuracy of 90% value, but I will say that appropriately prefixed objects substantially help with performance when you have tons of small-ish files. Maybe most orgs don't have that need but in operational realms doing this with your logs make the response faster.
  - ralusek 5y ago
    S3 naming no longer matters for performance. Rejoice.
- redwood 5y ago
  Your example really summarizes the challenge with the AWS paradigm: namely that they want you to believe that the thing to do is to spread the the backend of your application across a large number of distinct data systems. No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable querying, and then they bolt on Redis to make the disjointed backend feel fast. And I'm just talking operational use cases; ignoring analytics here. Honestly it doesn't need to be these particular technologies but this is the general phenomenon you see in so many companies that adopt a relational database, key/value store (could be Cassandra instead of DynamoDB eg like what Netflix does), a search engine, and a caching layer because they think that that's the only option This inherently leads to a complexity debt explosion, fragmentation in the experience, and an operationally brittle posture that becomes very difficult to dig out of (this is probably why AWS loves the paradigm).
  - nine_k 5y ago
    If this is not the only option, what would you suggest instead? How to simplify it?
    - urthor 5y ago
      The alternative is to go to GCP and use the big GCP selling point, which is Big Table/Big Query. Those databases build most of that in, and it's all one fairly excellent distributed monolith.
      - onlyrealcuzzo 5y ago
        Wouldn't Spanner be closer to what you're talking about?
      - Keyframe 5y ago
        It's still a marriage.
  - ndm000 5y ago
    > they bolt it onto Postgres I am working with a company that is redesigning an enterprise transactional system, currently backed by an Oracle database with 3000 tables. It’s B2B so loads are predictable and are expected to grow no more than 10% per year. They want to use DynamoDB as their primary data store, with Postgres for edge cases it seems to me the opposite would be more beneficial. At what point does DynamoDB become a better choice than Postgres? I know that at certain scales Postgres breaks down, but what are those thresholds?
- snorkel 5y ago
  If you don't need data persistence then consider redis instead (which can also do persistence if you enable AOF)
- twodayrice 5y ago
  I think this is a good summary, and it even gets more complicated if you start using the DAX cache. Your read/write provisioning for DAX is totally different than the underlying dynamodb tables. The write throughput for Dax is limited by the size of the master node in the cluster. Can you say bottleneck?
- Marazan 5y ago
  DyanmoDB is pretty much the opposite of magic. It is a resource that can often be the right tool for the job but you really have to understand what the job is and carefully measure Dynamo up for what you are doing. It is _easy_ to misunderstand or miss something that would make Dynamo hideously expensive for your use case.
  - uberdru 5y ago
    What use cases would likely make it hideously expensive, in your view? Like, what are the red flags?
    - Marazan 5y ago
      Hot keys are the primary one. They destroys your "average" calculations for your throughput. Bulk loading data is the other gotcha I've run into. Had a beautiful use case for steady read performance of a batch dataset that was incredibly economical on Dynamo but the cost/time for loading the dataset into Dynamo was totally prohibitive. Basically Dynamo is great for constant read/write of very small, randomly distributed documents. Once you are out of thay zone things can hey dicey fast.
    - rmbyrro 5y ago
      Hot keys are much lesser of an issue nowadays. It'd been a big one in old DDB architectures. I'd say requiring scans or filters as opposed to queries is one of the biggest issues that can bite your pocket. Think carefully about how you'll access your data later. You won't be able to change it drastically and cheaply later.
- ignoramous 5y ago
  > We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. If possible, put the json in Workers KV, and access it through Cloudflare Workers. You can also optionally cache reads from Workers KV into Cloudflare's zonal caches. > To be honest, I'd hoped that it could be a bit more 'magic', like S3 You could opt to use the slightly more expensive DynamoDB On-Demand, or the free DynamoDB Auto-Scaling modes, which are relatively no-config. For a very ready-heavy workload, you'd probably want to add DynamoDB Accelerator (an write-through in-memory cache) in front of your tables. Or, use S3 itself (but a S3 bucket doesn't really like when you load it with a tonne of small files) accelerated by CloudFront (which is what AWS Hyperplane, tech underpinning ALB and NLB, does: https://aws.amazon.com/builders-library/reliability-and-constant-work/ https://aws.amazon.com/builders-library/reliability-and-cons...) S3, much like DynamoDB, is a KV store: https://news.ycombinator.com/item?id=11161667 https://news.ycombinator.com/item?id=11161667 and https://www.allthingsdistributed.com/2009/03/keeping_your_database_simple_and_fast.html https://www.allthingsdistributed.com/2009/03/keeping_your_da...
- zurn 5y ago
  Your impressions are cordect: DynamoDB is quite low-level and more like a DB kit than ready to use DB, for most applications it's better to use something else.
- lkrubner 5y ago
  Exactly. This has been my experience with several AWS technologies. Like with their ElasticSearch service, where I had to constantly fine-tune various parameters, such as memory. I was curious why they couldn't auto-scale the memory, why I had to do that manually. There are several AWS services that should be a bit more magical, but they are not.
- jugg1es 5y ago
  I do not recommend starting off with a decision to use DynamoDB before you have worked with it directly for some time to understand it. You could spend months trying to shoehorn your use case into it before realizing you made a mistake. That said, DynamoDB can be incredibly powerful and inexpensive tool if used right.
  - rmbyrro 5y ago
    I think this can be said about any technology, really...
    - jugg1es 5y ago
      Yea, probably, but it is especially true for DynamoDB because it can initially appear as though your use cases are all supported but that is only because you haven't internalized how it works yet. By the time you realize you made a mistake, you are way too far in the weeds and have to start over from scratch. I would venture that more than 50% of DynamoDB users have had this happen to them early on. Anecdotally, just look at the comments on this post. There are so many horror stories with DynamoDB, but they're basically all people who decided to use it before they really understood it.
- restlake 5y ago
  If you know and understand S3 pretty well, and you purely need to generate, store, and read materialized static views, I highly recommend S3 for this use case. I say this as someone who really likes working with DDB daily and understands the tradeoffs with Dynamo. You can always layer on Athena or (simpler) S3 Select later if a SQL query model is a better fit than KV object lookups. S3 is loosely the fire and forget KV DB you’re describing IMO depending on your use case
- whalesalad 5y ago
  Dynamo is incredibly hard to use correctly I’d urge you to start writing a prototype, a lot of your assumptions might get thrown out the window. Dynamo is not necessarily good for reading high volume. You’ll end up needing to use a parallel scan approach which is not fast.
  - qvrjuec 5y ago
    I'd say Dynamo is extremely good at reading high volume, with the appropriate access pattern. It's very efficient at retrieving huge amounts of well partitioned data using the data's keys, but scanning isn't so efficient.
    - whalesalad 5y ago
      You can only ever fetch 1MB of data at a time though, even when using the more efficient query method (as opposed to scan). If your individual entities are not very tiny, it is hard to get for instance 2M items back in a reasonable amount of time.
  - mythrwy 5y ago
    Also can be _very_ expensive if you do not use it correctly.
- qaq 5y ago
  Save yourself a ton of pain and don't use DynamoDB
- _pdp_ 5y ago
  DynamoDB is like S3 but with query features. It is not a relational db. It is a document storage. So you need to use it for what it is. Our entire solution is basically based on top of lambda and dynamodb tables and it works really as long as you don't threat the tables like SQL.
- Tehnix 5y ago
  A lot of things that used to be a concern (hot partitions, etc) are not a concern anymore and most have been solved these days :) Put it on on-demand pricing (it'll be better and cheaper for you most likely), and it will handle any load you throw at it. Can you get it to throttle? Sure, if you absolutely blast it without ever having had that high of a need before (and it can actually be avoided[0]). You will need to understand how to model things for the NoSQL paradigm that DynamoDB uses, but that's a question of familiarity and not much else (you didn't magically know SQL either). My experience comes from scaling DynamoDB in production for several years, handling both massive IoT data ingestion in it as well as the user data as well. We were able to replace all things we thought we would need a relational database for, completely. My comparison between a traditional RDS setup: - DynamoDB issues? 0. Seriously. Only thing you need to monitor is billing. - RDS? Oh boy, need to provision for peak capacity, need to monitor replica lags, need to monitor the Replicas themselves, constant monitoring and scaling of IOPS, suddenly queries get slow as data increases, worrying about indexes and the data size, and much more... [0]: https://theburningmonk.com/2019/03/understanding-the-scaling-behaviour-of-dynamodb-ondemand-tables/ https://theburningmonk.com/2019/03/understanding-the-scaling...
- pbalau 5y ago
  Thinking like this both baffles me, but also makes me happy because there will always be a need for people like me, infra. AWS is not a magical tool that will replace your infra team, it is a magical tool that will allow your infra team to do more. I am the infra team of my startup and I estimate that only 50% of my time is doing infra work. The rest is supporting my peers, work in frameworky stuff, solve dev efficiency issues bla bla. Lets say that you operate in an AWS-less environment, with everything bare metal, in a datacenter. Your GOOD infra team has to do the following: Hardware: - make sure there is a channel to get new hardware, both for capacity increase and spares. What are you going to do? Buy 1 server and 2 spares? If one of the servers has an issue, isn't it quite likely that the other servers, from the same batch, to have the same issue? Is this affecting you, or not? Where do you store the spares? In a warehouse somewhere, making it harder to deploy? In the rack with the one in use, wasting rackspace/switch space? Are you going to rely on the datacenter to provide you with the hardware? What if you are one of their smaller customers and your requests get pushed back because some larger customer requests get higher priority? - make sure there is a way to deploy said hardware. You don't want to not be able to deploy a new server because there is no space in the rack, or no space in the switch. Where are your spares? In a warehouse miles away from the datacenter? Do you have access to said warehouse at midnight, on Thanksgiving? Oh shit, someone lost the key to your rack! Oh noes, we don't have any spare network cable/connectors/screws... Software: - did you patch your servers? did you patch your switches? - new server, we need to install the os. And a base set of software, including the agent we use to remote manage the server. - oh, we also need to run and maintain the management infra, say the control plane for k8. - oh, we want some read replicas for this db, not only we need the hardware to run the replicas on (and see above for what that means), now you need to add a bunch of monitoring and have plans in place to handle things like: replicas lagging, network links between master and replicas being full, failover for the above, master crapping out yada yada. I bet there are many other aspects I'm missing. Choices: Your GOOD infra team will have to decide things like: how many spares do we need, is the capacity we have atm enough for the launch of our next world-changing feature that half the internet wants to use? Are we lucky enough to survive a few months without spares or should we get estra capacity in another datacenter? Do we want to have replicas on the west coast or is the latency acceptable? These are the main areas of what an infra team is supposed to do: Hardware, Software and Choices. AWS (and most other cloud providers) is making the first 2 points non issues. For the last area you can do 2 things: get an infra team (could be a full fledged team, could be 1 person, you could do it) and teoretically you will get choices tailored to what your business needs OR let AWS do it for you. *AWS might make these choices based on a metric you disagree with and this is the main reason people complain*.
- phamilton 5y ago
  I don't know your scaling needs, but I would highly recommend just using Aurora postgresql for read-only workloads. We have some workloads that are essentially K/V store lookups that were previously slated for dynamodb. On an Aurora cluster of 3*r6g.xlarge we easily handle 25k qps with p99 in the single-digit ms range. Aurora can scale up to 15 instances and up to 24xlarge, so it would not be unreasonable to see 100x the read workload with similar latencies. Happy to talk more. We're actively moving a bunch of workloads away from DynamoDB and to Aurora so this is fresh on our minds.
  - afandian 5y ago
    Thanks, that's what I hope will work. I might drop you a mail at some point.
- giaour 5y ago
  If you use the "pay per request" billing model instead of provisioned throughput, DynamoDB scaling is self-managing, and you can treat your DB as a fire-and-forget key/value store. You need to plan how you'll query your data and structure the keys accordingly, but honestly, that applies even more to S3 than it does to Dynamo.
- AtlasBarfed 5y ago
  The salespeople always promise magic and handwave CAP away. But data at scale is about: 1) knowing your queries ahead of time (since you've presumably reached the limit of PG/maybesql/o-rackle. 2) dealing with CAP at the application level: distributed transactions, eventual consistency, network partitions. 3) dealing with a lot more operational complexity, not less. So if the snake oil salesmen say it will be seamless, they are very very very much lying. Either that, or you are paying a LOT of money for other people to do the hard work. Which is what happens with managing your own NoSQL vs DynamoDB. You'll pay through the roof for DynamoDB at true big data scales.
- nesarkvechnep 5y ago
  If it’s possible in your situation, instead of vendor lock-in, invest in cacheability of your service and leverage HTTP cache as much as possible.
- jedberg 5y ago
  > Is the dream of a self-managing, fire-and-forget key value database completely naive? It's not, if you plan it right. Learn about single table design for DynamoDB before you start. There are a lot of good resources from Amazon and the community. Here is a very accessible video from the community: https://www.youtube.com/watch?v=BnDKD_Zv0og https://www.youtube.com/watch?v=BnDKD_Zv0og Here is a video from Rick Houlihan, a senior leader from AWS who basically helps companies convert to single table design: https://www.youtube.com/watch?v=KYy8X8t4MB8 https://www.youtube.com/watch?v=KYy8X8t4MB8 And a good book on the topic: https://www.dynamodbbook.com https://www.dynamodbbook.com If you use single table design, you can turn on all of the auto-tuning features of DynamoDB and they will work as expected and get better and more efficient with more data. Some people worry that this breaks the cardinal rule of microservices: One database per service. But the actual rule is never have one service directly access the data of another, always use the API. So as long as your services use different keyspaces and never access each other's data, it can still work (but does require extra discipline).
- amzn-throw 5y ago
  The key benefit with DDB is predictability: https://brooker.co.za/blog/2022/01/19/predictability.html https://brooker.co.za/blog/2022/01/19/predictability.html Yes, you have to learn about all these things upfront. But once you figure it out, test it, and configure it - it will work as you expect. No surprises. Whereas Relational Databases work until they don't. A developer makes a tiny (even a no-op) change to a query or stored procedure, a different SQL plan gets chosen, and suddenly your performance/latency dramatically reduces, and you have no easy way to roll it back through source control/deployment pipelines. You have to page a DBA who has to go pull up the hood. With services like DDB, you maintain control.
- stickfigure 5y ago
  Take a look at Firestore / Google Cloud Datastore. It's pretty much exactly what you describe - fire and forget. There's no concept of "node" (at least not from the outside).
- emodendroket 5y ago
  Dynamo is like the opposite of fire and forget it. You really want to know your access patterns at design time.
victor106 5y ago
We tried to implement an application on DynamoDB about 2 years ago. We really struggled with implementing adhoc queries/search. For e.g:- select * from employees where name = X and city = Y. Any improvements in DynamoDB that make it easier to implement such queries?
- garydevenay 5y ago
  The most reliable way to build a system with DynamoDB is to plan queries upfront. Trying to use it like a SQL database and make Ad-Hoc queries won't work because it's not a SQL DB. Data should be stored in the fashion you wish for it to be read, and storing the same data in more than one configuration is acceptable. Good resource: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-general-nosql-design.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
- jon-wood 5y ago
  This will sound flippant, but that's not what Dynamo is for. If you want to do freeform relational queries like that then put it in a relational database. Dynamo is primarily designed for high volume storage/querying on well understood data sets with a few query patterns. If you want to be able to query information on employees based on their name and city you'll need to build another index keyed on name and city (in practice Dynamo makes that reasonably simple by adding a secondary index).
  - owenmarshall 5y ago
    Alternatively, practice single table design: structure your table keys in such a way that they can represent all (or at least most) of the queries you need to run. This is often easier said than done, but it can be far less expensive and more performant than adding an index for each search.
    - ellimilial 5y ago
      It's always great fun compounding new, manual 'indexes' when you discover you need another query.
gigatexal 5y ago
If you follow Rick Houlihan (@houlihan_rick) then all the accolades that AWS for DynamoDB pale in comparison to its current team and execution in that the company seems to not be investing in it so much so that Rick left to join MongoDB.
- vslira 5y ago
  Man I love Rick’s talks as much as anyone but let’s be real, he likely left AWS not for his love of first class geographical indexes but because Mongo offered a giant pile of money for him to evangelize their tech. Though I have no doubts that he actually had a lot of reservations around Dynamo’s DX before, he likely has some around mongodb but those won’t be the bulk of his content
  - gigatexal 5y ago
    At his rank at AWS I don’t know if money was such an issue. He strikes me as a person who cares deeply about the underlying tech. But I have no idea one way or the other.
    - awsthro00945 5y ago
      I think I've seen you post something similar on r/aws about how Rick was "top DynamoDb person at AWS" (apologies if that wasn't you). I think you are overestimating Rick's "rank". I just looked him up (I had not heard of him before seeing his name mentioned on r/aws a few days ago) and he was an L7 TPM/Practice Manager in AWS's sales organization. That's not really a notably high position, and in the grand scheme of Amazon pay scales, isn't that high up. An L7 TPM gets paid about the same as, or sometimes less than, an L6 software dev (L6 is "senior", which is ~5-10 years of experience). Also, him being in the sales org means he had practically nothing to do with the engineering of the service. AWS Sales is a revolving door of people. I mean no offense towards Rick (again, I didn't know him or even know of him before I read his name in a comment a few days ago), but I would not read anything at all into the fact that an L7 Sales TPM left for another company.
tybit 5y ago
For anyone else expecting this to be a paper given the domain name, it’s not. It’s a non technical interview with a couple of the original papers authors. Not bad, just not as exciting as I imagine a paper detailing what they’ve learnt from a distributed systems perspective etc operating Dynamo then DynamoDB for so long now.
- uvdn7 5y ago
  https://brooker.co.za/blog/2022/01/19/predictability.html https://brooker.co.za/blog/2022/01/19/predictability.html This might be something you are looking for.
- mjb 5y ago
  We don't have a paper on DynamoDB's internals (yet?), but here's a talk you might find interesting from one of the folks who built and ran DDB for a long time: https://www.youtube.com/watch?v=yvBR71D0nAQ https://www.youtube.com/watch?v=yvBR71D0nAQ And Doug Terry talking through the details of how DynamoDB's transaction protocol works: https://www.usenix.org/conference/fast19/presentation/terry https://www.usenix.org/conference/fast19/presentation/terry If we did publish more about the internals of DDB, what would you be looking to learn? Architecture? Operational experience? Developer experience? There's a lot of material we could share, and it's useful to hear where people would like us to focus.
  - pow_pp_-1_v 5y ago
    All of it - architecture, operational experience, best practices etc.
    - ldrndll 5y ago
      Just want to second this. All of the above sounds really interesting to me!
minaor 5y ago
[flagged]
jugg1es 5y ago
Every system I've built on DynamoDB just works. The APIs that use it have had virtually 100% uptime and have never needed database maintenance. It is not a replacement for a RDBMS, but for some use cases, it's a killer service.
graderjs 5y ago
The in-the-trenches, technical, battle-tested/battle-scarred comments on this thread are why I come to HN for the comments.
manishsharan 5y ago
Quite a few of the teams that were early adopters of AWS DynamoDB were not prepared for the pricing nuances that had to be taken into consideration when building their solutions.
- xtracto 5y ago
  I remember trying Dynamodb around 2015/2016: You had to specify your expected read and write throughout and you would be billed for that. At that time we had a pretty spikey traffic use case which made using dynamodb efficiently impossible
  - musingsole 5y ago
    I had a similar experience, but ultimately wrote a service to monitor our workloads and request increased provisioning during spikes. You could reduce your provisioning like 10 times a day, but after that you could only increase it and would be stuck with the higher rate for a time. And then on-demand provisioning was released and it was cheap enough to be worth simplifying our workflows.
- jvanvleet 5y ago
  I was one of these. However I now understand that the pricing nuances reflected a reality that I appreciate. We used DDB in a way that was not the best fit and the cost was a reflection of this.
salil999 5y ago
I can safely say that the team members working in DynamoDB are very skilled and they care deeply about the product. They really work hard and think of interesting solutions to a lot of problems that their biggest customers face which is great from a product standpoint. There are some pretty smart people working there. Engineering, however, was a disaster story. Code is horribly written and very few tests are maintained to make sure deployments go without issues. There was too much emphasis on deployment and getting fixes/features out over making sure it won't break anything else. It was a common scenario to release a new feature and put duct tape all around it to make sure it "works". And way too many operational issues. There are a lot of ways to break DynamoDB :) Overall, though, the product is very solid and it's one of the few database that you can say "just works" when it comes to scalability and reliability (as most AWS services are) I worked at DynamoDB for over 2 years.
- vasili111 5y ago
  >Engineering, however, was a disaster story. Code is horribly written and very few tests are maintained to make sure deployments go without issues. There was too much emphasis on deployment and getting fixes/features out over making sure it won't break anything else. It was a common scenario to release a new feature and put duct tape all around it to make sure it "works". And way too many operational issues. There are a lot of ways to break DynamoDB :) >Overall, though, the product is very solid and it's one of the few database that you can say "just works" when it comes to scalability and reliability (as most AWS services are) How those two can coexist?
  - badhombres 5y ago
    I mean eventually enough duct tape can be solid like a tank :)
    - jeffreygoesto 5y ago
      Or a bridge... https://www.popularmechanics.com/science/a5732/mythbusters-build-duct-tape-bridge/ https://www.popularmechanics.com/science/a5732/mythbusters-b...
dabfiend19 5y ago
knocking off mongo... 10 years later they still haven't caught up.
artembugara 5y ago
We realized how great Dynamo was only after we migrated off AWS. Dynamo was a key factor to us when we were releasing the MVP of our News API [0]. We used Dynamo, ElasticSearch, Lambda and could make it running in 60 days while being full-time employed. Also, the best tech talk I saw was given by Rick Houlihan on re:Invent [1] I highly recommend every engineer to watch it: it's a great overview of SQL vs NoSQL [0] https://newscatcherapi.com/blog/how-we-built-a-news-api-beta-in-60-days https://newscatcherapi.com/blog/how-we-built-a-news-api-beta... [1] https://www.youtube.com/watch?v=HaEPXoXVf2k https://www.youtube.com/watch?v=HaEPXoXVf2k
- pier25 5y ago
  BTW Rick Houlihan left AWS recently to work for Mongo. https://twitter.com/houlihan_rick/status/1472969503575265283 https://twitter.com/houlihan_rick/status/1472969503575265283 On that thread he criticizes AWS regarding DynamoDB openly. > I will always love DynamoDB, but the fact is it is losing ground fast because AWS focuses most of their resources on the half baked #builtfornopurpose database strategy. I always hated that idea, I just bit my tongue instead of saying it. > The problem is the other half-baked database services that all compete for the same business. DocumentDB, Keyspaces, Timestream, Neptune, etc. Databases take decades to optimize, the idea that you can pump them out like web apps is silly. > I was very tired of explaining over and over again that DynamoDB is actually not the dumbed down Key-Value store that the marketing message implied. When AWS created 6 different NoSQL databases they had to make up reasons for each one and the messaging makes no sense.
  - throwdbaaway 5y ago
    Interesting. MongoDB actually came to mind while I was reading the other comment here: > No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable querying, and then they bolt on Redis to make the disjointed backend feel fast. And I'm just talking operational use cases; ignoring analytics here. Perhaps MongoDB is prime for a comeback?
cmollis 5y ago
we use DynamoDB like a big hash table of s3 file locations.. we look up these locations via a key (at the time, it sounded like a pretty good use-case for it). I suppose we could have used some other managed redis or memcached thing, but being an AWS shop, it was, and is, pretty useful. I have to say, it's been pretty effortless to configure.. read/write units are really the only thing we've had to configure (other than the base index) The rest of it has been easy. It has about a 100 million entries that are read or written pretty quickly.
- Jach 5y ago
  I remember talking to someone who was playing with AWS stuff for the first time and they had a similar architecture, using Dynamo for a lookup store. It still seems a bit odd to me though. It's been a long time since I've worked with the S3 API, so maybe it just doesn't support the same sort of thing, but wouldn't it be nicer to just query S3 with some key and get back either the path/URL to render a link, or the content itself? Why the Dynamo intermediary? (And on the other side, if you don't need to render a link to serve the content, why not use Dynamo as the actual document store and skip S3? Storage cost?)
  - cmollis 5y ago
    You're not going to get O(1) lookup using the S3 API. The APIs are basically O(n). So we we use DynamoDB to store the S3 URL.
    - cmollis 5y ago
      Moreover, storing a large binary file (in this case) in Dynamo is probably not the best use-case for it..most likely would have to convert to base64 in and out of it.
deleted 5y ago
[deleted]
didip 5y ago
To be honest, as a customer, it is hard for me to justify using DynamoDB. Some of this criticism can be out of date: 1. DynamoDB is not as convenient. There are a bit too many dials to turn. 2. DynamoDB does not have a SQL facade on top. 3. DynamoDB is proprietary, I believe there's no OSS API equivalent if you want to migrate out. 4. DynamoDB was kind of expensive. But it has been a while since I last check the pricing page. It's simply much better to start with PostgreSQL Aurora and move to a more scalable storage based on specific uses-cases later. For example: Cassandra, Elastic, Druid, or CockroachDB.
- nathanfig 5y ago
  That’s surprising to me, I consider DynamoDB to be far simpler than any relational DB including the alternatives you list
  - deleted 5y ago
    [deleted]
- arecurrence 5y ago
  I strongly agree that most early stage businesses should be on Postgres. There's simply too much churn in early stage data models. Also, unforeseen esoteric needs jumping out of the wood work that you can knock out a SQL query for instead of having to build a solution come up constantly. However, this does assume that your development team has a competent understanding of SQL. I've been in a couple startups that went Dynamo first and development velocity was a pale shadow of velocity with Postgres. When one of those startups dumped dynamo for Postgres velocity multiplied immediately. I'd estimate we were moving at around 1000% and the complete transition took less time than even I expected (about a month). Once the business matures, moving tables onto dynamo and wrapping them in a microservice makes a lot of sense. Dynamo does solve a lot of problems that become increasingly material as the business evolves. Eventually, SQL's presence declines and transitions into an analytics system as narrower, but easier for ops, options proliferate.
- Zimnx 5y ago
  Ad 3. Lot of people don't know about it, but there's a open source, free, and DynamoDB compatibile databse called ScyllaDB - API it's called Alternator to be specific.
nathanfig 5y ago
I wonder if DynamoDB would be met with less criticism had it simply been named Dynamo Document Store.
ddoolin 5y ago
As a developer, I really have a love-hate relationship with Dynamo. I love how fast and easy it is to setup and get rolling. The partitioning scheme came off as confusing and opaque but I think that says more about Amazon's documentation than the scheme itself. I do not like that there's no really third party tooling integration to be able to query. Their UI in the console is _so freaking terrible_ yet you have no other way than code to query it. This problem is so bad that I will avoid using it where I can despite it being a good option, performance-wise.
- ejb999 5y ago
  My experience exactly.
- rmbyrro 5y ago
  Don't think this is a fair reason to avoid DynamoDB. There are reasonably better alternatives: - DynamoDB Workbench (free, AWS official): https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/workbench.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg... - Dynobase (paid, third-party): https://dynobase.dev/ https://dynobase.dev/
  - belter 5y ago
    Plus there is the local install ( Only for Dev and Testing purposes...) "Deploying DynamoDB Locally on Your Computer" https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.DownloadingAndRunning.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
- deleted 5y ago
  [deleted]
- wnolens 5y ago
  Yes the console UI is mind numbing, and was just made 10x worse in a recent redesign. But i like the python boto3 library https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html https://boto3.amazonaws.com/v1/documentation/api/latest/guid... Build yourself a few wrappers to make querying more convenient, and i query straight from a python repl pretty effectively
0xbadcafebee 5y ago
I haven't run into anyone who uses Dynamo for anything other than managing Terraform backend state locking. And I think that's still the best use case for it: you just want to store a couple random key-values somewhere and have more functionality than AWS Parameter Store. Trying to build anything large-scale with it will probably leave you wanting.
sakopov 5y ago
We landed on DynamoDB when we migrated a monolith to microservice architecture. I have to say that DynamoDB fits fairly well in the microservices world where the service scope is small, query patterns are pretty narrow and don't really change much. Building new things using DynamoDB when query patterns aren't necessarily known is very painful and require tedious migration strategies unless you don't mind paying for GSIs.
anonymousDan 5y ago
Can anyone recommend me a good paper (or other resource) describing the designs of DynamoDB and S3? Ideally something in the spirt of the original Dynamo paper (i.e. https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf https://www.allthingsdistributed.com/files/amazon-dynamo-sos... )
- schwarzmx 5y ago
  This one is pretty good for DynamoDB: https://youtu.be/yvBR71D0nAQ https://youtu.be/yvBR71D0nAQ
balls187 5y ago
DynamoDB is one of my favorite AWS products.
thekozmo 5y ago
Indeed quite a journey. If you love DynamoDB and like open source, give Scylla a try (disclosure me==founder): https://www.scylladb.com/alternator/ https://www.scylladb.com/alternator/
whoisjohnkid 5y ago
I absolutely love DynamoDB. Would definitely recommend the DynamoDB Book by Alex Debrie and Advanced DynamoDB Patterns by Rick H on YouTube. Once you understand how to properly use dynamo and what it’s good at it you get so much power; at a fraction of the cost depending on your workload. Was a breeze to setup multi region applications that utilize the single table design strategy. If you ever need complex queries just use dynamo streams to power whatever search solution you are comfortable with.
robertlagrant 5y ago
> Customers no longer want to just store and query the data in their databases. They then want to analyze that data to create value Presumably create more value. I know it's a marketing post but still - storing and retrieving things is pretty valuable in and of itself.

252 points by mariuz 5y ago

DynamoDB for me is the perfect database for my serverless / graphql API. My only gripe is the limitation of items in a transaction of 25. I've had to resort to layering another transaction mgmt system on the top of it.

oriettaxx 5y ago

The only experience I had with dynamodb has been in aws: we set up a testing DB with defaults... then we left it there.... two months later we realized we've lost about 1500$: our mistake was to use the setup defaults (which had some sort of auto-scaling): I hope in the meanwhile AWS corrected this: we did let them know)

afandian 5y ago

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for provisioning, considering node read/write ratios, data characteristics, etc. Basically, worrying about all the stuff you'd have to worry about with a traditional database. To be honest, I'd hoped that it could be a bit more 'magic', like S3, and it AWS would take care of provisioning, scaling, sharding etc. But it seemed disappointingly that you'd have to focus on proactively worrying about operations and provisioning. Is that sense correct? Is the dream of a self-managing, fire-and-forget key value database completely naive?

aneil 5y ago
Exactly my experience. I got sucked into using more than once, thinking it would be better next time, but there are just so many sharp edges. At one company, someone accidentally set the write rate rate high to transfer data into the db. This had the effect of permanently increasing the shard count to a huge number, basically making the DB useless.
augustl 5y ago
I have no direct experience with scaling DynamoDB in production, so take this with a grain of salt. But it seems to me that the on-demand scaling mode in DynamoDB has gotten _really_ good the last couple of years. For example, you used to have to manually set RCU/WCU to a high number when you expected a spike in traffic, since the ramp-up for on-demand scaling was pretty slow (could take up to 30 minutes). But these days, on-demand can handle spikes from 10s of requests a minute to 100s/1000s per second gracefully. The downside of on-demand is the pricing - it's more expensive if you have continuous load. But it can easily become _much_ cheaper if you have naturally spiky load patterns. Example: https://aws.amazon.com/blogs/database/running-spiky-workloads-and-optimizing-costs-by-more-than-90-using-amazon-dynamodb-on-demand-capacity-mode/ https://aws.amazon.com/blogs/database/running-spiky-workload...
- PaywallBuster 5y ago
  Indeed We've some regular jobs that require scaling up dynamodb in advance few times per day, but then dynamo is only able to scale down 4x per day, so we're probably paying for over capacity unnecessarily (10x or more) for a couple hours a day Now we just moved ondemand and let them handle it, works fine
- moduspol 5y ago
  > The downside of on-demand is the pricing - it's more expensive if you have continuous load. True, although you don't have to make that choice permanently. You can switch from provisioned to on demand once every 24 hours. And you can also set up application autoscaling in provisioned mode, which'll allow you to set parameters under which it'll scale your provisioned capacity up or down for you. This doesn't require any code and works pretty well if you can accept autoscaling adjustments being made in the timeframe of a minute or two.
  - PaywallBuster 5y ago
    scaling down is limited to 4x a day
    - 5y ago
jpgvm 5y ago
It is for now but it doesn't have to be. Dynamo's design isn't particularly amenable to dynamic and heterogenous shard topologies however. There could exist a fantasy database where you still tell it your hash and range keys, which are roughly how you tell the database which data isn't closely related to each other and which data is (and which you may want to scan) but instead of hard provisioning shard capacity it automagically splits shards when they hotspot and doesn't rely consistent hashing so that every shard can be sized differently depending on how hot it is. Right now such a database doesn't exist AFAICT as most places that need something the scales big enough also generally have the skill to avoid most of the pitfalls that cause problems on simple databases like Dynamo.
eknkc 5y ago
I believe it used to be static provisioning, you'd set the read and limit capacity beforehand. Then obviously there is autoscaling of those but it is still steps of capacity being provisioned. They now have a dynamic provisioning scheme, you simply don't care but it is more expensive so if you have predictible requirements it is still better to use static capacity provisioning. There is an option though. DynamoDB also requires the developer to know about its data storage model. While this is generally a good practice for any data storage solution, I feel like Dynamo requires a lot more careful planning. I also think that most of the best practices, articles etc apply to giant datasets with huge scale issues etc. If you are running a moderately active app, you probably can get away with a lot of stupid design decisions.
- paulgb 5y ago
  My experience with dynamic provisioning has been that it is pretty inelastic, at least at the lower range of capacity. E.g. if you have a few read units and then try to export the data using AWS's cli client, you can pretty quickly hit the capacity limit and have to start the export over again. Last time, I ended up manually bumping the capacity way up, waiting a few minutes for the new capacity to kick in, and then exporting. Not what I had in mind when I wanted a serverless database!
  - moduspol 5y ago
    I understand it's not really your point, but if you're actually looking to export all the data from the table, they've got an API call you can give to have DynamoDB write the whole table to S3. This doesn't use any of your available capacity. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DataExport.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg... Beyond that, though, it's really not designed for that kind of use case.
    - paulgb 5y ago
      Ah, fair point. Somehow I didn't encounter that when I was trying to export, even though it existed at the time. But it would have solved my problem.
k__ 5y ago
After looking into solutions like Fauna, Upstash, and Planetscale I don't understand why anyone is bothering with DDB anymore. I read "the dynamodb book" and almost got a stroke. So much idiosyncrasies, for what?!
manigandham 5y ago
Plenty of options already exist. DynamoDB has both autoscaling and serverless modes. AWS also has managed Cassandra (runs on top of DynamoDB) which doesn't need instance management. Azure has CosmosDB, GCP has Cloud Datastore/Firestore, and there are many DB vendors like Planetscale (mysql), CockroachDB (postgres), FaunaDB (custom document/relational) that have "serverless" options.
gonzo41 5y ago
There's not really magic with s3, you still need to name things with coherrent prefixes to spread around the load. DynamoDB is almost simple enough to learn in a day. And if you're doing nothing with it, you're only really paying for storage. Good luck with your decisions.
- jopsen 5y ago
  And S3 won't scale instantly... If your load is big enough :) Everything has limits, but S3 is remarkably hard to break if used right.
- PaywallBuster 5y ago
  Prefixes are not needed 90% of use cases
  - brodouevencode 5y ago
    I'm not going to speculate on the accuracy of 90% value, but I will say that appropriately prefixed objects substantially help with performance when you have tons of small-ish files. Maybe most orgs don't have that need but in operational realms doing this with your logs make the response faster.
- ralusek 5y ago
  S3 naming no longer matters for performance. Rejoice.
redwood 5y ago
Your example really summarizes the challenge with the AWS paradigm: namely that they want you to believe that the thing to do is to spread the the backend of your application across a large number of distinct data systems. No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable querying, and then they bolt on Redis to make the disjointed backend feel fast. And I'm just talking operational use cases; ignoring analytics here. Honestly it doesn't need to be these particular technologies but this is the general phenomenon you see in so many companies that adopt a relational database, key/value store (could be Cassandra instead of DynamoDB eg like what Netflix does), a search engine, and a caching layer because they think that that's the only option This inherently leads to a complexity debt explosion, fragmentation in the experience, and an operationally brittle posture that becomes very difficult to dig out of (this is probably why AWS loves the paradigm).
- nine_k 5y ago
  If this is not the only option, what would you suggest instead? How to simplify it?
  - urthor 5y ago
    The alternative is to go to GCP and use the big GCP selling point, which is Big Table/Big Query. Those databases build most of that in, and it's all one fairly excellent distributed monolith.
    - onlyrealcuzzo 5y ago
      Wouldn't Spanner be closer to what you're talking about?
    - Keyframe 5y ago
      It's still a marriage.
- ndm000 5y ago
  > they bolt it onto Postgres I am working with a company that is redesigning an enterprise transactional system, currently backed by an Oracle database with 3000 tables. It’s B2B so loads are predictable and are expected to grow no more than 10% per year. They want to use DynamoDB as their primary data store, with Postgres for edge cases it seems to me the opposite would be more beneficial. At what point does DynamoDB become a better choice than Postgres? I know that at certain scales Postgres breaks down, but what are those thresholds?
snorkel 5y ago
If you don't need data persistence then consider redis instead (which can also do persistence if you enable AOF)
twodayrice 5y ago
I think this is a good summary, and it even gets more complicated if you start using the DAX cache. Your read/write provisioning for DAX is totally different than the underlying dynamodb tables. The write throughput for Dax is limited by the size of the master node in the cluster. Can you say bottleneck?
Marazan 5y ago
DyanmoDB is pretty much the opposite of magic. It is a resource that can often be the right tool for the job but you really have to understand what the job is and carefully measure Dynamo up for what you are doing. It is _easy_ to misunderstand or miss something that would make Dynamo hideously expensive for your use case.
- uberdru 5y ago
  What use cases would likely make it hideously expensive, in your view? Like, what are the red flags?
  - Marazan 5y ago
    Hot keys are the primary one. They destroys your "average" calculations for your throughput. Bulk loading data is the other gotcha I've run into. Had a beautiful use case for steady read performance of a batch dataset that was incredibly economical on Dynamo but the cost/time for loading the dataset into Dynamo was totally prohibitive. Basically Dynamo is great for constant read/write of very small, randomly distributed documents. Once you are out of thay zone things can hey dicey fast.
  - rmbyrro 5y ago
    Hot keys are much lesser of an issue nowadays. It'd been a big one in old DDB architectures. I'd say requiring scans or filters as opposed to queries is one of the biggest issues that can bite your pocket. Think carefully about how you'll access your data later. You won't be able to change it drastically and cheaply later.
ignoramous 5y ago
> We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. If possible, put the json in Workers KV, and access it through Cloudflare Workers. You can also optionally cache reads from Workers KV into Cloudflare's zonal caches. > To be honest, I'd hoped that it could be a bit more 'magic', like S3 You could opt to use the slightly more expensive DynamoDB On-Demand, or the free DynamoDB Auto-Scaling modes, which are relatively no-config. For a very ready-heavy workload, you'd probably want to add DynamoDB Accelerator (an write-through in-memory cache) in front of your tables. Or, use S3 itself (but a S3 bucket doesn't really like when you load it with a tonne of small files) accelerated by CloudFront (which is what AWS Hyperplane, tech underpinning ALB and NLB, does: https://aws.amazon.com/builders-library/reliability-and-constant-work/ https://aws.amazon.com/builders-library/reliability-and-cons...) S3, much like DynamoDB, is a KV store: https://news.ycombinator.com/item?id=11161667 https://news.ycombinator.com/item?id=11161667 and https://www.allthingsdistributed.com/2009/03/keeping_your_database_simple_and_fast.html https://www.allthingsdistributed.com/2009/03/keeping_your_da...
zurn 5y ago
Your impressions are cordect: DynamoDB is quite low-level and more like a DB kit than ready to use DB, for most applications it's better to use something else.
lkrubner 5y ago
Exactly. This has been my experience with several AWS technologies. Like with their ElasticSearch service, where I had to constantly fine-tune various parameters, such as memory. I was curious why they couldn't auto-scale the memory, why I had to do that manually. There are several AWS services that should be a bit more magical, but they are not.
jugg1es 5y ago
I do not recommend starting off with a decision to use DynamoDB before you have worked with it directly for some time to understand it. You could spend months trying to shoehorn your use case into it before realizing you made a mistake. That said, DynamoDB can be incredibly powerful and inexpensive tool if used right.
- rmbyrro 5y ago
  I think this can be said about any technology, really...
  - jugg1es 5y ago
    Yea, probably, but it is especially true for DynamoDB because it can initially appear as though your use cases are all supported but that is only because you haven't internalized how it works yet. By the time you realize you made a mistake, you are way too far in the weeds and have to start over from scratch. I would venture that more than 50% of DynamoDB users have had this happen to them early on. Anecdotally, just look at the comments on this post. There are so many horror stories with DynamoDB, but they're basically all people who decided to use it before they really understood it.
restlake 5y ago
If you know and understand S3 pretty well, and you purely need to generate, store, and read materialized static views, I highly recommend S3 for this use case. I say this as someone who really likes working with DDB daily and understands the tradeoffs with Dynamo. You can always layer on Athena or (simpler) S3 Select later if a SQL query model is a better fit than KV object lookups. S3 is loosely the fire and forget KV DB you’re describing IMO depending on your use case
whalesalad 5y ago
Dynamo is incredibly hard to use correctly I’d urge you to start writing a prototype, a lot of your assumptions might get thrown out the window. Dynamo is not necessarily good for reading high volume. You’ll end up needing to use a parallel scan approach which is not fast.
- qvrjuec 5y ago
  I'd say Dynamo is extremely good at reading high volume, with the appropriate access pattern. It's very efficient at retrieving huge amounts of well partitioned data using the data's keys, but scanning isn't so efficient.
  - whalesalad 5y ago
    You can only ever fetch 1MB of data at a time though, even when using the more efficient query method (as opposed to scan). If your individual entities are not very tiny, it is hard to get for instance 2M items back in a reasonable amount of time.
- mythrwy 5y ago
  Also can be _very_ expensive if you do not use it correctly.
qaq 5y ago
Save yourself a ton of pain and don't use DynamoDB
_pdp_ 5y ago
DynamoDB is like S3 but with query features. It is not a relational db. It is a document storage. So you need to use it for what it is. Our entire solution is basically based on top of lambda and dynamodb tables and it works really as long as you don't threat the tables like SQL.
Tehnix 5y ago
A lot of things that used to be a concern (hot partitions, etc) are not a concern anymore and most have been solved these days :) Put it on on-demand pricing (it'll be better and cheaper for you most likely), and it will handle any load you throw at it. Can you get it to throttle? Sure, if you absolutely blast it without ever having had that high of a need before (and it can actually be avoided[0]). You will need to understand how to model things for the NoSQL paradigm that DynamoDB uses, but that's a question of familiarity and not much else (you didn't magically know SQL either). My experience comes from scaling DynamoDB in production for several years, handling both massive IoT data ingestion in it as well as the user data as well. We were able to replace all things we thought we would need a relational database for, completely. My comparison between a traditional RDS setup: - DynamoDB issues? 0. Seriously. Only thing you need to monitor is billing. - RDS? Oh boy, need to provision for peak capacity, need to monitor replica lags, need to monitor the Replicas themselves, constant monitoring and scaling of IOPS, suddenly queries get slow as data increases, worrying about indexes and the data size, and much more... [0]: https://theburningmonk.com/2019/03/understanding-the-scaling-behaviour-of-dynamodb-ondemand-tables/ https://theburningmonk.com/2019/03/understanding-the-scaling...
pbalau 5y ago
Thinking like this both baffles me, but also makes me happy because there will always be a need for people like me, infra. AWS is not a magical tool that will replace your infra team, it is a magical tool that will allow your infra team to do more. I am the infra team of my startup and I estimate that only 50% of my time is doing infra work. The rest is supporting my peers, work in frameworky stuff, solve dev efficiency issues bla bla. Lets say that you operate in an AWS-less environment, with everything bare metal, in a datacenter. Your GOOD infra team has to do the following: Hardware: - make sure there is a channel to get new hardware, both for capacity increase and spares. What are you going to do? Buy 1 server and 2 spares? If one of the servers has an issue, isn't it quite likely that the other servers, from the same batch, to have the same issue? Is this affecting you, or not? Where do you store the spares? In a warehouse somewhere, making it harder to deploy? In the rack with the one in use, wasting rackspace/switch space? Are you going to rely on the datacenter to provide you with the hardware? What if you are one of their smaller customers and your requests get pushed back because some larger customer requests get higher priority? - make sure there is a way to deploy said hardware. You don't want to not be able to deploy a new server because there is no space in the rack, or no space in the switch. Where are your spares? In a warehouse miles away from the datacenter? Do you have access to said warehouse at midnight, on Thanksgiving? Oh shit, someone lost the key to your rack! Oh noes, we don't have any spare network cable/connectors/screws... Software: - did you patch your servers? did you patch your switches? - new server, we need to install the os. And a base set of software, including the agent we use to remote manage the server. - oh, we also need to run and maintain the management infra, say the control plane for k8. - oh, we want some read replicas for this db, not only we need the hardware to run the replicas on (and see above for what that means), now you need to add a bunch of monitoring and have plans in place to handle things like: replicas lagging, network links between master and replicas being full, failover for the above, master crapping out yada yada. I bet there are many other aspects I'm missing. Choices: Your GOOD infra team will have to decide things like: how many spares do we need, is the capacity we have atm enough for the launch of our next world-changing feature that half the internet wants to use? Are we lucky enough to survive a few months without spares or should we get estra capacity in another datacenter? Do we want to have replicas on the west coast or is the latency acceptable? These are the main areas of what an infra team is supposed to do: Hardware, Software and Choices. AWS (and most other cloud providers) is making the first 2 points non issues. For the last area you can do 2 things: get an infra team (could be a full fledged team, could be 1 person, you could do it) and teoretically you will get choices tailored to what your business needs OR let AWS do it for you. *AWS might make these choices based on a metric you disagree with and this is the main reason people complain*.
phamilton 5y ago
I don't know your scaling needs, but I would highly recommend just using Aurora postgresql for read-only workloads. We have some workloads that are essentially K/V store lookups that were previously slated for dynamodb. On an Aurora cluster of 3*r6g.xlarge we easily handle 25k qps with p99 in the single-digit ms range. Aurora can scale up to 15 instances and up to 24xlarge, so it would not be unreasonable to see 100x the read workload with similar latencies. Happy to talk more. We're actively moving a bunch of workloads away from DynamoDB and to Aurora so this is fresh on our minds.
- afandian 5y ago
  Thanks, that's what I hope will work. I might drop you a mail at some point.
giaour 5y ago
If you use the "pay per request" billing model instead of provisioned throughput, DynamoDB scaling is self-managing, and you can treat your DB as a fire-and-forget key/value store. You need to plan how you'll query your data and structure the keys accordingly, but honestly, that applies even more to S3 than it does to Dynamo.
AtlasBarfed 5y ago
The salespeople always promise magic and handwave CAP away. But data at scale is about: 1) knowing your queries ahead of time (since you've presumably reached the limit of PG/maybesql/o-rackle. 2) dealing with CAP at the application level: distributed transactions, eventual consistency, network partitions. 3) dealing with a lot more operational complexity, not less. So if the snake oil salesmen say it will be seamless, they are very very very much lying. Either that, or you are paying a LOT of money for other people to do the hard work. Which is what happens with managing your own NoSQL vs DynamoDB. You'll pay through the roof for DynamoDB at true big data scales.
nesarkvechnep 5y ago
If it’s possible in your situation, instead of vendor lock-in, invest in cacheability of your service and leverage HTTP cache as much as possible.
jedberg 5y ago
> Is the dream of a self-managing, fire-and-forget key value database completely naive? It's not, if you plan it right. Learn about single table design for DynamoDB before you start. There are a lot of good resources from Amazon and the community. Here is a very accessible video from the community: https://www.youtube.com/watch?v=BnDKD_Zv0og https://www.youtube.com/watch?v=BnDKD_Zv0og Here is a video from Rick Houlihan, a senior leader from AWS who basically helps companies convert to single table design: https://www.youtube.com/watch?v=KYy8X8t4MB8 https://www.youtube.com/watch?v=KYy8X8t4MB8 And a good book on the topic: https://www.dynamodbbook.com https://www.dynamodbbook.com If you use single table design, you can turn on all of the auto-tuning features of DynamoDB and they will work as expected and get better and more efficient with more data. Some people worry that this breaks the cardinal rule of microservices: One database per service. But the actual rule is never have one service directly access the data of another, always use the API. So as long as your services use different keyspaces and never access each other's data, it can still work (but does require extra discipline).
amzn-throw 5y ago
The key benefit with DDB is predictability: https://brooker.co.za/blog/2022/01/19/predictability.html https://brooker.co.za/blog/2022/01/19/predictability.html Yes, you have to learn about all these things upfront. But once you figure it out, test it, and configure it - it will work as you expect. No surprises. Whereas Relational Databases work until they don't. A developer makes a tiny (even a no-op) change to a query or stored procedure, a different SQL plan gets chosen, and suddenly your performance/latency dramatically reduces, and you have no easy way to roll it back through source control/deployment pipelines. You have to page a DBA who has to go pull up the hood. With services like DDB, you maintain control.
stickfigure 5y ago
Take a look at Firestore / Google Cloud Datastore. It's pretty much exactly what you describe - fire and forget. There's no concept of "node" (at least not from the outside).
emodendroket 5y ago
Dynamo is like the opposite of fire and forget it. You really want to know your access patterns at design time.

victor106 5y ago

We tried to implement an application on DynamoDB about 2 years ago. We really struggled with implementing adhoc queries/search. For e.g:- select * from employees where name = X and city = Y. Any improvements in DynamoDB that make it easier to implement such queries?

garydevenay 5y ago
The most reliable way to build a system with DynamoDB is to plan queries upfront. Trying to use it like a SQL database and make Ad-Hoc queries won't work because it's not a SQL DB. Data should be stored in the fashion you wish for it to be read, and storing the same data in more than one configuration is acceptable. Good resource: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-general-nosql-design.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
jon-wood 5y ago
This will sound flippant, but that's not what Dynamo is for. If you want to do freeform relational queries like that then put it in a relational database. Dynamo is primarily designed for high volume storage/querying on well understood data sets with a few query patterns. If you want to be able to query information on employees based on their name and city you'll need to build another index keyed on name and city (in practice Dynamo makes that reasonably simple by adding a secondary index).
- owenmarshall 5y ago
  Alternatively, practice single table design: structure your table keys in such a way that they can represent all (or at least most) of the queries you need to run. This is often easier said than done, but it can be far less expensive and more performant than adding an index for each search.
  - ellimilial 5y ago
    It's always great fun compounding new, manual 'indexes' when you discover you need another query.

gigatexal 5y ago

If you follow Rick Houlihan (@houlihan_rick) then all the accolades that AWS for DynamoDB pale in comparison to its current team and execution in that the company seems to not be investing in it so much so that Rick left to join MongoDB.

vslira 5y ago
Man I love Rick’s talks as much as anyone but let’s be real, he likely left AWS not for his love of first class geographical indexes but because Mongo offered a giant pile of money for him to evangelize their tech. Though I have no doubts that he actually had a lot of reservations around Dynamo’s DX before, he likely has some around mongodb but those won’t be the bulk of his content
- gigatexal 5y ago
  At his rank at AWS I don’t know if money was such an issue. He strikes me as a person who cares deeply about the underlying tech. But I have no idea one way or the other.
  - awsthro00945 5y ago
    I think I've seen you post something similar on r/aws about how Rick was "top DynamoDb person at AWS" (apologies if that wasn't you). I think you are overestimating Rick's "rank". I just looked him up (I had not heard of him before seeing his name mentioned on r/aws a few days ago) and he was an L7 TPM/Practice Manager in AWS's sales organization. That's not really a notably high position, and in the grand scheme of Amazon pay scales, isn't that high up. An L7 TPM gets paid about the same as, or sometimes less than, an L6 software dev (L6 is "senior", which is ~5-10 years of experience). Also, him being in the sales org means he had practically nothing to do with the engineering of the service. AWS Sales is a revolving door of people. I mean no offense towards Rick (again, I didn't know him or even know of him before I read his name in a comment a few days ago), but I would not read anything at all into the fact that an L7 Sales TPM left for another company.

tybit 5y ago

For anyone else expecting this to be a paper given the domain name, it’s not. It’s a non technical interview with a couple of the original papers authors. Not bad, just not as exciting as I imagine a paper detailing what they’ve learnt from a distributed systems perspective etc operating Dynamo then DynamoDB for so long now.

uvdn7 5y ago
https://brooker.co.za/blog/2022/01/19/predictability.html https://brooker.co.za/blog/2022/01/19/predictability.html This might be something you are looking for.
mjb 5y ago
We don't have a paper on DynamoDB's internals (yet?), but here's a talk you might find interesting from one of the folks who built and ran DDB for a long time: https://www.youtube.com/watch?v=yvBR71D0nAQ https://www.youtube.com/watch?v=yvBR71D0nAQ And Doug Terry talking through the details of how DynamoDB's transaction protocol works: https://www.usenix.org/conference/fast19/presentation/terry https://www.usenix.org/conference/fast19/presentation/terry If we did publish more about the internals of DDB, what would you be looking to learn? Architecture? Operational experience? Developer experience? There's a lot of material we could share, and it's useful to hear where people would like us to focus.
- pow_pp_-1_v 5y ago
  All of it - architecture, operational experience, best practices etc.
  - ldrndll 5y ago
    Just want to second this. All of the above sounds really interesting to me!

minaor 5y ago

[flagged]

jugg1es 5y ago

Every system I've built on DynamoDB just works. The APIs that use it have had virtually 100% uptime and have never needed database maintenance. It is not a replacement for a RDBMS, but for some use cases, it's a killer service.

graderjs 5y ago

The in-the-trenches, technical, battle-tested/battle-scarred comments on this thread are why I come to HN for the comments.

manishsharan 5y ago

Quite a few of the teams that were early adopters of AWS DynamoDB were not prepared for the pricing nuances that had to be taken into consideration when building their solutions.

xtracto 5y ago
I remember trying Dynamodb around 2015/2016: You had to specify your expected read and write throughout and you would be billed for that. At that time we had a pretty spikey traffic use case which made using dynamodb efficiently impossible
- musingsole 5y ago
  I had a similar experience, but ultimately wrote a service to monitor our workloads and request increased provisioning during spikes. You could reduce your provisioning like 10 times a day, but after that you could only increase it and would be stuck with the higher rate for a time. And then on-demand provisioning was released and it was cheap enough to be worth simplifying our workflows.
jvanvleet 5y ago
I was one of these. However I now understand that the pricing nuances reflected a reality that I appreciate. We used DDB in a way that was not the best fit and the cost was a reflection of this.

salil999 5y ago

I can safely say that the team members working in DynamoDB are very skilled and they care deeply about the product. They really work hard and think of interesting solutions to a lot of problems that their biggest customers face which is great from a product standpoint. There are some pretty smart people working there. Engineering, however, was a disaster story. Code is horribly written and very few tests are maintained to make sure deployments go without issues. There was too much emphasis on deployment and getting fixes/features out over making sure it won't break anything else. It was a common scenario to release a new feature and put duct tape all around it to make sure it "works". And way too many operational issues. There are a lot of ways to break DynamoDB :) Overall, though, the product is very solid and it's one of the few database that you can say "just works" when it comes to scalability and reliability (as most AWS services are) I worked at DynamoDB for over 2 years.

vasili111 5y ago
>Engineering, however, was a disaster story. Code is horribly written and very few tests are maintained to make sure deployments go without issues. There was too much emphasis on deployment and getting fixes/features out over making sure it won't break anything else. It was a common scenario to release a new feature and put duct tape all around it to make sure it "works". And way too many operational issues. There are a lot of ways to break DynamoDB :) >Overall, though, the product is very solid and it's one of the few database that you can say "just works" when it comes to scalability and reliability (as most AWS services are) How those two can coexist?
- badhombres 5y ago
  I mean eventually enough duct tape can be solid like a tank :)
  - jeffreygoesto 5y ago
    Or a bridge... https://www.popularmechanics.com/science/a5732/mythbusters-build-duct-tape-bridge/ https://www.popularmechanics.com/science/a5732/mythbusters-b...

dabfiend19 5y ago

knocking off mongo... 10 years later they still haven't caught up.

artembugara 5y ago

We realized how great Dynamo was only after we migrated off AWS. Dynamo was a key factor to us when we were releasing the MVP of our News API [0]. We used Dynamo, ElasticSearch, Lambda and could make it running in 60 days while being full-time employed. Also, the best tech talk I saw was given by Rick Houlihan on re:Invent [1] I highly recommend every engineer to watch it: it's a great overview of SQL vs NoSQL [0] https://newscatcherapi.com/blog/how-we-built-a-news-api-beta-in-60-days https://newscatcherapi.com/blog/how-we-built-a-news-api-beta... [1] https://www.youtube.com/watch?v=HaEPXoXVf2k https://www.youtube.com/watch?v=HaEPXoXVf2k

pier25 5y ago
BTW Rick Houlihan left AWS recently to work for Mongo. https://twitter.com/houlihan_rick/status/1472969503575265283 https://twitter.com/houlihan_rick/status/1472969503575265283 On that thread he criticizes AWS regarding DynamoDB openly. > I will always love DynamoDB, but the fact is it is losing ground fast because AWS focuses most of their resources on the half baked #builtfornopurpose database strategy. I always hated that idea, I just bit my tongue instead of saying it. > The problem is the other half-baked database services that all compete for the same business. DocumentDB, Keyspaces, Timestream, Neptune, etc. Databases take decades to optimize, the idea that you can pump them out like web apps is silly. > I was very tired of explaining over and over again that DynamoDB is actually not the dumbed down Key-Value store that the marketing message implied. When AWS created 6 different NoSQL databases they had to make up reasons for each one and the messaging makes no sense.
- throwdbaaway 5y ago
  Interesting. MongoDB actually came to mind while I was reading the other comment here: > No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable querying, and then they bolt on Redis to make the disjointed backend feel fast. And I'm just talking operational use cases; ignoring analytics here. Perhaps MongoDB is prime for a comeback?

cmollis 5y ago

we use DynamoDB like a big hash table of s3 file locations.. we look up these locations via a key (at the time, it sounded like a pretty good use-case for it). I suppose we could have used some other managed redis or memcached thing, but being an AWS shop, it was, and is, pretty useful. I have to say, it's been pretty effortless to configure.. read/write units are really the only thing we've had to configure (other than the base index) The rest of it has been easy. It has about a 100 million entries that are read or written pretty quickly.

Jach 5y ago
I remember talking to someone who was playing with AWS stuff for the first time and they had a similar architecture, using Dynamo for a lookup store. It still seems a bit odd to me though. It's been a long time since I've worked with the S3 API, so maybe it just doesn't support the same sort of thing, but wouldn't it be nicer to just query S3 with some key and get back either the path/URL to render a link, or the content itself? Why the Dynamo intermediary? (And on the other side, if you don't need to render a link to serve the content, why not use Dynamo as the actual document store and skip S3? Storage cost?)
- cmollis 5y ago
  You're not going to get O(1) lookup using the S3 API. The APIs are basically O(n). So we we use DynamoDB to store the S3 URL.
  - cmollis 5y ago
    Moreover, storing a large binary file (in this case) in Dynamo is probably not the best use-case for it..most likely would have to convert to base64 in and out of it.

deleted 5y ago

[deleted]

didip 5y ago

To be honest, as a customer, it is hard for me to justify using DynamoDB. Some of this criticism can be out of date: 1. DynamoDB is not as convenient. There are a bit too many dials to turn. 2. DynamoDB does not have a SQL facade on top. 3. DynamoDB is proprietary, I believe there's no OSS API equivalent if you want to migrate out. 4. DynamoDB was kind of expensive. But it has been a while since I last check the pricing page. It's simply much better to start with PostgreSQL Aurora and move to a more scalable storage based on specific uses-cases later. For example: Cassandra, Elastic, Druid, or CockroachDB.

nathanfig 5y ago
That’s surprising to me, I consider DynamoDB to be far simpler than any relational DB including the alternatives you list
- deleted 5y ago
  [deleted]
arecurrence 5y ago
I strongly agree that most early stage businesses should be on Postgres. There's simply too much churn in early stage data models. Also, unforeseen esoteric needs jumping out of the wood work that you can knock out a SQL query for instead of having to build a solution come up constantly. However, this does assume that your development team has a competent understanding of SQL. I've been in a couple startups that went Dynamo first and development velocity was a pale shadow of velocity with Postgres. When one of those startups dumped dynamo for Postgres velocity multiplied immediately. I'd estimate we were moving at around 1000% and the complete transition took less time than even I expected (about a month). Once the business matures, moving tables onto dynamo and wrapping them in a microservice makes a lot of sense. Dynamo does solve a lot of problems that become increasingly material as the business evolves. Eventually, SQL's presence declines and transitions into an analytics system as narrower, but easier for ops, options proliferate.
Zimnx 5y ago
Ad 3. Lot of people don't know about it, but there's a open source, free, and DynamoDB compatibile databse called ScyllaDB - API it's called Alternator to be specific.

nathanfig 5y ago

I wonder if DynamoDB would be met with less criticism had it simply been named Dynamo Document Store.

ddoolin 5y ago

As a developer, I really have a love-hate relationship with Dynamo. I love how fast and easy it is to setup and get rolling. The partitioning scheme came off as confusing and opaque but I think that says more about Amazon's documentation than the scheme itself. I do not like that there's no really third party tooling integration to be able to query. Their UI in the console is _so freaking terrible_ yet you have no other way than code to query it. This problem is so bad that I will avoid using it where I can despite it being a good option, performance-wise.

ejb999 5y ago
My experience exactly.
rmbyrro 5y ago
Don't think this is a fair reason to avoid DynamoDB. There are reasonably better alternatives: - DynamoDB Workbench (free, AWS official): https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/workbench.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg... - Dynobase (paid, third-party): https://dynobase.dev/ https://dynobase.dev/
- belter 5y ago
  Plus there is the local install ( Only for Dev and Testing purposes...) "Deploying DynamoDB Locally on Your Computer" https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.DownloadingAndRunning.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
deleted 5y ago
[deleted]
wnolens 5y ago
Yes the console UI is mind numbing, and was just made 10x worse in a recent redesign. But i like the python boto3 library https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html https://boto3.amazonaws.com/v1/documentation/api/latest/guid... Build yourself a few wrappers to make querying more convenient, and i query straight from a python repl pretty effectively

0xbadcafebee 5y ago

I haven't run into anyone who uses Dynamo for anything other than managing Terraform backend state locking. And I think that's still the best use case for it: you just want to store a couple random key-values somewhere and have more functionality than AWS Parameter Store. Trying to build anything large-scale with it will probably leave you wanting.

sakopov 5y ago

We landed on DynamoDB when we migrated a monolith to microservice architecture. I have to say that DynamoDB fits fairly well in the microservices world where the service scope is small, query patterns are pretty narrow and don't really change much. Building new things using DynamoDB when query patterns aren't necessarily known is very painful and require tedious migration strategies unless you don't mind paying for GSIs.

anonymousDan 5y ago

Can anyone recommend me a good paper (or other resource) describing the designs of DynamoDB and S3? Ideally something in the spirt of the original Dynamo paper (i.e. https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf https://www.allthingsdistributed.com/files/amazon-dynamo-sos... )

schwarzmx 5y ago
This one is pretty good for DynamoDB: https://youtu.be/yvBR71D0nAQ https://youtu.be/yvBR71D0nAQ

balls187 5y ago

DynamoDB is one of my favorite AWS products.

thekozmo 5y ago

Indeed quite a journey. If you love DynamoDB and like open source, give Scylla a try (disclosure me==founder): https://www.scylladb.com/alternator/ https://www.scylladb.com/alternator/

whoisjohnkid 5y ago

I absolutely love DynamoDB. Would definitely recommend the DynamoDB Book by Alex Debrie and Advanced DynamoDB Patterns by Rick H on YouTube. Once you understand how to properly use dynamo and what it’s good at it you get so much power; at a fraction of the cost depending on your workload. Was a breeze to setup multi region applications that utilize the single table design strategy. If you ever need complex queries just use dynamo streams to power whatever search solution you are comfortable with.

robertlagrant 5y ago

> Customers no longer want to just store and query the data in their databases. They then want to analyze that data to create value Presumably create more value. I know it's a marketing post but still - storing and retrieving things is pretty valuable in and of itself.