12 ms·

There are very few suitable use cases for DynamoDB

232 points by vgt 9y ago

ezulich 9y ago
Full disclosure: I am a member of DynamoDB team at AWS, but my opinions here are mine only On the topic of DynamoDB use cases, here are some of DynamoDB users describing their use cases: - DataXu: "How DataXu scaled its System to handle billions of events with DynamoDB" -- https://youtu.be/lDiI0JMf_yQ?t=765 https://youtu.be/lDiI0JMf_yQ?t=765 (from AWS re:Invent 2016) - Lyft: “Lyft Easily Scales Up its Ride Location Tracking System with Amazon DynamoDB” -- https://www.youtube.com/watch?v=WlTbaPXj-jc https://www.youtube.com/watch?v=WlTbaPXj-jc. The story "AWS Helps Fuel Lyft's Expansion" in InformationWeek mentions Lyft's use of DynamoDB as well: http://www.informationweek.com/cloud/infrastructure-as-a-service/aws-helps-fuel-lyfts-expansion-/d/d-id/1326660 http://www.informationweek.com/cloud/infrastructure-as-a-ser... - Under Armour implemented cross-region replication using DynamoDB: https://youtu.be/NtaTC2Fq7Wo?t=699 https://youtu.be/NtaTC2Fq7Wo?t=699 - Amazon.com: Amazon Marketplace shares their story of migration to DynamoDB and why they did it: https://www.youtube.com/watch?v=gllNauRR8GM https://www.youtube.com/watch?v=gllNauRR8GM Finally, DynamoDB was created because Amazon needed a highly reliable and scalable key/value database. As Werner Vogels said it in his blog post announcement of DynamoDB back in 2012 (http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html http://www.allthingsdistributed.com/2012/01/amazon-dynamodb....): "This non-relational, or NoSQL, database was targeted at use cases that were core to the Amazon ecommerce operation, such as the shopping cart and session service."
jsemrau 9y ago
When I started building my first app in 2011 MongoDB was the rage. So I build the back-end using the futuristic 'No-SQL' technology. It turned out to be slow (~1 min Query time), inconsistent, and missing an RDBMS layer. Move the thing to PHP/Mysql problems were gone. I still have not found a use case outside of web (comments/discussion) sites where the high integration with Javascript actually makes sense.
- dijit 9y ago
  I hope you don't mind if I piggyback on this to echo this sentiment. Although I dislike MySQL for its many gotchas (data corruption level stuff too!) I was looking for a _long_ _long_ time for a high consistency NoSQL database.. we basically need document storage of large binary data. Ironically literally nothing in NoSQL land does write-through to disk, they just write to vfs and hope it works; additionally, those that support clustering opt for eventual consistency.. That just baffles my mind, so much potential for lost or corrupted data. We ended up doing deterministic sharding on postgresql, it worked incredibly well, even in failure modes you hope never to see.. and no corruptions! :D
  - jethro_tell 9y ago
    Isn't that the point of no_sql. You trade consistency for speed. Best used to mark temporary state in a web app. If you need atomic consistency in a database you should use a relational database that does atomic writes. That has always been the case. Why would it be so shocking to you? Atomic writes have already been built. Sounds like you were standing there with a hammer looking for a nail.
    - dijit 9y ago
      I always thought the term "document store" just meant arbitrary unstructured data that didn't need to be related. That's how you do things like we did regarding deterministic sharding- it's very easy to scale this way if you don't ever "JOIN" tables etc;
    - threeseed 9y ago
      Please be specific. NoSQL comprises hundreds of different databases with very different semantics. Technically PostgreSQL is a NoSQL database and it along with Cassandra for example have modes in which they are fully and atomically consistent.
  - threeseed 9y ago
    You are talking complete nonsense. HBase, Cassandra, MongoDB, Riak, Couchbase etc all write through to disk with proper fsync flushes. And I've never heard of any database that has a model where it writes to a virtual file system - whatever that even means. Please provide some specific examples.
avitzurel 9y ago
The title is very... what's the two words I'm looking for? Dynamo does not scale for this specific use case but I have used it successfully in production (at scale) with ZERO issues. Dynamo is a key value sharded and zero operations* database that most applications and companies will benefit from IMHO. is the hot key and evenly sending queries to nodes the only issues you concluded we should not use DynamoDB on?
karmakaze 9y ago
TL;DR - Don't use something as coarse as customer_id as a partition key. Alternatively move to GCP/BigTable. Any DynamoDB tuning advice will say how important it is to have well distributed hash keys. As for the second part, why not use Cloud Spanner? I wish AWS had something like it.
- deleted 9y ago
  [deleted]
- nemothekid 9y ago
  >Any DynamoDB tuning advice will say how important it is to have well distributed hash keys. Maybe I'm missing something, but as I understand it he did have well distributed hash keys (I'm assuming his customer ids were random UUIDs). The problem he had however was that he had so much data that a single custom could cause throughput exceptions on a very small amount of ops/s.
  - lotyrin 9y ago
    Customer UUID would only be well-distributed for queries of customers. If some customers are more active than others in a given unit time, some customers have more data than others, and customers' data is keyed by customer ID then no, it's not a well distributed key.
cherioo 9y ago
"Your business has millions of customers and no single customer can do so many actions so quickly that the individual could create a hot key. Under this key you are storing around 2KB of data. ... Potentially getting 1–5 requests per second for a given second but certainly not a sustained load of that. ... This will not work at scale in DynamoDb." What? Why? Suppose that's 5 million customers, you will only have a 10GB table which fits in a single DynamoDB shard, with no sharding. With the restriction of 1-5 operation per customer per second, this sounds like the ideal use case for DynamoDB. What am I missing?
- fragsworth 9y ago
  Amazon is limiting it to 5 operations per second per shard, not per customer. That is if you have 200 shards and 1000 write capacity. It's a problem with how Amazon calculates your capacity units as it scales up Dynamo for you, if you have a particularly large data set with a few very active users.
  - phamilton 9y ago
    The parent post is suggesting that 5 Million customers does not in any way imply 200 shards. It's much more likely that with 5M customers you have 1 or 2 shards. That's 4KB (one document) per customer.
nikanj 9y ago
The gist of this seems to be that DynamoDB becomes a problem if you have millions of customers. Don't worry. You don't. And there will be many good reasons to refactor the architecture before you do.
- mnutt 9y ago
  I often give this same advice, but assuming this is the case, why reach for DynamoDB at all? Are there small-data use cases where DynamoDB makes more sense than RDS or a hosted redis? At least with those, if you do run into scaling issues you haven't locked yourself into dynamo.
  - justinsaccount 9y ago
    I'm not sure how much this applies to dynamodb.. I think I couldn't even figure out how to get started with that. I used simpledb once as part of project. All I needed it for was to store a small amount of state to be shared between some otherwise stateless ec2 instances. So, kind of the perfect use case.. Except I don't think it is made to scale down that small. I don't know if I provisioned the capacity wrong or what, but it was so flaky. All I needed it to do was a few reads every few minutes. I think my issue was I had the provisioned scalability set down to the smallest value I could because I only needed to pull down ~10 rows every 60 seconds or so.. But when the app would start and try to sync up, simpledb would throttle it? I ended up having to just write a cron job to sync the db to a local json file, because doing the reads from inside my main loop would randomly fail or timeout.
  - ryandvm 9y ago
    I would imagine the use of AWS Lambda might be a pretty decent incentive to use DynamoDB. I'm guessing that stateless DynamoDB queries are faster to fire off than dealing with Postgres or MySQL connection setup and teardown.
    - TheAceOfHearts 9y ago
      I'm not familiarized with DynamoDB, but why would connecting to it be any faster than connecting to Postgres or MySQL?
brandur 9y ago
I've seen two large usages of DynamoDB at two different companies, and for what it's worth, in both cases we've had similar trouble as the author. In one case we ended up ripping it out and moving to a sharded Postgres scheme, and in the other we've left in place for now because a migration will be such a monumental effort, but it's pretty much universally maligned. Fundamentally, the problem seems to be that choosing a partitioning key that's appropriate for DynamoDB's operational properties is ... unlikely. In their own docs on choosing a partition key [1] they use "user ID" as an example of one with good uniformity, but in reality if you choose something like that, you're probably about to be in for a world of pain: in many systems big users can be 7+ orders of magnitude bigger than small users, so what initially looked like a respectable partitioning key turns out to be very lopsided. As mentioned in the article, you can then try to increase throughput, but you won't have enough control over the newly provisioned capacity to really address the problem. You can massively overprovision, but then you're paying for a lot of capacity that's sitting idle, and even then sometimes it's not enough. Your best bet is probably to choose a partition key that's perfectly uniformly distributed (like a random ID), but at that point you're designing your product around DynamoDB rather than vice versa, and you should probably wonder why you're not looking at alternatives. --- [1] http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForTables.html#GuidelinesForTables.UniformWorkload http://docs.aws.amazon.com/amazondynamodb/latest/developergu...
- hox 9y ago
  Conpletely agreed; using DynamoDB for primary storage suffers from needing to design data models around the underlying technology. This is true for secondary keys as well, and is even constrained by the odd pricing model that DynamoDB follows. I've been happy using DynamoDB as a large-scale caching layer, but even that only fits very specific use case criteria.
  - fragsworth 9y ago
    > using DynamoDB for primary storage suffers from needing to design data models around the underlying technology. Every database system, at large scale, suffers from this problem.
awfullyjohn 9y ago
He complains that DynamoDB doesn't work for him, then says you should instead use Google BigTable. But he doesn't offer evidence why you should use BigTable. And just says that it works for him. I don't buy it. I've used BigTable in the past and found it to be infuriating. Now, because it works for him, I'm supposed to believe that BigTable is right for me?
- double-a 9y ago
  What did you find infuriating about BigTable?
- Diederich 9y ago
  I believe the article specifically notes that BigTable gave far better performance than DynamoDB.
- mbrukman 9y ago
  Hey there, I'm the PM for Google Cloud Bigtable; if you want to talk about your experiences with Bigtable, I'd love to hear what your use case was, what your experience was, and your thoughts on how we can make Bigtable better for you. Feel free to follow me on Twitter (see my profile) and we can chat over DM if you wish. If we chat and decide that Bigtable is appropriate for your use case, maybe you'd be willing to give it another shot? :-)
windlep 9y ago
I don't know what the article author is storing, but it's noted that 10GB of data is stored per node. That's quite a bit of data for a single table, and the 10GB is per shard of a table, not per 'database' (DynamoDB only has a notion of tables). Amazon has deep dive talks on DynamoDB on YouTube[1] that go into lots of these details and how to avoid problems from them. It's not that different from understanding how to structure data for Cassandra, Mongo, etc. All the NoSQL systems require an understanding of how they work both to structure your data and ensure optimal performance. For example, maybe one's consistency constraints are better met by DynamoDB instead of BigTable's varying consistency on different query types[2] (the author of this article didn't address consistency at all). With DynamoDB, you can get strongly consistent reads and queries all the time [3]. Overall, it seems like a kind of weak reason to make the strong statement about "probably shouldn't be using DynamoDB". Maybe a better title would be "Understanding drawbacks of large datasets in DynamoDB". I do hope the author understands the consistency changes they may experience in BigTable, as it could easily require large changes to the code-base if strong consistency was assumed on all queries. [1] https://www.youtube.com/watch?v=bCW3lhsJKfw https://www.youtube.com/watch?v=bCW3lhsJKfw [2] https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore/ https://cloud.google.com/datastore/docs/articles/balancing-s... [3] http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html http://docs.aws.amazon.com/amazondynamodb/latest/developergu... Edit: Fixed inconsistency in what the 10GB limit was referring to, not per table, but per node (shard) of the table.
- lwansbrough 9y ago
  We store 100-1000 GB of data in each wide table on our untrendy on-prem SQL server boxes. 10GB is peanuts. In fact, I propose (given the limitations suggested by the author) that DynamoDB may have NO practical uses worth exploring. If NoSQL is about scale, and it can't scale, what's it good for? I can understand having to optimize your key space, but in this case it necessitates extreme premature optimization.
  - ryanworl 9y ago
    Each shard is 10GB. The unit of scale is a shard. You can have as many shards as you want.
dyeje 9y ago
Our team has been using it alongside Postgres as a scalable metric store and things have been pretty good. We had some growing pains tweaking our storage keys and switching to a daily table to avoid partitioning issues, but it's been quite stable for a while now.
hox 9y ago
The biggest issue faced with DynamoDB from my perspective has been the problem with any hosted service - that is, the operational stability is in the hands of others entrusted with operating a massive multi-tenant system, and any outage cannot follow your own operational recovery mechanisms unless you plan for failover yourself. And the moment you plan for failover, you need to evaluate why you don't just handle the primary system as well.
- elvinyung 9y ago
  I mean... of course? But I feel like the idea here is that you can "entrust" them with operating the system and not worry about rolling your own failover mechanism, as long as they seem to be upholding their SLAs. (Of course, cloud vs. bare metal is a long-running debate, but I think I take the position that multitenant clouds are generally cheap enough that it's worth the cost for the extra feature velocity, if you're not truly at scale.)
- apetresc 9y ago
  There are many things people justifiably complain about DynamoDB, but operational support has rarely been one of them.
  - hox 9y ago
    Dynamodb experienced a severe outage in us-east in September of 2015, dropping the service to three nines of uptime on just the single event. During the incident there was little-to-no knowledge of when service would be restored, so companies that relied on the service were flying blind. Such an outage can sometimes be mitigated when operations are internal to a company; but with DynamoDB, one has to build internal failover or implement a complex cross-region replication scheme. The complexity and reliability starts to reduce the attractiveness of the hosted service in general.
kevan 9y ago
>This will not work at scale in DynamoDb. I don't think we're getting the whole story from the author. I'm not the biggest fan of Dynamo either for reasons I won't get into here, but this type of workload is exactly what Dynamo was built for: serving a website with millions of customers. Disclaimer: I work at Amazon, my views are my own.
- thinkloop 9y ago
  So what about the issue of requiring heavy over-provisioning for common usage patterns, like a customer being logged in causing temporary bursts of access on specific keys?
  - mrep 9y ago
    Aka, the status quo before AWS allowed you to dynamically scale certain parts of your infrastructure. It's not perfect for all parts of your stack, but do you have a better alternative for databases?
    - deleted 9y ago
      [deleted]
banku_brougham 9y ago
The first part of the article sounds convincing, but i'm pretty sure amazon is using DynamoDB extensively and successfully at massive scales.
- coredog64 9y ago
  Does the internal charge back model for Dynamo work the same as it does for AWS customers? If I understand the author's point, you _can_ get great performance, but it requires over provisioning. So it's not so much a technical failing as much as it is a poor monetisation of the capability. I see that elsewhere in AWS: They require huge instance types for large EBS volumes with ES. For our workload, said instances barely break a sweat.
iampims 9y ago
rather poor example of "how dynamodb doesn't scale". DynamoDB is definitely not the silver bullet people hope it is but does work exceptionally well for what it was designed for.
jey 9y ago
So it's fine as long as I precompute the full transaction to apply to the key, then do it all at once? I expect large overall transaction volumes but am happy to only issue one DB change per user interaction.
sheeshkebab 9y ago
Dynamodb has a nice js interface and is a good helper store for various small aws specific automation projects. It's not a good choice for various read heavy enterprise apps (and frankly for write ones neither). It also doesn't scale - it doesn't even have a cross region active/active support (highly surprising for a key value store when cassandra supported it for ages). Don't use it for anything serious.
znep 9y ago
Does anyone have any real world experience with how the Amazon DynamoDB Accelerator that AWS released as a preview a few months ago can help out?
natekupp 9y ago
We use DynamoDB quite a bit at Thumbtack. Our biggest issue is backups - just wrote a short note about our experiences with DynamoDB here: https://medium.com/@natekupp/dynamodb-and-backups-16dba0dbcded https://medium.com/@natekupp/dynamodb-and-backups-16dba0dbcd...
- Dunedan 9y ago
  Oh yes. Backups for DynamoDB are a pain in the ass, especially as AWS doesn't offer an out-of-the-box solution for that. That Data Pipeline + EMR solution mentioned in the blog post (here is a better link for it: https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-importexport-ddb.html https://docs.aws.amazon.com/datapipeline/latest/DeveloperGui...) has several drawbacks: - too many moving parts, especially given the track record of EMR - might not even be available when your requirement is to keep the data in the same AWS region as the DynamoDB table, as only five regions support Data Pipeline The best approach I've seen so far is to use DynamoDB Streams and an AWS Lambda function to create incremental backups in an versioned S3-bucket. dynamodb-replicator (https://github.com/mapbox/dynamodb-replicator https://github.com/mapbox/dynamodb-replicator) implements that together with some scripts to do management tasks like back filling an S3 bucket with data which is already in DynamoDB or joining incremental backups into a single file. It's still pretty unpolished and definitely needs some love, but I think it's the right approach.
chairmanwow 9y ago
Quick aside: It seems that a lot of the qualms with various databases stem from a misunderstanding of their use cases. A lot of the features of major SQL databases, namely ACID, are misappropriated to be features of databases in general. This key misunderstanding seems to cause a lot of SWEs insane as they later realize that NoSQL DBs are not always Available and Strongly Consistent. Responding to Author: I wasn't totally convinced by the author's argument against DynamoDB. This article [1] offers a good solution to pretty much all of OP's problems. Most significantly, hashing user data using date. While DynamoDB is certainly different from most other databases, that doesn't mean that there aren't sensical usage methodologies. Links: [1] https://medium.com/building-timehop/one-year-of-dynamodb-at-timehop-f761d9fe5fa1 https://medium.com/building-timehop/one-year-of-dynamodb-at-...
cyberferret 9y ago
Hmm... We dabbled with DynamoDB on a couple of very small projects, but found some real oddities in how it stores JSON data, and manages keys. Querying the dataset was also a mammoth, frustrating task. Ended up switching to RethinkDB and haven't looked back - far better query syntax/language (REQL) and we can organise the JSON content just how we want it.
TheAceOfHearts 9y ago
Many of the comments here are saying that the author's use-case wasn't a good one for DynamoDB. Can anyone share some simple approachable resources that talk about the kinds of use-cases where these tools make sense? Whenever I read about NoSQL systems, I'm always left a unsure about its use-cases. I've only worked on systems where a traditional RDBMS made the most sense. How do you identify when it's appropriate to reach for one of the many NoSQL tools? I've had the suspicion that many applications that leverage NoSQL tools are usually used in conjunction with a relational database, and not in isolation. Based on my limited understanding, I can at least wrap my head around a few ways in which this could probably help. Am I off the mark here? One of the points I struggle with is that once you're storing data across multiple data stores, maintaining data integrity becomes much harder.
- qaq 9y ago
  when your data set grows beyond what can reasonably stored in RDBMS e.g. 20+ TB
  - phamilton 9y ago
    20TB in an RDBMS has very different characteristics than 100GB in an RDBMS. Once you can no longer store your dataset in memory, you start to see greater variance in latency. DynamoDB for us has given us very reliable latency for our datasets that are too large to hold in memory on an RDBMS.
    - qaq 9y ago
      You can get a box with 48TB RAM so 20TB has more to do with time it takes for DB to come up. To provision DynamoDB to the same performance as PG on i3.16xlarge instances you will be paying north of 50K/month vs 5K/month and that's with all limitations that come with DynamoDB.
      - phamilton 9y ago
        An i3.16xlarge is 488GB, not 48TB. The x1.32xlarge (largest amazon offers) is 2TB. Additionally, self-managed PG is a huge operational undertaking compared to DynamoDB. RDS is a bit closer, but the largest RDS offers is db.r3.8xlarge, which is 244GB. The db.r3.8xlarge on RDS Postgres all in with 3000 provisioned IOPS (just provisioning for writes, all reads should be served from memory) and 1.5TB storage ends up around $6k. A 3 year reservation will get that down to $3k. Comparable DynamoDB (3000 write units, 3000 read units, reserved IOPS) comes out to $1500. The key here is consistent performance and low operational overhead. As I said, we're pretty happy with it.
qaq 9y ago
I can see the future: article that outlines gotchas of Bigtable article you probably shouldn’t use Bigtable article the amount of money we could've saved using PG and not rewriting things 3 times. anything up to 10-15TB there are very few reasons not to use something like PG
- x0x0 9y ago
  And pg supports schema-free data (that they can index!) as columns in your standard sql tables. Check out jsonb if you haven't tried it. It's really really good. https://www.postgresql.org/docs/9.6/static/datatype-json.html https://www.postgresql.org/docs/9.6/static/datatype-json.htm...
ParadisoShlee 9y ago
Use case. Use case. Use case.
- ezulich 9y ago
  Full disclosure: I am a member of DynamoDB team at AWS, but my opinions here are mine only Here are some of DynamoDB customers describing their use cases: - DataXu: "How DataXu scaled its System to handle billions of events with DynamoDB" -- https://youtu.be/lDiI0JMf_yQ?t=765 https://youtu.be/lDiI0JMf_yQ?t=765 (from AWS re:Invent 2016) - Lyft: “Lyft Easily Scales Up its Ride Location Tracking System with Amazon DynamoDB” -- https://www.youtube.com/watch?v=WlTbaPXj-jc https://www.youtube.com/watch?v=WlTbaPXj-jc. The story "AWS Helps Fuel Lyft's Expansion" in InformationWeek mentions Lyft's use of DynamoDB as well: http://www.informationweek.com/cloud/infrastructure-as-a-service/aws-helps-fuel-lyfts-expansion-/d/d-id/1326660 http://www.informationweek.com/cloud/infrastructure-as-a-ser... - Under Armour implemented cross-region replication using DynamoDB: https://youtu.be/NtaTC2Fq7Wo?t=699 https://youtu.be/NtaTC2Fq7Wo?t=699 - Amazon.com: Amazon Marketplace shares their story of migration to DynamoDB: https://www.youtube.com/watch?v=gllNauRR8GM https://www.youtube.com/watch?v=gllNauRR8GM Finally, DynamoDB was created because Amazon needed a highly reliable and scalable key/value database. As Werner Vogels said it in his blog post announcement of DynamoDB back in 2012 (http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html http://www.allthingsdistributed.com/2012/01/amazon-dynamodb....): "This non-relational, or NoSQL, database was targeted at use cases that were core to the Amazon ecommerce operation, such as the shopping cart and session service."
imauld 9y ago
We DynamoDB to store user alerts and it works fairly well for our use case. However we still run into issues with consumed throughput being well under provisioned but still getting throughput exceptions. We don't use the auto incremented user ID's either. We create a hash of the user ID and some of the other data contained in the alert and it would appear we still have hot partitions.
macinjosh 9y ago
My company is in the online form builder space and usage of our MySQL RDS instances are nearing the limit of what is offered. I looked into moving certain high-volume/high-traffic data models to DynamoDB and realized we could not because each record is limited to a measly (for our needs) 400kb.
didibus 9y ago
The article forgets a very important detail: > A single partition can hold approximately 10 GB of data, and can support a maximum of 3,000 read capacity units or 1,000 write capacity units. DynamoDb will also split your data if you provision more then 3000 reads or 1000 writes. And the caveat is that it will not join back the shards if you later reduce the throughput back down. Instead, each shard will just get even less throughput then you might believe.
- didibus 9y ago
  Another caveat is that when a shard is split because it has reached its max size, the resulting new 2 shards each get half of the throughput the initial shard had, and not a proportion of the total throughput. So say you have 4000 write capacity and 0 reads (for simplicity). DynamoDb will allocate 4 shards for it, each getting 1000 writes. Now say shard 2 gets too big, and goes above 10GB. DynamoDb will split it in two. Now you have 5 shards, but they won't all get 4000 / 5 = 800 writes. Instead, the original shard 1 3 and 4 will each still have 1000, and shard 2a and 2b will have 500 each. That's because when dynamodb splits a shard, it redistribute the throughput of its parent to the shards.
  - nw-fcx 9y ago
    I am really surprised by this, I was convinced that the table-wide capacity is re-distributed evenly across shards. Looks like you're entirely correct: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForTables.html#GuidelinesForTables.Partitions.Throughput http://docs.aws.amazon.com/amazondynamodb/latest/developergu... I wonder how many times I scrolled past the explanation without internalizing it.
ZGF4 9y ago
There are a few misguided views in this article and in some of these comments. 1. Every shardable database (Cassandra, Dynamo, BigTable) has to worry about hot spots. Picking a UUID as a partition key is only step one. What happens if one user is a huge majority of your traffic? All of their reads/writes are going to a single partition and of course you are going to suffer from performance issues from that hot spot. It becomes important to further break down your partition into synthetic shards or break up your data by time (only keep a day of data per shard). BigTable does not innately solve this, they may deal better with a large partition but it will inevitably become a problem. 2. Some people are criticizing the choice of NoSQL citing the data size. Note you can have a small data size but have huge write traffic. An unsharded RDBMS will not scale well to this since you cannot distribute the writes across multiple nodes. Don't assume just because someone has a small data set they don't need to use NoSQL to deal with their volume
- jjirsa 9y ago
  If one user is the majority of your traffic, you don't shard on user ID.
- sametmax 9y ago
  It depends a lot of the write. If they can be batched, you can put them in a queue or in redis until it reach a threshold and write the update down in the RDBMS. It won't work for all the use cases, but more often that people think.
  - ZGF4 9y ago
    Yes microbatching is a great way to get a fixed write rate regardless of traffic, can even do it right at the application layer. The trade-off is that you can theoretically lose one interval of data when your service goes down. This might not matter for analytics workloads with a margin of error but some usecases require confirmed writes
  - jjirsa 9y ago
    If you have a persistent data problem, and introduce redis, then you'll have two persistent data problems.
deleted 9y ago
[deleted]
jjirsa 9y ago
The best argument against dynamodb is the aws "well architected" guidelines - how are you designing for resiliency with your single region, active-passive database with clunky bolt on replication using kinesis? Cheaper active-active-active options exist that don't require manual dr failover drills and manual failback when regions inevitably crash.
xendo 9y ago
I don't see how the scenario with a single customer ID is credible. Why not just put a cache in front of the db? Now there is even a fully managed solution for that: https://aws.amazon.com/dynamodb/dax/ https://aws.amazon.com/dynamodb/dax/
amygdyl 9y ago
I think the problem is that there are very few suitable length articles that are available for you at no cost, and the debate has been suffering from a lack of understanding in the widest sense of the problem and so little can be found on any website that tries to make a commercial subsistence which is any way better than you scan reading SO randomly the day before you go to present to management for their architectural planning meeting. /s is I hope self evident Only seriously for my purposes, the use I get from the typical article on databases which I find is linked to from HN, is a reverse index to the better discussions, long after the discussion is off the front page here. I'm merely a little confused about the fact that widespread consternation of the quality of geek journalism for programmers is not much more than merely a occasional mention or moan. Genuinely is a good dose of cynicism in force, which is invisible to me? I understand that when I see a story about science intended for a general audience, then HN is likely to become home to much greater detail and depth in discussion. But with a subject line arguing generalised conclusion from a subject matter of database architecture? Sometimes I think that I'm confused about whether I'm supposed to be confused about the point of the TFA.
dserban 9y ago
My background is in Cassandra and one company I worked for last year insisted that we use DynamoDB for a project. Here are a few things that ended up being show stoppers. 1. Both the partition key and the sort key are capped at 1 field. In an attempt to "think Cassandra data model", the ugly workaround was to stringify and concatenate things at the application layer, then parse / split on the other side. This made the code unreadable. 2. DynamoDB-Spark integration is a second-class citizen. (Cassandra-Spark integration is first-class and well-maintained.) 3. The other thing that made code unreadable was the accidental complexity introduced by exception handling / exponential backoff we needed to implement to protect against accidental read capacity underprovisioning. Although I made repeated pleas to switch to Cassandra, the (non-technical) CEO insisted that we keep using DynamoDB. I'm no longer at that company but I hear they have meanwhile switched to RedShift.
- pishpash 9y ago
  Cassandra isn't any better in those regards. For 1, Cassandra just does the concatenation under the hood and 3 happens all the time on Cassandra. The broken thing is using NoSQL as a DB, not Cassandra or DynamoDB.
  - jjirsa 9y ago
    Cassandra allows multiple clustering columns, it's not simple concatenation under the hood. There are no hidden exponential backoffs in Cassandra in any of the hot query paths, period. Suggesting that using nosql as a DB is broken is so ludicrous I'm not even going to try to refute it - you simply don't know what you're talking about.
    - deleted 9y ago
      [deleted]
  - ckalantzis 9y ago
    Quick! Someone should tell Netflix and Apple, that. They've been using Cassandra and Datastax as their main DB for almost a decade. They're always hiring. You should apply and show them the error of their ways.
luhn 9y ago
I took a look at BigTable, as recommended by the article, because I was evaluating DynamoDB myself just yesterday. It looks like the minimum price for that is ~$1500 a month. Granted, you're getting what you pay for (3 nodes that support 10,000QPS each), but the pricing is out of reach for smaller projects.
- 5ersi 9y ago
  You should take a look at Google Cloud Database which is a better option and has per-operation pricing instead of per-node pricing.
  - luhn 9y ago
    That is very attractive pricing. It looks like you don't even need to preallocate capacity, just read and write as you please.
luhn 9y ago
Related is The Million Dollar Engineering Problem [1] from Segment, which showed up on HN a few months ago. They shaved $300K from their annual AWS spend by identifying and dealing with hot shards on DynamoDB. [1] https://segment.com/blog/the-million-dollar-eng-problem/ https://segment.com/blog/the-million-dollar-eng-problem/
pishpash 9y ago
NoSQL should stay as a quick and dirty solution for storing key-values. When you start to schematize it and use it as a "DB," you are going down the wrong path. That is because NoSQL is a glorified cache, not a DB. It is essentially using the memory on a large number of nodes to buffer bursty throughput, and using background processes to collate the data later onto disk. There is almost no case where an explicit distributed caching or queuing solution backed by a traditional DB isn't strictly better.
- fennecfoxen 9y ago
  > It is essentially using the memory on a large number of nodes to buffer bursty throughput, and using background processes to collate the data later onto disk. You're not describing NoSQL. You're describing something like MongoDB specifically. You're right about the specific case, wrong in the general case. > There is almost no case where an explicit distributed caching or queuing solution backed by a traditional DB isn't strictly better. Bah. If you're looking at big enough data sets (think "Walmart scale", not "web scale") then you have to design your data store to cope with this volume. First, it has to be replicated to multiple servers to serve more queries and for reliability, and is an eventually consistent system a consequence (basically any time you use your read replica, or do anything on a multimaster configuration). You'll need something shardable or multimaster to achieve horizontally scalablity and you'll probably also lose your foreign key enforcement and uniqueness constraints when you do this, so you might as well go the distance and fully denormalise everything for consistent access times. (Have you ever had the Postgres query planner suddenly make something take 10x longer than it usually does because your data patterns changed and decided to use a different query plan than it's ever used before? Yeah, that's a world of fun. Now make that thing a time-critical daily batch job and really squirm.) When you do the things to make this work, you're basically restricting yourself, abandoning database features. As a consequence, you are going to be in for a world of pain, period, sorry. One key advantage of good NoSQL software is that it doesn't give you those features to begin with, so it's harder to get it wrong. It also has data structures designed to make this sort of thing work well and be performant in ways that general purpose databases won't be, on account of being designed for general purposes.
abalone 9y ago
So, Amazon actually kind of agrees! They talked about this very issue of hot keys and overprovisioning at their most recent conference. The thing is it was buried in a session on Aurora and they didn't mention DynamoDB by name -- they just called it nosql -- but they noted that a customer cut their costs 40% by moving to Aurora. Because it does automatic heat management and just bills you for what IO you use. This is somewhat at odds with their top-level messaging which still pushes DynamoDB as the most scalable solution. And perhaps it is... there are some scalability limits to Aurora. Writes are bottlenecked by one instance. 64TB max. I think performance drops when you exceed the in memory cache. But those limits are still quite large. Basically I sense some tension between the DynamoDB and Aurora teams and I wonder where this is all going to shake out in the long run. Here's the full quote (I transcribed it so may contain errors): "The one thing that surprised me is that there are some customers who are moving their nosql workload to aurora. There are two reasons for that. One, it’s a lot easier to use Aurora because it’s mysql compatible compared to nosql because the interfaces and transaction characteristics are so different. What is also interesting is people also saved money because the IO cost is much lower. In no SQL if you have a large table it gets partitioned then the IO gets partitioned across all the table partitions that you have. And if you have one partition that is hot then you have to provision based on the IO requirement of the hot partition. In the case of Aurora we do automatic heat management so we don’t have this hot parition issue. Second we don’t charge based on provisioned IO. It’s only the IO that you use. And that actually saves a lot of money. In this particular case this is a big social company, interaction company, I cannot tell the name, and they reduced their operational costs by 40% by moving from nosql to Aurora" [1] [1] https://youtu.be/60QumD2QsF0?t=17m01s https://youtu.be/60QumD2QsF0?t=17m01s
datashovel 9y ago
Generally speaking. Given that DynamoDB is a NoSQL database service, I'm not certain that moving larger clients to their own dedicated AWS resources should cause too many negative side effects. Especially ones who are so large they're causing scalability issues.
rushi_agrawal 9y ago
If DynamoDB guys can come up with a mechanism similar to their CPU credits concept, that'd be a really nice feature to have. Of course it can't be as straightforward as CPU credits.

232 points by vgt 9y ago

Full disclosure: I am a member of DynamoDB team at AWS, but my opinions here are mine only On the topic of DynamoDB use cases, here are some of DynamoDB users describing their use cases: - DataXu: "How DataXu scaled its System to handle billions of events with DynamoDB" -- https://youtu.be/lDiI0JMf_yQ?t=765 https://youtu.be/lDiI0JMf_yQ?t=765 (from AWS re:Invent 2016) - Lyft: “Lyft Easily Scales Up its Ride Location Tracking System with Amazon DynamoDB” -- https://www.youtube.com/watch?v=WlTbaPXj-jc https://www.youtube.com/watch?v=WlTbaPXj-jc. The story "AWS Helps Fuel Lyft's Expansion" in InformationWeek mentions Lyft's use of DynamoDB as well: http://www.informationweek.com/cloud/infrastructure-as-a-service/aws-helps-fuel-lyfts-expansion-/d/d-id/1326660 http://www.informationweek.com/cloud/infrastructure-as-a-ser... - Under Armour implemented cross-region replication using DynamoDB: https://youtu.be/NtaTC2Fq7Wo?t=699 https://youtu.be/NtaTC2Fq7Wo?t=699 - Amazon.com: Amazon Marketplace shares their story of migration to DynamoDB and why they did it: https://www.youtube.com/watch?v=gllNauRR8GM https://www.youtube.com/watch?v=gllNauRR8GM Finally, DynamoDB was created because Amazon needed a highly reliable and scalable key/value database. As Werner Vogels said it in his blog post announcement of DynamoDB back in 2012 (http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html http://www.allthingsdistributed.com/2012/01/amazon-dynamodb....): "This non-relational, or NoSQL, database was targeted at use cases that were core to the Amazon ecommerce operation, such as the shopping cart and session service."

jsemrau 9y ago

When I started building my first app in 2011 MongoDB was the rage. So I build the back-end using the futuristic 'No-SQL' technology. It turned out to be slow (~1 min Query time), inconsistent, and missing an RDBMS layer. Move the thing to PHP/Mysql problems were gone. I still have not found a use case outside of web (comments/discussion) sites where the high integration with Javascript actually makes sense.

dijit 9y ago
I hope you don't mind if I piggyback on this to echo this sentiment. Although I dislike MySQL for its many gotchas (data corruption level stuff too!) I was looking for a _long_ _long_ time for a high consistency NoSQL database.. we basically need document storage of large binary data. Ironically literally nothing in NoSQL land does write-through to disk, they just write to vfs and hope it works; additionally, those that support clustering opt for eventual consistency.. That just baffles my mind, so much potential for lost or corrupted data. We ended up doing deterministic sharding on postgresql, it worked incredibly well, even in failure modes you hope never to see.. and no corruptions! :D
- jethro_tell 9y ago
  Isn't that the point of no_sql. You trade consistency for speed. Best used to mark temporary state in a web app. If you need atomic consistency in a database you should use a relational database that does atomic writes. That has always been the case. Why would it be so shocking to you? Atomic writes have already been built. Sounds like you were standing there with a hammer looking for a nail.
  - dijit 9y ago
    I always thought the term "document store" just meant arbitrary unstructured data that didn't need to be related. That's how you do things like we did regarding deterministic sharding- it's very easy to scale this way if you don't ever "JOIN" tables etc;
  - threeseed 9y ago
    Please be specific. NoSQL comprises hundreds of different databases with very different semantics. Technically PostgreSQL is a NoSQL database and it along with Cassandra for example have modes in which they are fully and atomically consistent.
- threeseed 9y ago
  You are talking complete nonsense. HBase, Cassandra, MongoDB, Riak, Couchbase etc all write through to disk with proper fsync flushes. And I've never heard of any database that has a model where it writes to a virtual file system - whatever that even means. Please provide some specific examples.

avitzurel 9y ago

The title is very... what's the two words I'm looking for? Dynamo does not scale for this specific use case but I have used it successfully in production (at scale) with ZERO issues. Dynamo is a key value sharded and zero operations* database that most applications and companies will benefit from IMHO. is the hot key and evenly sending queries to nodes the only issues you concluded we should not use DynamoDB on?

karmakaze 9y ago

TL;DR - Don't use something as coarse as customer_id as a partition key. Alternatively move to GCP/BigTable. Any DynamoDB tuning advice will say how important it is to have well distributed hash keys. As for the second part, why not use Cloud Spanner? I wish AWS had something like it.

deleted 9y ago
[deleted]
nemothekid 9y ago
>Any DynamoDB tuning advice will say how important it is to have well distributed hash keys. Maybe I'm missing something, but as I understand it he did have well distributed hash keys (I'm assuming his customer ids were random UUIDs). The problem he had however was that he had so much data that a single custom could cause throughput exceptions on a very small amount of ops/s.
- lotyrin 9y ago
  Customer UUID would only be well-distributed for queries of customers. If some customers are more active than others in a given unit time, some customers have more data than others, and customers' data is keyed by customer ID then no, it's not a well distributed key.

cherioo 9y ago

"Your business has millions of customers and no single customer can do so many actions so quickly that the individual could create a hot key. Under this key you are storing around 2KB of data. ... Potentially getting 1–5 requests per second for a given second but certainly not a sustained load of that. ... This will not work at scale in DynamoDb." What? Why? Suppose that's 5 million customers, you will only have a 10GB table which fits in a single DynamoDB shard, with no sharding. With the restriction of 1-5 operation per customer per second, this sounds like the ideal use case for DynamoDB. What am I missing?

fragsworth 9y ago
Amazon is limiting it to 5 operations per second per shard, not per customer. That is if you have 200 shards and 1000 write capacity. It's a problem with how Amazon calculates your capacity units as it scales up Dynamo for you, if you have a particularly large data set with a few very active users.
- phamilton 9y ago
  The parent post is suggesting that 5 Million customers does not in any way imply 200 shards. It's much more likely that with 5M customers you have 1 or 2 shards. That's 4KB (one document) per customer.

nikanj 9y ago

The gist of this seems to be that DynamoDB becomes a problem if you have millions of customers. Don't worry. You don't. And there will be many good reasons to refactor the architecture before you do.

mnutt 9y ago
I often give this same advice, but assuming this is the case, why reach for DynamoDB at all? Are there small-data use cases where DynamoDB makes more sense than RDS or a hosted redis? At least with those, if you do run into scaling issues you haven't locked yourself into dynamo.
- justinsaccount 9y ago
  I'm not sure how much this applies to dynamodb.. I think I couldn't even figure out how to get started with that. I used simpledb once as part of project. All I needed it for was to store a small amount of state to be shared between some otherwise stateless ec2 instances. So, kind of the perfect use case.. Except I don't think it is made to scale down that small. I don't know if I provisioned the capacity wrong or what, but it was so flaky. All I needed it to do was a few reads every few minutes. I think my issue was I had the provisioned scalability set down to the smallest value I could because I only needed to pull down ~10 rows every 60 seconds or so.. But when the app would start and try to sync up, simpledb would throttle it? I ended up having to just write a cron job to sync the db to a local json file, because doing the reads from inside my main loop would randomly fail or timeout.
- ryandvm 9y ago
  I would imagine the use of AWS Lambda might be a pretty decent incentive to use DynamoDB. I'm guessing that stateless DynamoDB queries are faster to fire off than dealing with Postgres or MySQL connection setup and teardown.
  - TheAceOfHearts 9y ago
    I'm not familiarized with DynamoDB, but why would connecting to it be any faster than connecting to Postgres or MySQL?

brandur 9y ago

I've seen two large usages of DynamoDB at two different companies, and for what it's worth, in both cases we've had similar trouble as the author. In one case we ended up ripping it out and moving to a sharded Postgres scheme, and in the other we've left in place for now because a migration will be such a monumental effort, but it's pretty much universally maligned. Fundamentally, the problem seems to be that choosing a partitioning key that's appropriate for DynamoDB's operational properties is ... unlikely. In their own docs on choosing a partition key [1] they use "user ID" as an example of one with good uniformity, but in reality if you choose something like that, you're probably about to be in for a world of pain: in many systems big users can be 7+ orders of magnitude bigger than small users, so what initially looked like a respectable partitioning key turns out to be very lopsided. As mentioned in the article, you can then try to increase throughput, but you won't have enough control over the newly provisioned capacity to really address the problem. You can massively overprovision, but then you're paying for a lot of capacity that's sitting idle, and even then sometimes it's not enough. Your best bet is probably to choose a partition key that's perfectly uniformly distributed (like a random ID), but at that point you're designing your product around DynamoDB rather than vice versa, and you should probably wonder why you're not looking at alternatives. --- [1] http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForTables.html#GuidelinesForTables.UniformWorkload http://docs.aws.amazon.com/amazondynamodb/latest/developergu...

hox 9y ago
Conpletely agreed; using DynamoDB for primary storage suffers from needing to design data models around the underlying technology. This is true for secondary keys as well, and is even constrained by the odd pricing model that DynamoDB follows. I've been happy using DynamoDB as a large-scale caching layer, but even that only fits very specific use case criteria.
- fragsworth 9y ago
  > using DynamoDB for primary storage suffers from needing to design data models around the underlying technology. Every database system, at large scale, suffers from this problem.

awfullyjohn 9y ago

He complains that DynamoDB doesn't work for him, then says you should instead use Google BigTable. But he doesn't offer evidence why you should use BigTable. And just says that it works for him. I don't buy it. I've used BigTable in the past and found it to be infuriating. Now, because it works for him, I'm supposed to believe that BigTable is right for me?

double-a 9y ago
What did you find infuriating about BigTable?
Diederich 9y ago
I believe the article specifically notes that BigTable gave far better performance than DynamoDB.
mbrukman 9y ago
Hey there, I'm the PM for Google Cloud Bigtable; if you want to talk about your experiences with Bigtable, I'd love to hear what your use case was, what your experience was, and your thoughts on how we can make Bigtable better for you. Feel free to follow me on Twitter (see my profile) and we can chat over DM if you wish. If we chat and decide that Bigtable is appropriate for your use case, maybe you'd be willing to give it another shot? :-)

windlep 9y ago

I don't know what the article author is storing, but it's noted that 10GB of data is stored per node. That's quite a bit of data for a single table, and the 10GB is per shard of a table, not per 'database' (DynamoDB only has a notion of tables). Amazon has deep dive talks on DynamoDB on YouTube[1] that go into lots of these details and how to avoid problems from them. It's not that different from understanding how to structure data for Cassandra, Mongo, etc. All the NoSQL systems require an understanding of how they work both to structure your data and ensure optimal performance. For example, maybe one's consistency constraints are better met by DynamoDB instead of BigTable's varying consistency on different query types[2] (the author of this article didn't address consistency at all). With DynamoDB, you can get strongly consistent reads and queries all the time [3]. Overall, it seems like a kind of weak reason to make the strong statement about "probably shouldn't be using DynamoDB". Maybe a better title would be "Understanding drawbacks of large datasets in DynamoDB". I do hope the author understands the consistency changes they may experience in BigTable, as it could easily require large changes to the code-base if strong consistency was assumed on all queries. [1] https://www.youtube.com/watch?v=bCW3lhsJKfw https://www.youtube.com/watch?v=bCW3lhsJKfw [2] https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore/ https://cloud.google.com/datastore/docs/articles/balancing-s... [3] http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html http://docs.aws.amazon.com/amazondynamodb/latest/developergu... Edit: Fixed inconsistency in what the 10GB limit was referring to, not per table, but per node (shard) of the table.

lwansbrough 9y ago
We store 100-1000 GB of data in each wide table on our untrendy on-prem SQL server boxes. 10GB is peanuts. In fact, I propose (given the limitations suggested by the author) that DynamoDB may have NO practical uses worth exploring. If NoSQL is about scale, and it can't scale, what's it good for? I can understand having to optimize your key space, but in this case it necessitates extreme premature optimization.
- ryanworl 9y ago
  Each shard is 10GB. The unit of scale is a shard. You can have as many shards as you want.

dyeje 9y ago

Our team has been using it alongside Postgres as a scalable metric store and things have been pretty good. We had some growing pains tweaking our storage keys and switching to a daily table to avoid partitioning issues, but it's been quite stable for a while now.

hox 9y ago

The biggest issue faced with DynamoDB from my perspective has been the problem with any hosted service - that is, the operational stability is in the hands of others entrusted with operating a massive multi-tenant system, and any outage cannot follow your own operational recovery mechanisms unless you plan for failover yourself. And the moment you plan for failover, you need to evaluate why you don't just handle the primary system as well.

elvinyung 9y ago
I mean... of course? But I feel like the idea here is that you can "entrust" them with operating the system and not worry about rolling your own failover mechanism, as long as they seem to be upholding their SLAs. (Of course, cloud vs. bare metal is a long-running debate, but I think I take the position that multitenant clouds are generally cheap enough that it's worth the cost for the extra feature velocity, if you're not truly at scale.)
apetresc 9y ago
There are many things people justifiably complain about DynamoDB, but operational support has rarely been one of them.
- hox 9y ago
  Dynamodb experienced a severe outage in us-east in September of 2015, dropping the service to three nines of uptime on just the single event. During the incident there was little-to-no knowledge of when service would be restored, so companies that relied on the service were flying blind. Such an outage can sometimes be mitigated when operations are internal to a company; but with DynamoDB, one has to build internal failover or implement a complex cross-region replication scheme. The complexity and reliability starts to reduce the attractiveness of the hosted service in general.

kevan 9y ago

>This will not work at scale in DynamoDb. I don't think we're getting the whole story from the author. I'm not the biggest fan of Dynamo either for reasons I won't get into here, but this type of workload is exactly what Dynamo was built for: serving a website with millions of customers. Disclaimer: I work at Amazon, my views are my own.

thinkloop 9y ago
So what about the issue of requiring heavy over-provisioning for common usage patterns, like a customer being logged in causing temporary bursts of access on specific keys?
- mrep 9y ago
  Aka, the status quo before AWS allowed you to dynamically scale certain parts of your infrastructure. It's not perfect for all parts of your stack, but do you have a better alternative for databases?
  - deleted 9y ago
    [deleted]

banku_brougham 9y ago

The first part of the article sounds convincing, but i'm pretty sure amazon is using DynamoDB extensively and successfully at massive scales.

coredog64 9y ago
Does the internal charge back model for Dynamo work the same as it does for AWS customers? If I understand the author's point, you _can_ get great performance, but it requires over provisioning. So it's not so much a technical failing as much as it is a poor monetisation of the capability. I see that elsewhere in AWS: They require huge instance types for large EBS volumes with ES. For our workload, said instances barely break a sweat.

iampims 9y ago

rather poor example of "how dynamodb doesn't scale". DynamoDB is definitely not the silver bullet people hope it is but does work exceptionally well for what it was designed for.

jey 9y ago

So it's fine as long as I precompute the full transaction to apply to the key, then do it all at once? I expect large overall transaction volumes but am happy to only issue one DB change per user interaction.

sheeshkebab 9y ago

Dynamodb has a nice js interface and is a good helper store for various small aws specific automation projects. It's not a good choice for various read heavy enterprise apps (and frankly for write ones neither). It also doesn't scale - it doesn't even have a cross region active/active support (highly surprising for a key value store when cassandra supported it for ages). Don't use it for anything serious.

znep 9y ago

Does anyone have any real world experience with how the Amazon DynamoDB Accelerator that AWS released as a preview a few months ago can help out?

natekupp 9y ago

We use DynamoDB quite a bit at Thumbtack. Our biggest issue is backups - just wrote a short note about our experiences with DynamoDB here: https://medium.com/@natekupp/dynamodb-and-backups-16dba0dbcded https://medium.com/@natekupp/dynamodb-and-backups-16dba0dbcd...

Dunedan 9y ago
Oh yes. Backups for DynamoDB are a pain in the ass, especially as AWS doesn't offer an out-of-the-box solution for that. That Data Pipeline + EMR solution mentioned in the blog post (here is a better link for it: https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-importexport-ddb.html https://docs.aws.amazon.com/datapipeline/latest/DeveloperGui...) has several drawbacks: - too many moving parts, especially given the track record of EMR - might not even be available when your requirement is to keep the data in the same AWS region as the DynamoDB table, as only five regions support Data Pipeline The best approach I've seen so far is to use DynamoDB Streams and an AWS Lambda function to create incremental backups in an versioned S3-bucket. dynamodb-replicator (https://github.com/mapbox/dynamodb-replicator https://github.com/mapbox/dynamodb-replicator) implements that together with some scripts to do management tasks like back filling an S3 bucket with data which is already in DynamoDB or joining incremental backups into a single file. It's still pretty unpolished and definitely needs some love, but I think it's the right approach.

chairmanwow 9y ago

Quick aside: It seems that a lot of the qualms with various databases stem from a misunderstanding of their use cases. A lot of the features of major SQL databases, namely ACID, are misappropriated to be features of databases in general. This key misunderstanding seems to cause a lot of SWEs insane as they later realize that NoSQL DBs are not always Available and Strongly Consistent. Responding to Author: I wasn't totally convinced by the author's argument against DynamoDB. This article [1] offers a good solution to pretty much all of OP's problems. Most significantly, hashing user data using date. While DynamoDB is certainly different from most other databases, that doesn't mean that there aren't sensical usage methodologies. Links: [1] https://medium.com/building-timehop/one-year-of-dynamodb-at-timehop-f761d9fe5fa1 https://medium.com/building-timehop/one-year-of-dynamodb-at-...

cyberferret 9y ago

Hmm... We dabbled with DynamoDB on a couple of very small projects, but found some real oddities in how it stores JSON data, and manages keys. Querying the dataset was also a mammoth, frustrating task. Ended up switching to RethinkDB and haven't looked back - far better query syntax/language (REQL) and we can organise the JSON content just how we want it.

TheAceOfHearts 9y ago

Many of the comments here are saying that the author's use-case wasn't a good one for DynamoDB. Can anyone share some simple approachable resources that talk about the kinds of use-cases where these tools make sense? Whenever I read about NoSQL systems, I'm always left a unsure about its use-cases. I've only worked on systems where a traditional RDBMS made the most sense. How do you identify when it's appropriate to reach for one of the many NoSQL tools? I've had the suspicion that many applications that leverage NoSQL tools are usually used in conjunction with a relational database, and not in isolation. Based on my limited understanding, I can at least wrap my head around a few ways in which this could probably help. Am I off the mark here? One of the points I struggle with is that once you're storing data across multiple data stores, maintaining data integrity becomes much harder.

qaq 9y ago
when your data set grows beyond what can reasonably stored in RDBMS e.g. 20+ TB
- phamilton 9y ago
  20TB in an RDBMS has very different characteristics than 100GB in an RDBMS. Once you can no longer store your dataset in memory, you start to see greater variance in latency. DynamoDB for us has given us very reliable latency for our datasets that are too large to hold in memory on an RDBMS.
  - qaq 9y ago
    You can get a box with 48TB RAM so 20TB has more to do with time it takes for DB to come up. To provision DynamoDB to the same performance as PG on i3.16xlarge instances you will be paying north of 50K/month vs 5K/month and that's with all limitations that come with DynamoDB.
    - phamilton 9y ago
      An i3.16xlarge is 488GB, not 48TB. The x1.32xlarge (largest amazon offers) is 2TB. Additionally, self-managed PG is a huge operational undertaking compared to DynamoDB. RDS is a bit closer, but the largest RDS offers is db.r3.8xlarge, which is 244GB. The db.r3.8xlarge on RDS Postgres all in with 3000 provisioned IOPS (just provisioning for writes, all reads should be served from memory) and 1.5TB storage ends up around $6k. A 3 year reservation will get that down to $3k. Comparable DynamoDB (3000 write units, 3000 read units, reserved IOPS) comes out to $1500. The key here is consistent performance and low operational overhead. As I said, we're pretty happy with it.

qaq 9y ago

I can see the future: article that outlines gotchas of Bigtable article you probably shouldn’t use Bigtable article the amount of money we could've saved using PG and not rewriting things 3 times. anything up to 10-15TB there are very few reasons not to use something like PG

x0x0 9y ago
And pg supports schema-free data (that they can index!) as columns in your standard sql tables. Check out jsonb if you haven't tried it. It's really really good. https://www.postgresql.org/docs/9.6/static/datatype-json.html https://www.postgresql.org/docs/9.6/static/datatype-json.htm...

ParadisoShlee 9y ago

Use case. Use case. Use case.

ezulich 9y ago
Full disclosure: I am a member of DynamoDB team at AWS, but my opinions here are mine only Here are some of DynamoDB customers describing their use cases: - DataXu: "How DataXu scaled its System to handle billions of events with DynamoDB" -- https://youtu.be/lDiI0JMf_yQ?t=765 https://youtu.be/lDiI0JMf_yQ?t=765 (from AWS re:Invent 2016) - Lyft: “Lyft Easily Scales Up its Ride Location Tracking System with Amazon DynamoDB” -- https://www.youtube.com/watch?v=WlTbaPXj-jc https://www.youtube.com/watch?v=WlTbaPXj-jc. The story "AWS Helps Fuel Lyft's Expansion" in InformationWeek mentions Lyft's use of DynamoDB as well: http://www.informationweek.com/cloud/infrastructure-as-a-service/aws-helps-fuel-lyfts-expansion-/d/d-id/1326660 http://www.informationweek.com/cloud/infrastructure-as-a-ser... - Under Armour implemented cross-region replication using DynamoDB: https://youtu.be/NtaTC2Fq7Wo?t=699 https://youtu.be/NtaTC2Fq7Wo?t=699 - Amazon.com: Amazon Marketplace shares their story of migration to DynamoDB: https://www.youtube.com/watch?v=gllNauRR8GM https://www.youtube.com/watch?v=gllNauRR8GM Finally, DynamoDB was created because Amazon needed a highly reliable and scalable key/value database. As Werner Vogels said it in his blog post announcement of DynamoDB back in 2012 (http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html http://www.allthingsdistributed.com/2012/01/amazon-dynamodb....): "This non-relational, or NoSQL, database was targeted at use cases that were core to the Amazon ecommerce operation, such as the shopping cart and session service."

imauld 9y ago

We DynamoDB to store user alerts and it works fairly well for our use case. However we still run into issues with consumed throughput being well under provisioned but still getting throughput exceptions. We don't use the auto incremented user ID's either. We create a hash of the user ID and some of the other data contained in the alert and it would appear we still have hot partitions.

macinjosh 9y ago

My company is in the online form builder space and usage of our MySQL RDS instances are nearing the limit of what is offered. I looked into moving certain high-volume/high-traffic data models to DynamoDB and realized we could not because each record is limited to a measly (for our needs) 400kb.

didibus 9y ago

The article forgets a very important detail: > A single partition can hold approximately 10 GB of data, and can support a maximum of 3,000 read capacity units or 1,000 write capacity units. DynamoDb will also split your data if you provision more then 3000 reads or 1000 writes. And the caveat is that it will not join back the shards if you later reduce the throughput back down. Instead, each shard will just get even less throughput then you might believe.

didibus 9y ago
Another caveat is that when a shard is split because it has reached its max size, the resulting new 2 shards each get half of the throughput the initial shard had, and not a proportion of the total throughput. So say you have 4000 write capacity and 0 reads (for simplicity). DynamoDb will allocate 4 shards for it, each getting 1000 writes. Now say shard 2 gets too big, and goes above 10GB. DynamoDb will split it in two. Now you have 5 shards, but they won't all get 4000 / 5 = 800 writes. Instead, the original shard 1 3 and 4 will each still have 1000, and shard 2a and 2b will have 500 each. That's because when dynamodb splits a shard, it redistribute the throughput of its parent to the shards.
- nw-fcx 9y ago
  I am really surprised by this, I was convinced that the table-wide capacity is re-distributed evenly across shards. Looks like you're entirely correct: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForTables.html#GuidelinesForTables.Partitions.Throughput http://docs.aws.amazon.com/amazondynamodb/latest/developergu... I wonder how many times I scrolled past the explanation without internalizing it.

ZGF4 9y ago

There are a few misguided views in this article and in some of these comments. 1. Every shardable database (Cassandra, Dynamo, BigTable) has to worry about hot spots. Picking a UUID as a partition key is only step one. What happens if one user is a huge majority of your traffic? All of their reads/writes are going to a single partition and of course you are going to suffer from performance issues from that hot spot. It becomes important to further break down your partition into synthetic shards or break up your data by time (only keep a day of data per shard). BigTable does not innately solve this, they may deal better with a large partition but it will inevitably become a problem. 2. Some people are criticizing the choice of NoSQL citing the data size. Note you can have a small data size but have huge write traffic. An unsharded RDBMS will not scale well to this since you cannot distribute the writes across multiple nodes. Don't assume just because someone has a small data set they don't need to use NoSQL to deal with their volume

jjirsa 9y ago
If one user is the majority of your traffic, you don't shard on user ID.
sametmax 9y ago
It depends a lot of the write. If they can be batched, you can put them in a queue or in redis until it reach a threshold and write the update down in the RDBMS. It won't work for all the use cases, but more often that people think.
- ZGF4 9y ago
  Yes microbatching is a great way to get a fixed write rate regardless of traffic, can even do it right at the application layer. The trade-off is that you can theoretically lose one interval of data when your service goes down. This might not matter for analytics workloads with a margin of error but some usecases require confirmed writes
- jjirsa 9y ago
  If you have a persistent data problem, and introduce redis, then you'll have two persistent data problems.

deleted 9y ago

[deleted]

jjirsa 9y ago

The best argument against dynamodb is the aws "well architected" guidelines - how are you designing for resiliency with your single region, active-passive database with clunky bolt on replication using kinesis? Cheaper active-active-active options exist that don't require manual dr failover drills and manual failback when regions inevitably crash.

xendo 9y ago

I don't see how the scenario with a single customer ID is credible. Why not just put a cache in front of the db? Now there is even a fully managed solution for that: https://aws.amazon.com/dynamodb/dax/ https://aws.amazon.com/dynamodb/dax/

amygdyl 9y ago

I think the problem is that there are very few suitable length articles that are available for you at no cost, and the debate has been suffering from a lack of understanding in the widest sense of the problem and so little can be found on any website that tries to make a commercial subsistence which is any way better than you scan reading SO randomly the day before you go to present to management for their architectural planning meeting. /s is I hope self evident Only seriously for my purposes, the use I get from the typical article on databases which I find is linked to from HN, is a reverse index to the better discussions, long after the discussion is off the front page here. I'm merely a little confused about the fact that widespread consternation of the quality of geek journalism for programmers is not much more than merely a occasional mention or moan. Genuinely is a good dose of cynicism in force, which is invisible to me? I understand that when I see a story about science intended for a general audience, then HN is likely to become home to much greater detail and depth in discussion. But with a subject line arguing generalised conclusion from a subject matter of database architecture? Sometimes I think that I'm confused about whether I'm supposed to be confused about the point of the TFA.

dserban 9y ago

My background is in Cassandra and one company I worked for last year insisted that we use DynamoDB for a project. Here are a few things that ended up being show stoppers. 1. Both the partition key and the sort key are capped at 1 field. In an attempt to "think Cassandra data model", the ugly workaround was to stringify and concatenate things at the application layer, then parse / split on the other side. This made the code unreadable. 2. DynamoDB-Spark integration is a second-class citizen. (Cassandra-Spark integration is first-class and well-maintained.) 3. The other thing that made code unreadable was the accidental complexity introduced by exception handling / exponential backoff we needed to implement to protect against accidental read capacity underprovisioning. Although I made repeated pleas to switch to Cassandra, the (non-technical) CEO insisted that we keep using DynamoDB. I'm no longer at that company but I hear they have meanwhile switched to RedShift.

pishpash 9y ago
Cassandra isn't any better in those regards. For 1, Cassandra just does the concatenation under the hood and 3 happens all the time on Cassandra. The broken thing is using NoSQL as a DB, not Cassandra or DynamoDB.
- jjirsa 9y ago
  Cassandra allows multiple clustering columns, it's not simple concatenation under the hood. There are no hidden exponential backoffs in Cassandra in any of the hot query paths, period. Suggesting that using nosql as a DB is broken is so ludicrous I'm not even going to try to refute it - you simply don't know what you're talking about.
  - deleted 9y ago
    [deleted]
- ckalantzis 9y ago
  Quick! Someone should tell Netflix and Apple, that. They've been using Cassandra and Datastax as their main DB for almost a decade. They're always hiring. You should apply and show them the error of their ways.

luhn 9y ago

I took a look at BigTable, as recommended by the article, because I was evaluating DynamoDB myself just yesterday. It looks like the minimum price for that is ~$1500 a month. Granted, you're getting what you pay for (3 nodes that support 10,000QPS each), but the pricing is out of reach for smaller projects.

5ersi 9y ago
You should take a look at Google Cloud Database which is a better option and has per-operation pricing instead of per-node pricing.
- luhn 9y ago
  That is very attractive pricing. It looks like you don't even need to preallocate capacity, just read and write as you please.

luhn 9y ago

Related is The Million Dollar Engineering Problem [1] from Segment, which showed up on HN a few months ago. They shaved $300K from their annual AWS spend by identifying and dealing with hot shards on DynamoDB. [1] https://segment.com/blog/the-million-dollar-eng-problem/ https://segment.com/blog/the-million-dollar-eng-problem/

pishpash 9y ago

NoSQL should stay as a quick and dirty solution for storing key-values. When you start to schematize it and use it as a "DB," you are going down the wrong path. That is because NoSQL is a glorified cache, not a DB. It is essentially using the memory on a large number of nodes to buffer bursty throughput, and using background processes to collate the data later onto disk. There is almost no case where an explicit distributed caching or queuing solution backed by a traditional DB isn't strictly better.

fennecfoxen 9y ago
> It is essentially using the memory on a large number of nodes to buffer bursty throughput, and using background processes to collate the data later onto disk. You're not describing NoSQL. You're describing something like MongoDB specifically. You're right about the specific case, wrong in the general case. > There is almost no case where an explicit distributed caching or queuing solution backed by a traditional DB isn't strictly better. Bah. If you're looking at big enough data sets (think "Walmart scale", not "web scale") then you have to design your data store to cope with this volume. First, it has to be replicated to multiple servers to serve more queries and for reliability, and is an eventually consistent system a consequence (basically any time you use your read replica, or do anything on a multimaster configuration). You'll need something shardable or multimaster to achieve horizontally scalablity and you'll probably also lose your foreign key enforcement and uniqueness constraints when you do this, so you might as well go the distance and fully denormalise everything for consistent access times. (Have you ever had the Postgres query planner suddenly make something take 10x longer than it usually does because your data patterns changed and decided to use a different query plan than it's ever used before? Yeah, that's a world of fun. Now make that thing a time-critical daily batch job and really squirm.) When you do the things to make this work, you're basically restricting yourself, abandoning database features. As a consequence, you are going to be in for a world of pain, period, sorry. One key advantage of good NoSQL software is that it doesn't give you those features to begin with, so it's harder to get it wrong. It also has data structures designed to make this sort of thing work well and be performant in ways that general purpose databases won't be, on account of being designed for general purposes.

abalone 9y ago

So, Amazon actually kind of agrees! They talked about this very issue of hot keys and overprovisioning at their most recent conference. The thing is it was buried in a session on Aurora and they didn't mention DynamoDB by name -- they just called it nosql -- but they noted that a customer cut their costs 40% by moving to Aurora. Because it does automatic heat management and just bills you for what IO you use. This is somewhat at odds with their top-level messaging which still pushes DynamoDB as the most scalable solution. And perhaps it is... there are some scalability limits to Aurora. Writes are bottlenecked by one instance. 64TB max. I think performance drops when you exceed the in memory cache. But those limits are still quite large. Basically I sense some tension between the DynamoDB and Aurora teams and I wonder where this is all going to shake out in the long run. Here's the full quote (I transcribed it so may contain errors): "The one thing that surprised me is that there are some customers who are moving their nosql workload to aurora. There are two reasons for that. One, it’s a lot easier to use Aurora because it’s mysql compatible compared to nosql because the interfaces and transaction characteristics are so different. What is also interesting is people also saved money because the IO cost is much lower. In no SQL if you have a large table it gets partitioned then the IO gets partitioned across all the table partitions that you have. And if you have one partition that is hot then you have to provision based on the IO requirement of the hot partition. In the case of Aurora we do automatic heat management so we don’t have this hot parition issue. Second we don’t charge based on provisioned IO. It’s only the IO that you use. And that actually saves a lot of money. In this particular case this is a big social company, interaction company, I cannot tell the name, and they reduced their operational costs by 40% by moving from nosql to Aurora" [1] [1] https://youtu.be/60QumD2QsF0?t=17m01s https://youtu.be/60QumD2QsF0?t=17m01s

datashovel 9y ago

Generally speaking. Given that DynamoDB is a NoSQL database service, I'm not certain that moving larger clients to their own dedicated AWS resources should cause too many negative side effects. Especially ones who are so large they're causing scalability issues.

rushi_agrawal 9y ago

If DynamoDB guys can come up with a mechanism similar to their CPU credits concept, that'd be a really nice feature to have. Of course it can't be as straightforward as CPU credits.