13 ms·

Some notes on DynamoDB 2022 paper

132 points by c4pt0r 4y ago

simlevesque 4y ago
The link does not work for me.
- c4pt0r 4y ago
  https://www.notion.so/Some-notes-on-DynamoDB-2022-72a2a1f22596452f87ab2c43ec810a5b https://www.notion.so/Some-notes-on-DynamoDB-2022-72a2a1f225... I also put a copy on notion, in case the link doesn't work.
- nighthawk454 4y ago
  working fine here
  - jerryjerryjerry 4y ago
    Same here.
- binwiederhier 4y ago
  Oh my. The author put his website on a host ame that includes an underscore: http://_.0xffff.me/dynamodb2022.html http://_.0xffff.me/dynamodb2022.html While underscores are valid hostnames per the DNS spec, they are not valid for hostnames in URLs. Firefox honors the HTTP spec and fails the request, but Chrome seems more lenient and displays the page. To the author. Please put your site on a valid hostname. Edit: Better explanation: https://stackoverflow.com/a/2183140 https://stackoverflow.com/a/2183140
  - philipkglass 4y ago
    Firefox honors the HTTP spec and fails the request, but Chrome seems more lenient and displays the page. I just successfully opened this link in Firefox 103.
    - LilBytes 4y ago
      Doesn't work on the Firefox android app.
      - binwiederhier 4y ago
        Yeah that's what I tested with. Firefox (desktop) works. Firefox mobile (Android) does not.
  - buzer 4y ago
    Works for me on Firefox nightly. One thing to note is that it's HTTP-only site, so if you have some kind of extension/setting to force HTTPS it's not going to work.
    - binwiederhier 4y ago
      It doesn't work with Firefox mobile (Android).
deleted 4y ago
[deleted]
flakiness 4y ago
Related: "The DynamoDB Paper" https://news.ycombinator.com/item?id=32094046 https://news.ycombinator.com/item?id=32094046
tonyxq 4y ago
xthrowawayxx 4y ago
Here's my notes on DynamoDB: How to spend $100k on what would cost $10k with an sql server for a 100x worse service
- atwood22 4y ago
  A master carpenter always blames his tools.
  - btown 4y ago
    A master carpenter is knowledgeable enough to be critical of those who aggressively peddle tools to unsuspecting customers who are likely to be unfamiliar with just how dangerous and potentially project-destroying those tools can be when not the right tools for the job.
  - otterley 4y ago
    That’s not the saying.
- icedchai 4y ago
  You need to use the right tool for the right job. I know people using DynamoDB for a tiny dataset that would easily fit in sqlite (or any other DB) running on a $20/month VPS. That wouldn't be serverless, of course, so it's a no-go.
  - cebert 4y ago
    That same dataset cloud then be modeled and stored in DynamoDB for even leas than that, right?
    - BreakfastB0b 4y ago
      Yeah I have no idea what icedchai is talking about, DynamoDB free tier is super generous https://aws.amazon.com/dynamodb/pricing/on-demand/ https://aws.amazon.com/dynamodb/pricing/on-demand/. It's going to cost you nothing until you have enough customers to afford to pay for it. Correctly modelling single table design on the other hand ...
gime_tree_fiddy 4y ago
> because it doesn't need to, Shared Nothing systems look similar, and the authors know exactly who the readers of this paper are ^_^) I didn't understand this. Who is the author referring to, and what is he implying?
- c4pt0r 4y ago
  I think a lot of shared-nothing systems are similar from high level, connection handling / storage node (with small shards) / Metadata (routing)
shepherdjerred 4y ago
> In 2021, during the 66-hour Amazon Prime Day shopping event, Amazon systems - including Alexa, the Amazon.com sites, and Amazon fulfill- ment centers, made trillions of API calls to DynamoDB, peak- ing at 89.2 million requests per second, while experiencing high availability with single-digit millisecond performance The scale that DDB operates at is mind-boggling. Where would someone even start when designing a system that can handle nearly 100 million requests per second?
- ZephyrBlu 4y ago
  Definitely insane scale, but I wonder how much of that is horizontal scaling.
  - yazaddaruvala 4y ago
    Almost all of it. “During Prime day” implies this is all read dominated traffic. The RCUs are all going to be provisioned so there are appropriate replicas pre-created in the correct regions. Disclaimer: Used to work at Amazon.
- jjtheblunt 4y ago
  a multicore machine? perhaps realized across physical cpus and os instances, or tons of memory with a high core count cpu?
- Gh0stRAT 4y ago
  Yeah, it totally blew my mind when I first heard these performance numbers as well. RE "nearly 100 million requests per second": as of Prime Day 2022, it has actually handled >100 Mreq/sec! >Over the course of Prime Day, these sources made trillions of calls to the DynamoDB API. DynamoDB maintained high availability while delivering single-digit millisecond responses and peaking at 105.2 million requests per second. [0] [0] https://aws.amazon.com/blogs/aws/amazon-prime-day-2022-aws-for-the-win/ https://aws.amazon.com/blogs/aws/amazon-prime-day-2022-aws-f...
- potamic 4y ago
  You can easily hit 100k rps on a typical NOSQL cluster. Scaling it to 100 million is just a matter of running a few thousand instances. Of course, operating a system with thousands of nodes is an engineering feat, but from a design perspective it's not super complicated.
samsquire 4y ago
I wrote a toy database that can be queried similar to dynamodb HTTPS://GitHub.com/samsquire/hash-db It's a trie in front of a HashMap.
- eatonphil 4y ago
  Wow this is a really, really cool project!
  - samsquire 4y ago
    Wow you are so kind. Thank you! The code I tried to keep simple and I tried to keep it small. I'm trying to do the most basic thing that shall work. I want to add document storage and unify the storage mechanism so the database can be multimodel like OrientDB and ArrangoDB. So graphs should be stored same way as documents and SQL. Currently the graph data model is separate from the SQL data model so you cannot query a graph as SQL or vice versa.

132 points by c4pt0r 4y ago

simlevesque 4y ago

The link does not work for me.

c4pt0r 4y ago
https://www.notion.so/Some-notes-on-DynamoDB-2022-72a2a1f22596452f87ab2c43ec810a5b https://www.notion.so/Some-notes-on-DynamoDB-2022-72a2a1f225... I also put a copy on notion, in case the link doesn't work.
nighthawk454 4y ago
working fine here
- jerryjerryjerry 4y ago
  Same here.
binwiederhier 4y ago
Oh my. The author put his website on a host ame that includes an underscore: http://_.0xffff.me/dynamodb2022.html http://_.0xffff.me/dynamodb2022.html While underscores are valid hostnames per the DNS spec, they are not valid for hostnames in URLs. Firefox honors the HTTP spec and fails the request, but Chrome seems more lenient and displays the page. To the author. Please put your site on a valid hostname. Edit: Better explanation: https://stackoverflow.com/a/2183140 https://stackoverflow.com/a/2183140
- philipkglass 4y ago
  Firefox honors the HTTP spec and fails the request, but Chrome seems more lenient and displays the page. I just successfully opened this link in Firefox 103.
  - LilBytes 4y ago
    Doesn't work on the Firefox android app.
    - binwiederhier 4y ago
      Yeah that's what I tested with. Firefox (desktop) works. Firefox mobile (Android) does not.
- buzer 4y ago
  Works for me on Firefox nightly. One thing to note is that it's HTTP-only site, so if you have some kind of extension/setting to force HTTPS it's not going to work.
  - binwiederhier 4y ago
    It doesn't work with Firefox mobile (Android).

deleted 4y ago

[deleted]

flakiness 4y ago

Related: "The DynamoDB Paper" https://news.ycombinator.com/item?id=32094046 https://news.ycombinator.com/item?id=32094046

tonyxq 4y ago

xthrowawayxx 4y ago

Here's my notes on DynamoDB: How to spend $100k on what would cost $10k with an sql server for a 100x worse service

atwood22 4y ago
A master carpenter always blames his tools.
- btown 4y ago
  A master carpenter is knowledgeable enough to be critical of those who aggressively peddle tools to unsuspecting customers who are likely to be unfamiliar with just how dangerous and potentially project-destroying those tools can be when not the right tools for the job.
- otterley 4y ago
  That’s not the saying.
icedchai 4y ago
You need to use the right tool for the right job. I know people using DynamoDB for a tiny dataset that would easily fit in sqlite (or any other DB) running on a $20/month VPS. That wouldn't be serverless, of course, so it's a no-go.
- cebert 4y ago
  That same dataset cloud then be modeled and stored in DynamoDB for even leas than that, right?
  - BreakfastB0b 4y ago
    Yeah I have no idea what icedchai is talking about, DynamoDB free tier is super generous https://aws.amazon.com/dynamodb/pricing/on-demand/ https://aws.amazon.com/dynamodb/pricing/on-demand/. It's going to cost you nothing until you have enough customers to afford to pay for it. Correctly modelling single table design on the other hand ...

gime_tree_fiddy 4y ago

> because it doesn't need to, Shared Nothing systems look similar, and the authors know exactly who the readers of this paper are ^_^) I didn't understand this. Who is the author referring to, and what is he implying?

c4pt0r 4y ago
I think a lot of shared-nothing systems are similar from high level, connection handling / storage node (with small shards) / Metadata (routing)

shepherdjerred 4y ago

> In 2021, during the 66-hour Amazon Prime Day shopping event, Amazon systems - including Alexa, the Amazon.com sites, and Amazon fulfill- ment centers, made trillions of API calls to DynamoDB, peak- ing at 89.2 million requests per second, while experiencing high availability with single-digit millisecond performance The scale that DDB operates at is mind-boggling. Where would someone even start when designing a system that can handle nearly 100 million requests per second?

ZephyrBlu 4y ago
Definitely insane scale, but I wonder how much of that is horizontal scaling.
- yazaddaruvala 4y ago
  Almost all of it. “During Prime day” implies this is all read dominated traffic. The RCUs are all going to be provisioned so there are appropriate replicas pre-created in the correct regions. Disclaimer: Used to work at Amazon.
jjtheblunt 4y ago
a multicore machine? perhaps realized across physical cpus and os instances, or tons of memory with a high core count cpu?
Gh0stRAT 4y ago
Yeah, it totally blew my mind when I first heard these performance numbers as well. RE "nearly 100 million requests per second": as of Prime Day 2022, it has actually handled >100 Mreq/sec! >Over the course of Prime Day, these sources made trillions of calls to the DynamoDB API. DynamoDB maintained high availability while delivering single-digit millisecond responses and peaking at 105.2 million requests per second. [0] [0] https://aws.amazon.com/blogs/aws/amazon-prime-day-2022-aws-for-the-win/ https://aws.amazon.com/blogs/aws/amazon-prime-day-2022-aws-f...
potamic 4y ago
You can easily hit 100k rps on a typical NOSQL cluster. Scaling it to 100 million is just a matter of running a few thousand instances. Of course, operating a system with thousands of nodes is an engineering feat, but from a design perspective it's not super complicated.

samsquire 4y ago

I wrote a toy database that can be queried similar to dynamodb HTTPS://GitHub.com/samsquire/hash-db It's a trie in front of a HashMap.

eatonphil 4y ago
Wow this is a really, really cool project!
- samsquire 4y ago
  Wow you are so kind. Thank you! The code I tried to keep simple and I tried to keep it small. I'm trying to do the most basic thing that shall work. I want to add document storage and unify the storage mechanism so the database can be multimodel like OrientDB and ArrangoDB. So graphs should be stored same way as documents and SQL. Currently the graph data model is separate from the SQL data model so you cannot query a graph as SQL or vice versa.