7 ms·

Latency Comparison: DynamoDB vs. FaunaDB vs. Redis

57 points by noahfschr 5y ago

k__ 5y ago
Is this the price Fauna pays for its consistency guarantees?
- noahfschr 5y ago
  probably. https://blog.upstash.com/latency-comparison#why-is-faunadb-slow-in-this-benchmark https://blog.upstash.com/latency-comparison#why-is-faunadb-s...
- databrecht 5y ago
  No it's not, that price is far far smaller and should not impact pure reads. This is probably an artifact of an anti-pattern where the same documents are constantly updated which creates significant history. At this point, that can have an impact on the index. We are working on optimizing that in which case history will no longer have an impact on these latencies while retaining the possibility to go back in time or get changesets.
deleted 5y ago
[deleted]
ntoshev 5y ago
Network latency is not accounted for: they use AWS lambda calling DynamoDB / Redis in the same data center; Fauna endpoint is somewhere else. Curious what Firebase latency would be in comparison, when called from the same GCP data center or AWS.
- aeyes 5y ago
  According to the blog post this site runs in AWS us-west-1. With Fauna you don't know the data center but according to their status page they have infrastructure in AWS us-west-2. Latency from us-west-1 to us-west-2 is 25ms so you can subtract that from the total time. Fauna claims to route your request to the nearest data center so I'm interested in validating this. Seeing 400ms latency where I'd expect <50ms is important to me, especially on Lambda where you are billed waiting for the response.
  - databrecht 5y ago
    Which should be the case, depending on what you do you will probably experience 10-50ms read latencies. Look for Evan's answers why the measured values here are higher.
wging 5y ago
My suspicion is that this may not tell the full story. For example, availability-wise I bet there are differences between these databases. As just one example, I bet this person wasn't running with a multi-AZ setup for Upstash, since https://docs.upstash.com/overall/databasetypes https://docs.upstash.com/overall/databasetypes says "Multi Zone Replication" is a premium feature. Whereas DDB doesn't even let you store your data in a single AZ, AFAIK (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...). (My understanding is shallow compared to real experts, but even so I know this is a deep topic and this is only one example of the type of thing you'd want to consider when figuring out whether to take this comparison at face value.)
- astuyvenberg 5y ago
  From other comments, it appears OP is associated with upstash.
jph 5y ago
FaunaDB is doing more than Upstash and DynamoDB in the author's examples, as the author describes in the related blog post: - FaunaDB is providing strong consistency and isolation; Upstash and DynamoDB are providing eventual consistency. - FaunaDB is replicating the data worldwide and offering similar access everywhere; Upstash and DynamoDB are deliberately configured in the same AWS region as the lambda function.
- deleted 5y ago
  [deleted]
- k__ 5y ago
  DynamoDB can be used with strong consistency, would be interesting if it doubles the latency.
  - evanweaver 5y ago
    Dynamo strongly consistent transactions are still limited to a single region, and are eventually consistent outside of that region. For example, in Dynamo, it is not possible to enforce uniqueness via multi-region strongly consistent transactions. Fauna can do this. The Dynamo transactions do increase latency, but not to the same degree as Fauna. However, they are not achieving the same level of transactional correctness either, or really any correctness at all in a multi-region context.
    - k__ 5y ago
      Yes, I know. I was just interested if a ConsistentRead would change the latency.
      - evanweaver 5y ago
        It has to to acquire and release locks, so yes.
        deleted 5y ago
        [deleted]
pier25 5y ago
I've been using Fauna for a year or so. What's slowing Fauna here are the global writes and the non-idiomatic FQL queries. Right now the code is doing a bunch of separate queries, but in idiomatic FQL this would be done in a single transaction. Edit: I'm going to do a PR to update the FQL code if the author accepts it.
- noahfschr 5y ago
  Sure, I will.
  - pier25 5y ago
    Thanks for the merge. As you pointed in the PR you're only measuring the latency of the first query so it will have no effect in the benchmark. I'm guessing the "current request" latency will improve though, no? I have to say the latencies you're getting are much higher that I've experienced with Fauna. Obviously it's expected for a KV database to be faster, but I'd be surprised to get more than 100ms of 50th percentile.
    - noahfschr 5y ago
      Current request latency is also the read latency (from lambda function -> db). I should have been more clear on that.
      - pier25 5y ago
        Still, almost a second of total latency (50th percentile) is super high. I'd love to do the same test from Cloudflare Workers instead of AWS Lambda. Could you please DM on Twitter to discuss this? https://twitter.com/PierB https://twitter.com/PierB Edit: I just saw this comment by Evan from Fauna which explains why the latency could be so high: https://news.ycombinator.com/item?id=26806541 https://news.ycombinator.com/item?id=26806541
evanweaver 5y ago
As others have pointed out, measuring latency from an AWS Lambda function to a co-located single node in-memory non-durable key-value database (Redis), or to a co-located single AZ eventually consistent key-value database (DynamoDB), doesn't have anything to do with measuring client-observed latency from the browser to a globally distributed ACID-compliant document database. A similar process co-located with a Fauna region normally can also perform simple reads in small numbers of milliseconds. Similarly, a browser client querying a lambda function multiple times from the other side of the world will also be quite slow, even if the lambda "thinks" its queries are fast because its database is right next to it. It is not completely clear to me what else is going wrong, but the basic premise of the benchmark is invalid, and there are other errors in regard to Fauna. For example, index serializability only affects write throughput; it has nothing to do with read latency. The status page reports write latency, not read latency, etc. For a more even-handed comparison of Fauna to DynamoDB, see this blog post: https://fauna.com/blog/comparing-fauna-and-dynamodb-pricing-features https://fauna.com/blog/comparing-fauna-and-dynamodb-pricing-...
- mfateev 5y ago
  AWS DynamoDB is multi-AZ and is strongly consistent (at least for a single key update). I believe you confused it with a Dynamo DB described in a paper Amazon published long ago. AWS DynamoDB has nothing to do with a eventually consistent design the paper describes. It was purely marketing gimmick to call the new AWS Service that.
  - evanweaver 5y ago
    Please read the blog post: https://fauna.com/blog/comparing-fauna-and-dynamodb-pricing-features https://fauna.com/blog/comparing-fauna-and-dynamodb-pricing-...
    - k__ 5y ago
      Good article, thanks. I'm currently reading The DynamoDB Book, and even it's author acknowledges that DynamoDB is serverless more as a byproduct and shouldn't be used for that reason alone But as a product developer all these highly integrated AWS services that can be provisioned with CloudFormation are pretty convincing. What's Fauna's IaC story?
tirrex 5y ago
As a general rule in benchmarks, if product A does something in 1 ms while product B doing it in 500 ms, something is wrong. Either you misconfigured it or these products serve different purposes so they are not comparable. Either way, your benchmark is flawed unfortunately. As a general courtesy, when you see a dramatic difference, it’s better to involve parties or ask for review before publishing your results.
thdxr 5y ago
I would love to see more options in the serverless persistence space. Given the infinite combination of tradeoffs that exist with databases there's a lot of room for different solutions. AWS Aurora Serverless v2 is close to what I'm looking for but think it's held back by trying to make non-serverless technology serverless. Fauna is close but I don't really need global consistency and don't want to pay the 150ms latency price for it: https://status.fauna.com/ https://status.fauna.com/
- databrecht 5y ago
  Region selection is coming up if that interests you. We are actively working on it :)
  - thdxr 5y ago
    I need to do some benchmarking myself but it seems that even in a local region writes are in the 100s of ms. I'm aiming for my Lambda functions to be < 100ms so using Fauna seems difficult to work in. Unless you meant I can limit "how global" my data is and that would improve write speeds?
    - ec109685 5y ago
      The spanner approach could dynamically shift the leader location for parts of the key space, so writes that tend to be done from one location could avoid needing to communicate outside the local region.
void_mint 5y ago
Sorry, what is the overlap in usecase between DynamoDB vs. Redis vs. FaunaDB? There aren't many problems I would be evaluating either DynamoDB or Redis for - usually the problem domain is one or the other (an in-mem DB vs. a persistent store). I also have no idea what FaunaDB is, so I'm not sure how it compares to either.
agacera 5y ago
Comparing Fauna with DynamoDB and Redis gives the wrong impression. Trade-offs are totally different. However, the product in question (Upstash), looks very interesting: serverless storage with a pay as you go model with a maximum price cap is an excelent idea. I would love to have more options in this space, especially in GCP to use together with Cloud Run.
wgjordan 5y ago
I would be interested in seeing how DynamoDB fronted by DynamoDB Accelerator (DAX) performs in a latency comparison such as this.

57 points by noahfschr 5y ago

k__ 5y ago

Is this the price Fauna pays for its consistency guarantees?

noahfschr 5y ago
probably. https://blog.upstash.com/latency-comparison#why-is-faunadb-slow-in-this-benchmark https://blog.upstash.com/latency-comparison#why-is-faunadb-s...
databrecht 5y ago
No it's not, that price is far far smaller and should not impact pure reads. This is probably an artifact of an anti-pattern where the same documents are constantly updated which creates significant history. At this point, that can have an impact on the index. We are working on optimizing that in which case history will no longer have an impact on these latencies while retaining the possibility to go back in time or get changesets.

deleted 5y ago

[deleted]

ntoshev 5y ago

Network latency is not accounted for: they use AWS lambda calling DynamoDB / Redis in the same data center; Fauna endpoint is somewhere else. Curious what Firebase latency would be in comparison, when called from the same GCP data center or AWS.

aeyes 5y ago
According to the blog post this site runs in AWS us-west-1. With Fauna you don't know the data center but according to their status page they have infrastructure in AWS us-west-2. Latency from us-west-1 to us-west-2 is 25ms so you can subtract that from the total time. Fauna claims to route your request to the nearest data center so I'm interested in validating this. Seeing 400ms latency where I'd expect <50ms is important to me, especially on Lambda where you are billed waiting for the response.
- databrecht 5y ago
  Which should be the case, depending on what you do you will probably experience 10-50ms read latencies. Look for Evan's answers why the measured values here are higher.

wging 5y ago

My suspicion is that this may not tell the full story. For example, availability-wise I bet there are differences between these databases. As just one example, I bet this person wasn't running with a multi-AZ setup for Upstash, since https://docs.upstash.com/overall/databasetypes https://docs.upstash.com/overall/databasetypes says "Multi Zone Replication" is a premium feature. Whereas DDB doesn't even let you store your data in a single AZ, AFAIK (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...). (My understanding is shallow compared to real experts, but even so I know this is a deep topic and this is only one example of the type of thing you'd want to consider when figuring out whether to take this comparison at face value.)

astuyvenberg 5y ago
From other comments, it appears OP is associated with upstash.

jph 5y ago

FaunaDB is doing more than Upstash and DynamoDB in the author's examples, as the author describes in the related blog post: - FaunaDB is providing strong consistency and isolation; Upstash and DynamoDB are providing eventual consistency. - FaunaDB is replicating the data worldwide and offering similar access everywhere; Upstash and DynamoDB are deliberately configured in the same AWS region as the lambda function.

deleted 5y ago
[deleted]
k__ 5y ago
DynamoDB can be used with strong consistency, would be interesting if it doubles the latency.
- evanweaver 5y ago
  Dynamo strongly consistent transactions are still limited to a single region, and are eventually consistent outside of that region. For example, in Dynamo, it is not possible to enforce uniqueness via multi-region strongly consistent transactions. Fauna can do this. The Dynamo transactions do increase latency, but not to the same degree as Fauna. However, they are not achieving the same level of transactional correctness either, or really any correctness at all in a multi-region context.
  - k__ 5y ago
    Yes, I know. I was just interested if a ConsistentRead would change the latency.
    - evanweaver 5y ago
      It has to to acquire and release locks, so yes.
      - deleted 5y ago
        [deleted]

pier25 5y ago

I've been using Fauna for a year or so. What's slowing Fauna here are the global writes and the non-idiomatic FQL queries. Right now the code is doing a bunch of separate queries, but in idiomatic FQL this would be done in a single transaction. Edit: I'm going to do a PR to update the FQL code if the author accepts it.

noahfschr 5y ago
Sure, I will.
- pier25 5y ago
  Thanks for the merge. As you pointed in the PR you're only measuring the latency of the first query so it will have no effect in the benchmark. I'm guessing the "current request" latency will improve though, no? I have to say the latencies you're getting are much higher that I've experienced with Fauna. Obviously it's expected for a KV database to be faster, but I'd be surprised to get more than 100ms of 50th percentile.
  - noahfschr 5y ago
    Current request latency is also the read latency (from lambda function -> db). I should have been more clear on that.
    - pier25 5y ago
      Still, almost a second of total latency (50th percentile) is super high. I'd love to do the same test from Cloudflare Workers instead of AWS Lambda. Could you please DM on Twitter to discuss this? https://twitter.com/PierB https://twitter.com/PierB Edit: I just saw this comment by Evan from Fauna which explains why the latency could be so high: https://news.ycombinator.com/item?id=26806541 https://news.ycombinator.com/item?id=26806541

evanweaver 5y ago

As others have pointed out, measuring latency from an AWS Lambda function to a co-located single node in-memory non-durable key-value database (Redis), or to a co-located single AZ eventually consistent key-value database (DynamoDB), doesn't have anything to do with measuring client-observed latency from the browser to a globally distributed ACID-compliant document database. A similar process co-located with a Fauna region normally can also perform simple reads in small numbers of milliseconds. Similarly, a browser client querying a lambda function multiple times from the other side of the world will also be quite slow, even if the lambda "thinks" its queries are fast because its database is right next to it. It is not completely clear to me what else is going wrong, but the basic premise of the benchmark is invalid, and there are other errors in regard to Fauna. For example, index serializability only affects write throughput; it has nothing to do with read latency. The status page reports write latency, not read latency, etc. For a more even-handed comparison of Fauna to DynamoDB, see this blog post: https://fauna.com/blog/comparing-fauna-and-dynamodb-pricing-features https://fauna.com/blog/comparing-fauna-and-dynamodb-pricing-...

mfateev 5y ago
AWS DynamoDB is multi-AZ and is strongly consistent (at least for a single key update). I believe you confused it with a Dynamo DB described in a paper Amazon published long ago. AWS DynamoDB has nothing to do with a eventually consistent design the paper describes. It was purely marketing gimmick to call the new AWS Service that.
- evanweaver 5y ago
  Please read the blog post: https://fauna.com/blog/comparing-fauna-and-dynamodb-pricing-features https://fauna.com/blog/comparing-fauna-and-dynamodb-pricing-...
  - k__ 5y ago
    Good article, thanks. I'm currently reading The DynamoDB Book, and even it's author acknowledges that DynamoDB is serverless more as a byproduct and shouldn't be used for that reason alone But as a product developer all these highly integrated AWS services that can be provisioned with CloudFormation are pretty convincing. What's Fauna's IaC story?

tirrex 5y ago

As a general rule in benchmarks, if product A does something in 1 ms while product B doing it in 500 ms, something is wrong. Either you misconfigured it or these products serve different purposes so they are not comparable. Either way, your benchmark is flawed unfortunately. As a general courtesy, when you see a dramatic difference, it’s better to involve parties or ask for review before publishing your results.

thdxr 5y ago

I would love to see more options in the serverless persistence space. Given the infinite combination of tradeoffs that exist with databases there's a lot of room for different solutions. AWS Aurora Serverless v2 is close to what I'm looking for but think it's held back by trying to make non-serverless technology serverless. Fauna is close but I don't really need global consistency and don't want to pay the 150ms latency price for it: https://status.fauna.com/ https://status.fauna.com/

databrecht 5y ago
Region selection is coming up if that interests you. We are actively working on it :)
- thdxr 5y ago
  I need to do some benchmarking myself but it seems that even in a local region writes are in the 100s of ms. I'm aiming for my Lambda functions to be < 100ms so using Fauna seems difficult to work in. Unless you meant I can limit "how global" my data is and that would improve write speeds?
  - ec109685 5y ago
    The spanner approach could dynamically shift the leader location for parts of the key space, so writes that tend to be done from one location could avoid needing to communicate outside the local region.

void_mint 5y ago

Sorry, what is the overlap in usecase between DynamoDB vs. Redis vs. FaunaDB? There aren't many problems I would be evaluating either DynamoDB or Redis for - usually the problem domain is one or the other (an in-mem DB vs. a persistent store). I also have no idea what FaunaDB is, so I'm not sure how it compares to either.

agacera 5y ago

Comparing Fauna with DynamoDB and Redis gives the wrong impression. Trade-offs are totally different. However, the product in question (Upstash), looks very interesting: serverless storage with a pay as you go model with a maximum price cap is an excelent idea. I would love to have more options in this space, especially in GCP to use together with Cloud Run.

wgjordan 5y ago

I would be interested in seeing how DynamoDB fronted by DynamoDB Accelerator (DAX) performs in a latency comparison such as this.