5 ms·

One of the really cool DynamoDB features I love (at least in theory) is CDC / Streams. Also the fact it automagically hooks up to Kinesis is neat. Unfortunately

by sargun 6y ago

One of the really cool DynamoDB features I love (at least in theory) is CDC / Streams. Also the fact it automagically hooks up to Kinesis is neat. Unfortunately, for personal projects, this can lead to runaway spending. Does Fauna have strongly ordered CDC stream?

pier25 6y ago
Yeah they recently announced strongly consistent streaming. https://fauna.com/blog/live-ui-updates-with-faunas-real-time-document-streaming https://fauna.com/blog/live-ui-updates-with-faunas-real-time...
jcims 6y ago
I think the pricing of DynamoDB is the killer for personal projects. I ran it as a bit of a persistent cache one night and ran up $60 in charges.
- ledauphin 6y ago
  this is pretty weird for me to hear, as a developer on a project with 3000+ monthly average users where DynamoDB is my only database and the costs are in the hundreds per month. I'm sure that any database can be "too expensive" if your access patterns are out-of-this-world intense, but it's very difficult to compare apples to apples when we have no idea how beefy the PostreSQL instance to support your cache would have been.
  - pier25 6y ago
    > DynamoDB is my only database and the costs are in the hundreds per month Is your use case very DB intense? Seems quite high just to support a couple thousand users.
    - redis_mlc 6y ago
      Most busy sites have to put Redis/Elasticache in front of Dynamodb, or do app caching of hot keys, to save money. Typically 90% less.
    - ledauphin 6y ago
      pretty intense, yes. for each user who uses it, they take several hundreds of actions per day that each require a database write. we could optimize those but it's not worth it.
      - tluyben2 6y ago
        Like you said, hard to compare apples-apples without more info but couple 1000 users with 100s of writes/user/day sounds like it would handle fine on a $1/month vps (we have far higher workloads running few $1/mo, if I understand you correctly). Not saying that you should spend time on that; it's probably insignificant on the p&l, however, other people should know there are alternatives still and aws is not always, automatically, the right choice (in my mind, it almost never is actually).
        pier25 6y ago
        > sounds like it would handle fine on a $1/month vps You need someone to manage that VPS though and that costs a lot more than $1 per month.
        tluyben2 6y ago
        How is that different from managing aws? Because in my experience (of 20+ years) there isn't any for most projects. There are reasons to pick aws, but, for most of them, this is not one of them.
        pier25 6y ago
        I guess it depends on what AWS services you're using and how complex is your usage. If you're only using Dynamo you shouldn't need that much management, or if you're using Fauna you need zero management.
  - jcims 6y ago
    This was for some analytics against WARC files with 60 container tasks running for ~5 hrs
databrecht 6y ago
If I'm not mistaking, Dynamo's streaming is pulling under the hood. I suppose it depends on how frequently it pulls how quickly the price goes up and how expensive such a pull is in reads. In Fauna we have temporality as a first-class citizen. You can get efficient and cheap changesets by leveraging temporality since you can just ask: "what has been added/removed in this collection or index match after timestamp X" and can combine that by writing an index that delivers you the answer to "what are the updated documents in a collection after a certain timestamp?". That brings you very cheap pull-based CDC. We recently introduced a second possibility that allows for push-based streaming for documents. Document-based streaming allows you to open separate streams for multiple existing documents to get updates on those. This is only the first phase, sets (such as index matches or whole collections) are coming up. Streaming becomes cheaper if you want your data to be really life (<1s) which is excellent for UI redraws but could potentially also be used for CDC (probably in combination with the temporal features if you need to restart streams). Both push-based as pull-based are strongly ordered. I describe how to get the query for the pull-based approach here: https://forums.fauna.com/t/example-custom-subscription-function/267 https://forums.fauna.com/t/example-custom-subscription-funct... And a blog on the streaming API can be found here: https://fauna.com/blog/live-ui-updates-with-faunas-real-time-document-streaming https://fauna.com/blog/live-ui-updates-with-faunas-real-time... Once set streaming (next to document streaming) is out as well, we have two strong solutions and you can choose what suits you best and what is most efficient for your use case. Do you want instant UI redraws? Use push-based, are you pulling a changeset every houre? Use pull-based.

by sargun 6y ago

pier25 6y ago

Yeah they recently announced strongly consistent streaming. https://fauna.com/blog/live-ui-updates-with-faunas-real-time-document-streaming https://fauna.com/blog/live-ui-updates-with-faunas-real-time...

jcims 6y ago

I think the pricing of DynamoDB is the killer for personal projects. I ran it as a bit of a persistent cache one night and ran up $60 in charges.

ledauphin 6y ago
this is pretty weird for me to hear, as a developer on a project with 3000+ monthly average users where DynamoDB is my only database and the costs are in the hundreds per month. I'm sure that any database can be "too expensive" if your access patterns are out-of-this-world intense, but it's very difficult to compare apples to apples when we have no idea how beefy the PostreSQL instance to support your cache would have been.
- pier25 6y ago
  > DynamoDB is my only database and the costs are in the hundreds per month Is your use case very DB intense? Seems quite high just to support a couple thousand users.
  - redis_mlc 6y ago
    Most busy sites have to put Redis/Elasticache in front of Dynamodb, or do app caching of hot keys, to save money. Typically 90% less.
  - ledauphin 6y ago
    pretty intense, yes. for each user who uses it, they take several hundreds of actions per day that each require a database write. we could optimize those but it's not worth it.
    - tluyben2 6y ago
      Like you said, hard to compare apples-apples without more info but couple 1000 users with 100s of writes/user/day sounds like it would handle fine on a $1/month vps (we have far higher workloads running few $1/mo, if I understand you correctly). Not saying that you should spend time on that; it's probably insignificant on the p&l, however, other people should know there are alternatives still and aws is not always, automatically, the right choice (in my mind, it almost never is actually).
      - pier25 6y ago
        > sounds like it would handle fine on a $1/month vps You need someone to manage that VPS though and that costs a lot more than $1 per month.
        tluyben2 6y ago
        How is that different from managing aws? Because in my experience (of 20+ years) there isn't any for most projects. There are reasons to pick aws, but, for most of them, this is not one of them.
        pier25 6y ago
        I guess it depends on what AWS services you're using and how complex is your usage. If you're only using Dynamo you shouldn't need that much management, or if you're using Fauna you need zero management.
- jcims 6y ago
  This was for some analytics against WARC files with 60 container tasks running for ~5 hrs

databrecht 6y ago

If I'm not mistaking, Dynamo's streaming is pulling under the hood. I suppose it depends on how frequently it pulls how quickly the price goes up and how expensive such a pull is in reads. In Fauna we have temporality as a first-class citizen. You can get efficient and cheap changesets by leveraging temporality since you can just ask: "what has been added/removed in this collection or index match after timestamp X" and can combine that by writing an index that delivers you the answer to "what are the updated documents in a collection after a certain timestamp?". That brings you very cheap pull-based CDC. We recently introduced a second possibility that allows for push-based streaming for documents. Document-based streaming allows you to open separate streams for multiple existing documents to get updates on those. This is only the first phase, sets (such as index matches or whole collections) are coming up. Streaming becomes cheaper if you want your data to be really life (<1s) which is excellent for UI redraws but could potentially also be used for CDC (probably in combination with the temporal features if you need to restart streams). Both push-based as pull-based are strongly ordered. I describe how to get the query for the pull-based approach here: https://forums.fauna.com/t/example-custom-subscription-function/267 https://forums.fauna.com/t/example-custom-subscription-funct... And a blog on the streaming API can be found here: https://fauna.com/blog/live-ui-updates-with-faunas-real-time-document-streaming https://fauna.com/blog/live-ui-updates-with-faunas-real-time... Once set streaming (next to document streaming) is out as well, we have two strong solutions and you can choose what suits you best and what is most efficient for your use case. Do you want instant UI redraws? Use push-based, are you pulling a changeset every houre? Use pull-based.