3 ms·

Asynchronous API with DynamoDB Streams

29 points by kiyanwang 5y ago

deleted 5y ago
[deleted]
rlayton2 5y ago
What are some general patterns for retreiving the data, client side, for async apis like this? I.e. client makes a long request in browser, then gets back a response saying the job is logged. The task completes in the background and then what should happen? I am thinking of JavaScript but more concerned about the patterns.
- furstenheim 5y ago
  Something that work for me, have a websocket open to the client and notify him that it's done. Additionally register this notifications in the database, so that if the user had closed the browser he'll see the notification next time that he logs in
  - cpursley 5y ago
    Supabase does something like this.
    - burggraf 5y ago
      Yes, look at the Supabase Realtime API here: https://supabase.io/docs/reference/javascript/subscribe https://supabase.io/docs/reference/javascript/subscribe https://supabase.io/docs/reference/javascript/removesubscription https://supabase.io/docs/reference/javascript/removesubscrip... https://supabase.io/docs/reference/javascript/getsubscriptions https://supabase.io/docs/reference/javascript/getsubscriptio...
- bionhoward 5y ago
  one option to avoid client waiting around is to make client-side and server-side queues and maintain an open websocket, then send messages as needed, and have an optimistic switch statement which performs the correct state transition given a certain request and arguments (see useReducer or Redux, zustand, etc). If the client is offline, then the user requests just build up in the queue, but the local state keeps updating so you can use your app. Then when the net is up, the queue of requests can be sent to the server to update the remote data store. A big problem is inconsistency, for example if you're making local changes and somebody else makes a remote change to the same data. For that reason the community seems to look at CRDTs, which are just data-structures with explicit order, or where order doesn't matter, because then no matter how changes happen, we can merge them all into an 'eventually-consistent' thing. IMHO this is a pretty huge amount of complexity to add to your app so if you can avoid it by doing immediate request/response processing it would simplify your stuff dramatically... but "offline first" is pretty cool for certain apps, and some jobs just take way too long. Queueing helps de-couple frontend from backend. That can be useful for zero-downtime database migrations and maintenance, if your client still works when the server is down... Tools for this, are newer, and evolving rapidly, I haven't seen a whole lot which I consider super mature and impressive...anybody else find a good offline-first state framework?
tader 5y ago
Interesting! (Typo? Should be `if (error)`?)
gunnarmorling 5y ago
That's a really use case for DynamoDB Streams! Shameless plug: if you'd like to implement this pattern with other databases such as Postgres or MySQL, take a look at Debezium (debezium.io), which provides open-source change data capture support for a range of databases. Mostly, this is used with Apache Kafka, but you also can run Debezium embedded into your JVM application. Probably should do a blog post too showing how to integrate that with AWS Lambda :) Disclaimer: I work on Debezium
- lbhdc 5y ago
  Can you recommend a guide for embedding Debezium?
  - gunnarmorling 5y ago
    The reference documentation is a good starting point: https://debezium.io/documentation/reference/1.5/development/engine.html https://debezium.io/documentation/reference/1.5/development/.... It may also be interesting to look into the implementation of Debezium Server (https://github.com/debezium/debezium/tree/master/debezium-server https://github.com/debezium/debezium/tree/master/debezium-se...), which is based on the embedded engine and allows to propagate change events to streaming platforms other than Kafka, such as Apache Pulsar, AWS Kinesis, GCP Cloud Pub/Sub, etc.

29 points by kiyanwang 5y ago

deleted 5y ago

[deleted]

What are some general patterns for retreiving the data, client side, for async apis like this? I.e. client makes a long request in browser, then gets back a response saying the job is logged. The task completes in the background and then what should happen? I am thinking of JavaScript but more concerned about the patterns.

furstenheim 5y ago
Something that work for me, have a websocket open to the client and notify him that it's done. Additionally register this notifications in the database, so that if the user had closed the browser he'll see the notification next time that he logs in
- cpursley 5y ago
  Supabase does something like this.
  - burggraf 5y ago
    Yes, look at the Supabase Realtime API here: https://supabase.io/docs/reference/javascript/subscribe https://supabase.io/docs/reference/javascript/subscribe https://supabase.io/docs/reference/javascript/removesubscription https://supabase.io/docs/reference/javascript/removesubscrip... https://supabase.io/docs/reference/javascript/getsubscriptions https://supabase.io/docs/reference/javascript/getsubscriptio...
bionhoward 5y ago
one option to avoid client waiting around is to make client-side and server-side queues and maintain an open websocket, then send messages as needed, and have an optimistic switch statement which performs the correct state transition given a certain request and arguments (see useReducer or Redux, zustand, etc). If the client is offline, then the user requests just build up in the queue, but the local state keeps updating so you can use your app. Then when the net is up, the queue of requests can be sent to the server to update the remote data store. A big problem is inconsistency, for example if you're making local changes and somebody else makes a remote change to the same data. For that reason the community seems to look at CRDTs, which are just data-structures with explicit order, or where order doesn't matter, because then no matter how changes happen, we can merge them all into an 'eventually-consistent' thing. IMHO this is a pretty huge amount of complexity to add to your app so if you can avoid it by doing immediate request/response processing it would simplify your stuff dramatically... but "offline first" is pretty cool for certain apps, and some jobs just take way too long. Queueing helps de-couple frontend from backend. That can be useful for zero-downtime database migrations and maintenance, if your client still works when the server is down... Tools for this, are newer, and evolving rapidly, I haven't seen a whole lot which I consider super mature and impressive...anybody else find a good offline-first state framework?

tader 5y ago

Interesting! (Typo? Should be `if (error)`?)

gunnarmorling 5y ago

That's a really use case for DynamoDB Streams! Shameless plug: if you'd like to implement this pattern with other databases such as Postgres or MySQL, take a look at Debezium (debezium.io), which provides open-source change data capture support for a range of databases. Mostly, this is used with Apache Kafka, but you also can run Debezium embedded into your JVM application. Probably should do a blog post too showing how to integrate that with AWS Lambda :) Disclaimer: I work on Debezium

lbhdc 5y ago
Can you recommend a guide for embedding Debezium?
- gunnarmorling 5y ago
  The reference documentation is a good starting point: https://debezium.io/documentation/reference/1.5/development/engine.html https://debezium.io/documentation/reference/1.5/development/.... It may also be interesting to look into the implementation of Debezium Server (https://github.com/debezium/debezium/tree/master/debezium-server https://github.com/debezium/debezium/tree/master/debezium-se...), which is based on the embedded engine and allows to propagate change events to streaming platforms other than Kafka, such as Apache Pulsar, AWS Kinesis, GCP Cloud Pub/Sub, etc.