3 ms·

What I would love is a turnkey service I could set up with logical replication and a straightforward configuration that can listen for any changes on certain ta

What I would love is a turnkey service I could set up with logical replication and a straightforward configuration that can listen for any changes on certain tables (ideally with optional filters) that will take those changes and put them in a queue / ESB / whatever. If youre designing a new application from scratch and you have only one place that certain logical changes happen you can do this from the start. But in any system not designed that way from the start, the database is the place where all things go. Using that as your event source allows everything to keep on working like they always have but allow you to now notify new processes when something changes.

joelhaasnoot 3y ago
- https://hevodata.com/ https://hevodata.com/ - https://airbyte.com/ https://airbyte.com/ - https://www.stitchdata.com/ https://www.stitchdata.com/ Unfortunately none of them are perfect as your data scales, each have up and downsides...
cpursley 3y ago
If you're using Elixir, check out https://github.com/cpursley/walex https://github.com/cpursley/walex I've actually been thinking about turning this idea into a product where you can just point it at your postgres database and select the tables you want to listen to (with filters, like you describe). And have that forwarded to a webhook (with possibility of other protocols like websockets). I'd love to hear folks thoughts on that (and if it would be something people would pay for). And if anyone might want to partner up on this.
- evntdrvn 3y ago
  What advantages does this have compared to Debezium?
  - cpursley 3y ago
    You wouldn't have to host it yourself (not everyone knows how or wants to run a heavy Java app). Add your database string, a couple extra migrations, your webhook endpoint - and you're off to the races. Target customer would be low-code space (think admin dashboards on postgres, webhook integration tools).
- chasers 3y ago
  Yo :D This is what Supabase Realtime does! https://github.com/supabase/realtime https://github.com/supabase/realtime Spin up a Supabase database and then subscribe to changes with WebSockets. You can play with it here once you have a db: https://realtime.supabase.com/inspector/new https://realtime.supabase.com/inspector/new
  - cpursley 3y ago
    Yeah, Supabase is awesome. In fact, WalEx originally started out as stolen code from realtime so I could do stuff right in Elixir (biz logic).
alibero 3y ago
I used to work at Yelp, which had something that I think it similar to what you are describing called Data Pipeline (https://engineeringblog.yelp.com/2019/12/cassandra-source-connector-part-1.html https://engineeringblog.yelp.com/2019/12/cassandra-source-co...). I remember it being pretty simple (like, run one or two bash commands) to get a source table streamed into a kafka topic, or get a kafka topic streamed into a sink datastore (S3, mysql, cassandra, redshift, etc). Kafka topics can also be filtered/transformed pretty easily. E.g. in https://engineeringblog.yelp.com/2021/04/powering-messaging-enabledness-with-yelps-data-infrastructure.html https://engineeringblog.yelp.com/2021/04/powering-messaging-... they run `datapipe datalake add-connection --namespace main --source message_enabledness`, which results in the `message_enabledness` table being streamed into a (daily?) parquet snapshot in S3, registered in AWS Glue. It is open source but it's more of the "look at how we did this" open source VS the "it would be easy to stick this into your infra and use it" kind of open source :(
bzzzt 3y ago
Something like Debezium? https://debezium.io https://debezium.io

joelhaasnoot 3y ago

- https://hevodata.com/ https://hevodata.com/ - https://airbyte.com/ https://airbyte.com/ - https://www.stitchdata.com/ https://www.stitchdata.com/ Unfortunately none of them are perfect as your data scales, each have up and downsides...

cpursley 3y ago

If you're using Elixir, check out https://github.com/cpursley/walex https://github.com/cpursley/walex I've actually been thinking about turning this idea into a product where you can just point it at your postgres database and select the tables you want to listen to (with filters, like you describe). And have that forwarded to a webhook (with possibility of other protocols like websockets). I'd love to hear folks thoughts on that (and if it would be something people would pay for). And if anyone might want to partner up on this.

evntdrvn 3y ago
What advantages does this have compared to Debezium?
- cpursley 3y ago
  You wouldn't have to host it yourself (not everyone knows how or wants to run a heavy Java app). Add your database string, a couple extra migrations, your webhook endpoint - and you're off to the races. Target customer would be low-code space (think admin dashboards on postgres, webhook integration tools).
chasers 3y ago
Yo :D This is what Supabase Realtime does! https://github.com/supabase/realtime https://github.com/supabase/realtime Spin up a Supabase database and then subscribe to changes with WebSockets. You can play with it here once you have a db: https://realtime.supabase.com/inspector/new https://realtime.supabase.com/inspector/new
- cpursley 3y ago
  Yeah, Supabase is awesome. In fact, WalEx originally started out as stolen code from realtime so I could do stuff right in Elixir (biz logic).

alibero 3y ago

I used to work at Yelp, which had something that I think it similar to what you are describing called Data Pipeline (https://engineeringblog.yelp.com/2019/12/cassandra-source-connector-part-1.html https://engineeringblog.yelp.com/2019/12/cassandra-source-co...). I remember it being pretty simple (like, run one or two bash commands) to get a source table streamed into a kafka topic, or get a kafka topic streamed into a sink datastore (S3, mysql, cassandra, redshift, etc). Kafka topics can also be filtered/transformed pretty easily. E.g. in https://engineeringblog.yelp.com/2021/04/powering-messaging-enabledness-with-yelps-data-infrastructure.html https://engineeringblog.yelp.com/2021/04/powering-messaging-... they run `datapipe datalake add-connection --namespace main --source message_enabledness`, which results in the `message_enabledness` table being streamed into a (daily?) parquet snapshot in S3, registered in AWS Glue. It is open source but it's more of the "look at how we did this" open source VS the "it would be easy to stick this into your infra and use it" kind of open source :(

bzzzt 3y ago

Something like Debezium? https://debezium.io https://debezium.io