shayonj

10 ms·

121.
by shayonj 2y ago
+1 for this being common knowledge but there is still decent bit mocking that happens IME.
122.
by shayonj 2y ago
Doesn’t directly answer your question but at least in Postgres I am curious about UNLOGGED mode and see if it results in faster specs. Trade off being, crash recovery doesn’t work but that’s fine in CI. There is also something to be said ab
123.
Database mocks are not worth it(shayon.dev)
178 points by shayonj 2y ago|262 comments
124.
by shayonj 2y ago
The true "intelligence" I want isn't in rewriting or summarizing emails and texts IMHO - that's actually confusing and frustrating from a product usability POV. What I'd truly love is the ability to dictate lists to
125.
by shayonj 2y ago
Thank you Bryce <3
126.
by shayonj 2y ago
> Scrolling through a list of UUIDs should be exciting! I wanna see bb166283-2e09-4b72-ba32-70a43521c70e, not 00000000-0000-4000-8000-000000000000! This was my favorite bit and speaks a lot about the craft behind this page! v cool <3
127.
by shayonj 2y ago
oof! that's rough. I was also not particularly impressed with "limitless". Esp. because you can't have partitions on the postgres limitless offering.
128.
by shayonj 2y ago
Very cool release! Checking it out now
129.
by shayonj 2y ago
Absolutely stoked for pg_mooncake. I really want to see some of these things happening inside PG and taking advantage of the PG internals + native storage. Only bummer is adoption by places where users are currently, say Aurora. But thats p
130.
by shayonj 2y ago
It depends how you are backing up to S3 and in what format. If you are using something like pg_dump, this tool may not make sense. But if you want to say sync to S3 in parquet or something like that, then it might. Parquet support is TBD fo
131.
by shayonj 2y ago
Thank you! Would love to receive feedback, if you are up for it.
132.
by shayonj 2y ago
Yeah, it would be, since its using logical replication. However, the standard logical replication limitation applies [1] pg_flo can perform common DDLs for you (not something that's in built in logical replication today, hopefully soon
133.
by shayonj 2y ago
Heya! v0.0.4 is now out with `--copy` support, which will stream all the data from the table and the transformation/routing will get applied before the changes land in destination. [1] https://github.com/shayonj/pg
134.
by shayonj 2y ago
Perf! From testing on some of my staging workloads, the footprint isn't too high and I can get 5-6k messages/s. Esp. since there is only one worker instance involved (for strict ordering). Yes, it does use NATS JetStream.
135.
by shayonj 2y ago
Yeah, totally fair. Are you ok with a NATS dependency ? Happy to work with you in supporting a new destination like ES. Also looking to make NATS optional for smaller/simpler setups ( https://github.com/shayonj/pg_f
136.
by shayonj 2y ago
Hi! thanks for checking it out. Mostly based on my own familiarity and interest at the time, especially with NATS Jetstream being out now. That said, depending on the feedback and roadmap evolves - I have thought about, say, supporting diff
137.
by shayonj 2y ago
Thanks for sharing. That is an interesting use case. I reckon you will be able to take advantage of the rules or routing feature in pg_flo. Is there some column that can help you identify the two separate types of deletion operations or per
138.
by shayonj 2y ago
I think PeerDB is a smart choice if you are looking to sync the data to Clickhouse. They have nice integration support as well. re: 2. I have some plans around a control plane that allows users to define these transformation rules and routi
139.
by shayonj 2y ago
Thanks for checking it out - Curious are you syncing/storing the data in some specific format in s3?
140.
by shayonj 2y ago
Sequin sounds like a nice fit for that. I like it and yeah it does have replay. With pg_flo, I am looking for the tool to be fairly hands off, while declarative at the same time for data filtering, transformation and re-routing like tasks,
141.
by shayonj 2y ago
re: worker/replicator dying, if the worker dies, nothing is basically reading from subject/queue (NATS), so the changes aren't reaching the destination. The default config is to have one worker. If the replicator dies, the ch
142.
by shayonj 2y ago
Thank you! Today, an active stream is resumeable, say in case a shutdown / SIGTERM come up. It also stores some state every flush. However, as for copy-and-stream, it’s not quite resumable today, but that's coming soon. Some more
143.
by shayonj 2y ago
amazing! Will plan a release by Tuesday latest. You can experiment with `--copy-and-stream ` in the meantime [1] [1] https://github.com/shayonj/pg_flo?tab=readme-ov-file#streami...
144.
by shayonj 2y ago
I think this also might be of interest to you - just one time copies and apply transformations - https://github.com/shayonj/pg_flo/issues/6 . I will look into shipping that very soon.
145.
by shayonj 2y ago
Thank you for giving this a spin. That's correct, today you'd need to drop the table before another sync with pg_flo. That said, I have given delta syncs some thought, and also looking into a control plane that can make some of th
146.
by shayonj 2y ago
Yeah, absolutely! There are other benefits of being an extension, and yes you can use pg_flo with any PostgreSQL database or service since it uses logical replication to listen on changes and CTIDs for bulk copies.
147.
by shayonj 2y ago
Thank you! Still very early days, would love to hear any feedback.
148.
pg_flo – Stream, transform, and re-route PostgreSQL data in real-time(pgflo.io)
239 points by shayonj 2y ago|54 comments
149.
Stream, transform, and route PostgreSQL data in real-time(github.com)
6 points by shayonj 2y ago|0 comments
150.
Using PostgreSQL Page-Based (CTID) Pagination for Data Cleanups(shayon.dev)
1 points by shayonj 2y ago|0 comments

More ›