levkk

13 ms·

151.
Primary Keys in Sharded Databases(pgdog.dev)
2 points by levkk 1y ago|0 comments
152.
by levkk 1y ago
That can easily be perceived as violating the ToS and get you kicked off the marketplace. Big risk to take if that's where you're making most of your $.
153.
by levkk 1y ago
Does that work with S3, etc.? I don't remember them allowing partial file uploads.
154.
by levkk 1y ago
No. Postgres data files are 1Gb each. When you change just one byte in a table, a whole 1Gb file gets updated (just 1 byte change, effectively). Your file-based backup tool now has to upload 1Gb of data to save 1 byte of actual changes.
155.
by levkk 1y ago
Gotta love these are zero-indexed. Written by an engineer for engineers.
156.
by levkk 1y ago
Finally! Been looking for this a long time. File-based backups for large Pg databases are not very scalable (even incremental), having this in my toolkit would be amazing.
157.
PgDog adds support for Rust plugins(pgdog.dev)
3 points by levkk 1y ago|0 comments
158.
by levkk 1y ago
Competition. Go build a better product.
159.
Scaling Postgres Listen/Notify(pgdog.dev)
1 points by levkk 1y ago|0 comments
160.
by levkk 1y ago
Sounds like we both got hit with this one :)
161.
Sharding Postgres at Network Speed(pgdog.dev)
2 points by levkk 1y ago|0 comments
162.
Resharding Postgres at 6.5 GB per second(pgdog.dev)
2 points by levkk 1y ago|0 comments
163.
by levkk 1y ago
My last big co, we had a team of 10 who's entire job was to sync data from Postgres into Elastic. It would take weeks and fallover regularly due to traffic. If we could have a DB that could do search and be a store of record, it would
164.
Sharding multi-tenant Postgres databases(pgdog.dev)
2 points by levkk 1y ago|0 comments
165.
by levkk 1y ago
I want to use this right now, but two issues: - Bash script from Internet requiring sudo, no way - VSCode plugin? I don't use VSCode. I'm not switching from Zed (literally built in Rust for Rust development) Help me out, what can
166.
by levkk 1y ago
That's good to know. Don't hesitate to reach out if you ever want to chat about this stuff. My email: lev@pgdog.dev
167.
by levkk 1y ago
Thanks!
168.
by levkk 1y ago
Interesting. I've been thinking about it as well. I wrote some code to prototype it here: https://github.com/pgdogdev/pgdog/blob/main/pgdog/src/backen...
169.
by levkk 1y ago
That's a good question. To echo the comment above/below, we would have to define what a "direct connection" is. In my interpretation (and as the founder, I think that matters because I would be the only one who would hav
170.
by levkk 1y ago
Definitely. I wouldn't be such a big fan of Postgres if this wasn't the case. That being said, if sharding was natively supported and worked well, they would of done it ages ago. All massive Pg deployments today are so because we
171.
by levkk 1y ago
From an infra management perspective, a multi-TB table is not fun to maintain. If you can comfortably shard before you get there, you'll be happy you did. Caveat being, sharding has to be ergonomic and not get in the way.
172.
by levkk 1y ago
It sounds like we just need to add a config: cross_shard_queries = off What else am I missing?
173.
by levkk 1y ago
Echoing the comment below (above), since we can fingerprint queries using the Postgres parser, we can create an allow list and a more fine-grained ruleset.
174.
by levkk 1y ago
Everything is fine. You never have to release your code for anything that's not PgDog itself. If you're running PgDog as a service, anyone who connects to it can ask you for the source code of PgDog. That's what AGPL says. In
175.
by levkk 1y ago
Thank you! We used Makara for this at Instacart, but eventually grew to using other languages, e.g. Python, Go, etc. and reinventing the wheel every time was pretty tedious.
176.
by levkk 1y ago
Hey Jake! I would love to collaborate. Email me: lev@pgdog.dev. Config management is a solved problem, we can use K8s or any number of CD tools. PgDog config reloading can be synchronized. Best effort transactions for schema changes across
177.
by levkk 1y ago
Small consolation prize is we can generate unique primary keys: https://docs.pgdog.dev/features/sharding/primary-keys/ . I would like to implement cross-shard unique indexes, but they are expensive to check fo
178.
by levkk 1y ago
Check this out and let me know what you think: https://pgdog.dev/blog/multi-tenant-pg-can-be-easy I think there are a few good solutions for multi-tenant safety. We just need ergonomic wrappers at the DB layer to make
179.
by levkk 1y ago
Thanks! I never considered this. What would that look like?
180.
by levkk 1y ago
I've been thinking about other algorithms as well, like range-based. We can definitely override the algorithm for hot shards. Ideally, we match what Postgres does with partitions: range, hash, and list. That way, we can shard both insi

More ›