8 ms·

PgDog is funded and coming to a database near you

• htrp 4mo ago
  >PgDog is a sharder, connection pooler and load balancer for PostgreSQL. Written in Rust, PgDog is fast, reliable and scales databases horizontally without requiring changes to application code. Still trying to figure out how this works technically, is the performance gain really just re-write in rust?
  • levkk 4mo ago
    Not quite. The performance gain is to bring those features to Postgres! Edit: Performance gains are from having the ability to load balance reads (horizontal scaling for read queries) and scale out writes (with sharding). Once instance bottleneck in Postgres has many faces: 1. Behind schedule vacuums because of too many dead tuples (too many writes) 2. The WALWriter is single-threaded and IO-bound - Postgres can only do about 200-300MB/sec in writes per instance (real prod numbers on EC2 with NVMes and ZFS, basically best case scenario). 3. Bulkheading: single primary is a single point of failure. With 12 primaries, if one fails, 91% of your customers don't notice. The list goes on. Rust is just a side effect. We love it because it's fast and correct - the perfect match for a database product.
    • VeninVidiaVicii 4mo ago
      Oh thanks for clearing that up.
      • levkk 4mo ago
        Sorry, out walking the dog (not a pun). I'll post more details in a few.
    • hylaride 4mo ago
      So to oversimplify, is the idea to bring an AWS Aurora-style storage mechanism natively to Postgres?
      • levkk 4mo ago
        Yes, except it doesn't have any cross-dependencies on the same volume, so the uptime here should be higher.
        • jeremyjh 4mo ago
          Aurora has a completely different storage backend. PgDog is a front end proxy - each server in the cluster is still using standard Postgres right?
          • levkk 4mo ago
            Yup!
      • inigyou 4mo ago
        Aurora is one big database, isn't it? PgDog is just a proxy where you tell it which shard to access.
• Pet_Ant 4mo ago
  I hope people pronounce this as „pig-dog” and has a mascot that looks like „man-bear-pig”
  • levkk 4mo ago
    Crap! Missed opportunity.
• fulafel 4mo ago
  Does making it "just work" here come with any caveats vs standard PG?
  • levkk 4mo ago
    Getting there! Cross-shard writes do because of 2pc. Reads are eventually consistent.
  • danielheath 4mo ago
    Given that they implement connection pooling and sharding, I'm going to say "not at all". You _could_ make that ACID, but it's not going to be faster than a single machine.
• melon_tsui 4mo ago
  2M qps in production is legit. Curious how much RAM and CPU that takes on average per deployment though
  • levkk 4mo ago
    Depends. Only pooling, very little. Load balancing/sharding needs to parse queries, so a bit more. Could go up to a GB per pod, sometimes more if you have a lot of unique SQL queries (unique by text, not by parameters). We cache query ASTs to avoid parsing them on each request - that's the bulk of memory usage.
    • parthdesai 4mo ago
      Semi related question - I have always wondered, how do you tackle OOM issues at the proxy layer, i.e. let's say a particular SQL query requires proxy to fan out the query to multiple shards, which return a pretty large dataset. I'm assuming you would need to load this dataset in the ram to perform certain operations. What happens if the resulting dataset causes the proxy pod to go OOM?
      • levkk 4mo ago
        Two schools of thought: 1. Let it crash. Increase the RAM, try again. 2. Page to disk (swap), make it slow but ultimately work. Both have their trade-offs. There is no free lunch here.
• skiwithuge 4mo ago
  we are using PG bouncer in production. Interesting, I will follow the evolution of this project
• kjuulh 4mo ago
  I tried out PgDog a while ago, but couldn't find a good way of handling the config except for having this users / pgdog toml file, which makes it a bit awkward to handle in kubernetes where we often do multi-tenancy in postgres - or rather having many databases on the same instance(s), and have them come and go at will. Also had an issue with it because it cached authentication requests when doing passthrough it seems, I'd changed the roles password, but it kept using the old one, which was no bueno ;). PgDog seems to make more sense when you really care about a few databases that need massive scale, rather than a simple proxy in front of postgres. I'll keep following the development though, it is much needed in this space, postgres can use all the investment it can get to get it past the single machine scale that it excels at currently.
  • maherbeg 4mo ago
    Happy to chat about this, but we use the AWS secrets manager flowing into External Secrets Operator to generate a pgdog_users.toml. We then kick off a workflow to refresh things, but our rate of change here is much smaller than a super dynamic multi-tenant system. You could also build a watcher side car that watches for changes of the pgdog_users.toml and have pgdog refresh itself then too with this combination. We thought about that but prefer to control the reloads for our needs.
  • apt-get 4mo ago
    We successfully did this with pgdog at $JOB using our own "controller" -- the same service that handles deploying new instances of our application (instancing an argoCD Application that fires Crossplane DB creation, making new Deployments of bricks, etc) will also, at the end of that process, scan the cluster for Database CRDs, use those to generate a new pgdog.toml + users.toml, update the Secrets in the cluster, enable maintenance mode on all pgdog pods, do a live config reload on each of them, then disable maintenance mode (this is to make the change atomic between all the pgdog instances). Downtime there is about 2-3 seconds and all it does is make new SQL requests from existing clients wait, it doesn't break the connection or anything.
  • levkk 4mo ago
    Not the place and not the time, but we are building an enterprise edition that "just works" out of the box. Not saying that the open source experience cannot be better - it always can and we'll keep improving. What you've experienced is definitely a known issue with our specific implementation of passthrough auth. Scram made things a bit harder, since we can't validate user's passwords at login time anymore (that's what makes scram secure fwiw). We'll get there.
• jeremyjh 4mo ago
  It’s surprising they don’t mention advantages over other sharding systems like Citus. Maybe it’s just the fact that it’s only a proxy and not core extensions? But that could limit capabilities.
  • levkk 4mo ago
    We do, just buried deep in our blog: https://pgdog.dev/blog/pgdog-vs-citus https://pgdog.dev/blog/pgdog-vs-citus The same old processes vs. threads debate, plus having the ability to scale the coordinator past a single machine. So, if you're OLTP, definitely consider PgDog. OLAP - Citus still wins because of its advanced query engine. We'll get there.
    • jeremyjh 4mo ago
      Excellent article, this makes a lot of sense! TLDR: Tokio concurrency > Process concurrency in OLTP.
    • ahachete 4mo ago
      > having the ability to scale the coordinator past a single machine Since Citus v11 (released 4 years ago), any worker node can also work as a "query router" (a node that you can query against [1], and works from this perspective as a pure coordinator: > for very demanding applications, you now have the option to load balance distributed queries across the workers You can also setup such query routers as dedicated nodes by setting the `shouldhaveshards` to `false`, becoming an effective coordinator (for querying; not for metadata operations). So with Citus you can absolutely have as many query routers (coordinators if you wish) as you want. [1]: https://www.citusdata.com/updates/v11-0/#metadata-sync https://www.citusdata.com/updates/v11-0/#metadata-sync Edit: formatting, typo
• ParadisoShlee 4mo ago
  I've moved from pgbouncer to pgdog a few months ago without issue. Huge fan.
• moralestapia 4mo ago
  Cool work, thanks. Wrt. the pooler, how do you compare with pgbouncer? I'm interested because I have a postgres instance, low-traffic but still like ... tens of r(eads)ps. I was not running anything close to the machine limits but still added pgbouncer to improve performance and didn't see a noticeable difference. I was stress-testing the machine obv., I'm not talking about the 10 rps, lol. For context, my numbers were something like 10k rps +/- 1k vanilla postgres and like 9k rps +/- 1k with pgbouncer in front of it. So ... slightly slower but big error bars so I wouldn't say for sure. I ended up not using pgbouncer as the benefit was immaterial. Also yeah, in case you want to check it out, it's the db that backs this project: https://httpstate.com https://httpstate.com.
  • levkk 4mo ago
    Old benchmark, but still good: https://pgdog.dev/blog/pgbouncer-vs-pgdog https://pgdog.dev/blog/pgbouncer-vs-pgdog
• faangguyindia 4mo ago
  i am not using any tool like pgbouncer and have not run into any issues so far. Is it even required these days? Have you guys tested your setup without these connection poolers/multiplexers?
  • rswail 4mo ago
    Each connection is a process on the server, that takes up both CPU and RAM, it will run out. This solves the thousands of clients case for read in a way that is transparent to the clients. Yes it's required at large scale, especially if you want to distribute reads or shard to a particular geographical area.
• simonw 4mo ago
  Suggestion: have more than just helm and Docker in your quickstart documentation. I'd like to try this out just to see what it can do, but not quite enough to fire up one of those systems for it. Is there a binary I can run directly?
  • frogbydjsd 4mo ago
    [dead]
  • levkk 4mo ago
    We should add it to brew/apt/etc for sure. Also, we could add it to crates.io so you could do something like `cargo install pgdog`. Distribution, distribution, distribution.
    • simonw 4mo ago
      I also appreciate GitHub releases with pre-compiled binaries for different platforms. The more options the better!
  • e12e 4mo ago
    In addition - the docker compose example doesn't set up any data volumes for the postgres instances - that might be considered a bug? Then again, sharding on a single host probably isn't very useful anyway - but it might work with docker in swarm mode?
    • levkk 4mo ago
      The docker compose example is just a demo. I don't know anyone who runs Postgres with docker compose / swarm in prod :) But yes, happy to add volumes so it seems more real.
• bourbonproof 4mo ago
  the reason mongo is a joy to use in scaled env is because no additional setup/software needed and all drivers natively support secondary/primary writes/reads and topological changes. so it's end to end, and adding is as a new proxy in frontend of postgres leads to all clients being incompatible or the code itself has no control anymore about when to use a secondary and what allowed stall is acceptable for a particular query. Any solutions to this by pgdog?
  • saghm 4mo ago
    > all drivers natively support secondary/primary writes/reads and topological changes. Expanding on that a bit, mongo drivers even have a shared specification of the state machine for monitoring topology changes[1] and algorithm for selecting the server to send an operation to[2] (along with various declarative test cases that the drivers use to validate them alongside the specs in the repo). I think people sometimes underestimate how important the client-side work is to this sort of experience; for all of the faults mongo has had over the years, the amount of investment that they put into the client libraries is something I've never seen anywhere else (although having spent several years working on some of these libraries, my take is likely very biased). [1]: https://github.com/mongodb/specifications/blob/master/source/server-discovery-and-monitoring/server-monitoring.md https://github.com/mongodb/specifications/blob/master/source... [2]: https://github.com/mongodb/specifications/blob/master/source/server-selection/server-selection.md https://github.com/mongodb/specifications/blob/master/source...
  • dzonga 4mo ago
    once mongo rewrote their engine - it's performant, scales & easy to run. seems a lot of devs got burnt by the early issues don't consider it all together. its probably the easiest database to run at scale. run & forget. you just have to do a little more work on the data modeling part before you write your application i.e consider your query patterns.
• maherbeg 4mo ago
  I'm a big PGDog fan! It really helped us scale our connection proxy needs pretty substantially and it has great features like auto mode to support Aurora failovers neatly. It's infra that just works.
• chrisvenum 4mo ago
  I am trying to gain a basic understanding of this: Right now I have a 4TB DB on one large box. Is the idea that using a proxy tool like PGDog I could spin up 8 smaller boxes handling ~500GB each and then one medium box for the proxy? Right now I have a project that has very heavy write traffic from multiple services and a web app that reads from this. We are starting to hit the point where no amount of indexing, query optimisation, caching or box upgrades is helping us. We are looking at maybe moving the bulk of the static data to clickhouse to reduce the DB size but I would love to hear if PgDog or other kind of sharding could be useful for this use case.
  • levkk 4mo ago
    > 8 smaller boxes handling ~500GB each and then one medium box for the proxy? That's exactly right. Get in touch (lev@pgdog.dev), happy to help or at the very least tell you what current works (or doesn't) so you know what your options are.
  • inigyou 4mo ago
    That's the idea of sharding. If you read the pgdog docs, you'll notice you need to tell it which shard server to route your request to - it doesn't just magically work. It's still providing value by reusing connections, which are particularly expensive in postgres. Because it's not magic, you do still have to know what's going on under the hood, e.g. no cross-shard transactions. I'd see if my application can benefit from read replicas before doing sharding, because sharding is difficult (if you care about data consistency). With replicas, each replica does have a full copy of the data and you only write to the master - you have to decide which transactions are suitable for running against replicas, which can lag slightly behind realtime. E.g. reading data to build a webpage is probably safe to do from a replica - any read-modify-write is not.
    • levkk 4mo ago
      fwiw, we support cross-shard transactions. They are not magic though, just good old 2pc and a bit of coordination.
• 999900000999 4mo ago
  How are 3 developers going to QA this properly ?
  • pantulis 4mo ago
    How are 3 developers going to sell that to any company? Procurement will have a field day.
    • rswail 4mo ago
      They have funding. That's what it will be for. I wish them well and appreciate that people are still doing FOSS. As long as they don't get undercut by the equivalent of AWS https://aws.amazon.com/rds/proxy/ https://aws.amazon.com/rds/proxy/ which is a managed pgbouncer.
      • 999900000999 4mo ago
        The issue is if the DB layer fails your product is going to completely stop working. You’d need a ton of faith in these 3 people. Feels more like it would work better inside of a bigger organization. The QA tester in me is kinda risk adverse.
        • rswail 4mo ago
          The source code is available for inspection as is the biographies of the people involved. They rely on the libraries that are part of Postgres itself to ensure they are parsing the SQL etc "correctly" (where "correct" means "the same as Postgres itself). Bigger organizations do not necessarily mean higher quality. What bigger organization is testing PostgreSQL itself? What are the relative quality measurements of Postgres vs MariaDB vs Oracle vs SQL Server?
          • 999900000999 4mo ago
            MariaDB has over 200 employees. https://mariadb.com/about-us/careers/ https://mariadb.com/about-us/careers/ Oracle's department that handles DBs probably has at least a few hundred. 3 people would be an ultra lean QA department for a product like this. I'd have a hard time convincing my boss to go with PGDog over a more stable and tested solution. This doesn't mean it's bad, just not ready yet
• yabones 4mo ago
  I'm curious how this might help with our biggest downtime-causer with postgres, which is major version upgrades. Poolers do a great job for failover and load balancing, but we consistently need ~10-20 minutes of downtime once or twice a year to do upgrades. Logical replication between old->new versions could probably help, but it would still require flipping everything over to the new cluster without partial writes or anything silly. Anybody have experience with this?
  • boxed 4mo ago
    Seconded. Coming from MySQL this is a huge regression that makes Postgres look like something from the 80s. I still wonder why this isn't seen as the absolutely highest priority.
    • Blackthorn 4mo ago
      Probably because it's an open source project and apparently none of its users cared about this feature enough to develop it or fund it.
      • deleted 4mo ago
        [deleted]
      • jeltz 4mo ago
        It is also a bit tricky tradeoff. You do not want to be stuck with the same data format forever. So databases like MySQL and PostgreSQL need a downtime when doing a major version upgrade. They both try to keep it short, usually seconds, but minutes can happen in either database.
    • jeltz 4mo ago
      I have not ran MySQL for some years but it at least used to have exactly the same issue. Upgrading a database with MySQL can take a long time if you have many tables. The main difference is only really that PostgreSQL does it with a separate tool, pg_upgrade, while MySQL does it as part of the main binary. For both MySQL and PostgreSQL you will need to use some kind of logical upgrades if you want no downtime.
• Ozzie_osman 4mo ago
  We sharded over 20 TB that we know about. This is probably a typo, right? 20TB isn't that big. I would imagine they've sharded a lot more than that
  • GiorgioG 4mo ago
    For a vast majority of use cases 20TB is positively enormous.
    • tingletech 4mo ago
      that article seems to suggest 20TB total over the dozen deployments in prod.
    • happyopossum 4mo ago
      Sure, but 20TB in “the only database you need” is mere hours or minutes worth of data for many workflows.
    • returningfory2 4mo ago
      This product is for Postgres deployments that are so large they need to be sharded. For these use cases, I think 20TB is about normal.
    • jeltz 4mo ago
      Yes. But for most workloads it is not much for PostgreSQL. You often will not have to shard at all.
    • mplanchard 4mo ago
      RDS caps out at 64 TB unless you use Aurora, so 20 TB is totally manageable without sharding.
  • rbranson 4mo ago
    You are correct. As a point of comparison: almost ten years ago at Segment we had a single Aurora PostgreSQL instance with ~50T of data, it was used to index potential identity data in a much larger corpus of files stored in S3.
• drchaim 4mo ago
  Good stuff, although I’m not quite sure about the fast OLAP use case. If you’re already sharding by tenant for other reasons, OK… But I see CDC to a true OLAP system as more scalable. PostgreSQL still needs real columnar tables in the core, hopefully one day
  • levkk 4mo ago
    OLAP means different things to different people. For us, it's just making sure your admin dashboard keeps working basically: SELECT tenant_id, COUNT(clicks) FROM users GROUP BY tenant_id ORDER BY 2 DESC LIMIT 25; Performance is a side effect - definitely needed and we'll do everything we can, but we are not competing with ClickHouse or Snowflake - just trying to make sharded Postgres work with your app.
  • christoff12 4mo ago
    Re OLAP: It's probably ~good enough~ for a lean team that's trying to keep the tech stack standard and/or doesn't have a dedicated data person to take advantage of a columnar store.
  • vira28 4mo ago
    Tomas Vondra, a major Postgres contributor recently revived a thread on using Bloom filters - https://www.postgresql.org/message-id/flat/5cd8c20c-14b5-4b0d-bedc-69bf714e87eb%40vondra.me https://www.postgresql.org/message-id/flat/5cd8c20c-14b5-4b0... So there is more core work happening on support OLAP but I do think it will take some time. In the meantime, I think we have all the pieces (storage, query engine, table format) to set up a true OLAP. For instance, I created https://github.com/viggy28/streambed https://github.com/viggy28/streambed to pressure test this idea.
• mnbbrown 4mo ago
  I've loved using pgdog for the last 6 months. It's been incredibly stable. It's nifty how they've solved the LISTEN/NOTIFY on a transaction pooler problem.
• tschellenbach 4mo ago
  PgDog, Neki, multigres, awesome to see. And yes this is the main issue with postgres. Well this and not having index hints, looking forward to 19
  • welder 4mo ago
    Don't forget the original PgBouncer. Hard to setup, but with the help of AI these days it's easier to configure.
  • paulryanrogers 4mo ago
    The pg_hint_plan extension isn't in core, yet is pretty competent when you need to override planner.
• welder 4mo ago
  Three real-world issues I've run into recently with PgBouncer + Postgres are: 1. pool exhaustion from idle connections inside open long-running transactions 2. SQLAlchemy's client-side pool using dead connections that PgBouncer had already killed, causing periodic request errors 3. Some tasks have to bypass PgBouncer when they use SET or prepared statements I've already sharded large datasets at the application layer, but looks like PgDog solves the above problems for any future work?
  • tempest_ 4mo ago
    SQLA async is a bit of a struggle with pgbouncer. I had to disable application pooling as it was causing read only transactions I could couldnt pin down the cause.
  • frollogaston 4mo ago
    #1 is a problem with the client's code, I don't know any easy workaround. Usually a long-running transaction means you're accidentally waiting on stuff like RPCs in the middle, or maybe doing something that doesn't really need to be in a xact. #2, shouldn't the client<->PgBouncer connections stay open? #3 is why I just use client-side pools instead of PgBouncer, but that gets annoying when you have a replicated service so you have to think about the sum of connections across all pools, so I get why people use PgBouncer.
• sandeepkd 4mo ago
  Nit-Pick: It might be anti-marketing, still it would be helpful if the use cases can be articulated in a way where it would make sense to use this Vs any other type of database. Honesty goes a long way with the more technical folks for anything related to infrastructure. Surfacing where and how PG is better than Dynamo or any other database is probably a good starting point instead of calling out PG a silver bullet for everything. At the end of the day its all a trade-off.
  • levkk 4mo ago
    Always is. Marketing is not our strong suit (only engineers here). We'll get better at it.
    • rabidferret 4mo ago
      Crap, I'm supposed to be an engineer?
• codegeek 4mo ago
  "Why Us" => "I ran Postgres at Instacart, where we scaled the company 5x in April of 2020. The biggest problem we had was making Postgres serve 100,000s of grocery delivery orders per minute" Couldn't be a better why us :)
  • paoliniluis 4mo ago
    Legends
  • deleted 4mo ago
    [deleted]
  • deleted 4mo ago
    [deleted]
  • aurareturn 4mo ago
    Is 100k order per minute a lot? Even a single Postgres instance should serve that fine?
    • smt88 4mo ago
      One assumes they mean 100,000s (plural) concurrent users actively building carts
      • aurareturn 4mo ago
        Is that still a lot? Feels like a single 64-core, 256GB RDS instance with some caching should handle that fine. RDS has instances up to 192-core and 768GB.
        • smt88 4mo ago
          Keep in mind they’re doing real-time logistics and messaging, as well as type-ahead search and managing ads and promotions
          • 4mo ago
• valorzard 4mo ago
  I've seen a couple of these "distributed" postgres extensions. My question is, has any of them been talked about being upstreamed to postgres itself? Or, adding a custom built in feature to postgres itself?
  • levkk 4mo ago
    This is not an extension, it's a proxy! Very different. You can deploy it anywhere already without having to wait for upstreaming or your cloud provider adding support for it. It's one of the two reasons why we built it this way, the other being performance (it's much faster to do this in the proxy than inside Postgres).
  • inigyou 4mo ago
    It doesn't actually distribute postgres. It lets you use one connection to talk to multiple postgres databases by switching between them and if you're very careful you can sort of see it like a single database, ht it's not really.
• orliesaurus 4mo ago
  how does it compare to PlanetScale ?
  • christoff12 4mo ago
    PgDog is GalaxyScale </joke>
• Wonnk13 4mo ago
  I wish them all the best. Supabase, Timescale, etc etc. there's a whole cottage industry of extending postgres to whatever you need.
• mamcx 4mo ago
  I do tenant per PG schema, most are smallish some are bigger (not much, can do all in a single box) but moving forward eventually will need something like this. Also plan to provide "get your own VPS" for more enterprise customers. This kind of tool will help in this case?
  • levkk 4mo ago
    Yup. We support schema-based sharding: https://docs.pgdog.dev/configuration/pgdog.toml/sharded_schemas/ https://docs.pgdog.dev/configuration/pgdog.toml/sharded_sche...
• jeremyjh 4mo ago
  > With $5.5M from Basis Set, YC, Pioneer Fund and other great investors, we have years of runway, This is years of product development with a three person team. If Enterprise sales and support are a big part of your business plan it will suck up a lot more than that.
  • pphysch 4mo ago
    Presumably enterprise sales will bring in revenue on its own
    • jeremyjh 4mo ago
      Yes but the sales process for enterprise is often 9 months or more, and you have to have support coverage before it pays for itself.
  • GHanku 4mo ago
    [dead]
• xenophonf 4mo ago
  This commit looks... odd. https://github.com/pgdogdev/pgdog/commit/36434f93f03dec1d7d4822e154b7d15e6928332f https://github.com/pgdogdev/pgdog/commit/36434f93f03dec1d7d4... I want to have as much fun as the next developer, but that makes me worry, what with supply chain attacks in the news and all.
  • levkk 4mo ago
    I see you met Sage, our newest founding engineer :) If you're not having fun at your job... In all seriousness, we review every single line of code that goes in and only people who work for PgDog Inc are allowed to merge.
  • rabidferret 4mo ago
    I am odd, yes. I also care deeply about supply chain security and focused on it when I led the crates.io team as well as during my time at the Rust Foundation. You can rest assured that my occasional shitposts are not opening an attack avenue for your supply chain. - Sage
    • rabidferret 4mo ago
      I will not stop shitposting on main though
• eikenberry 4mo ago
  > The reason DBs like Mongo or Dynamo exist is because Postgres has a scaling problem. I've used Postgres at a few places and the #1 problem was always high availability, not scaling. One Postgres cluster could easily handle 100000 transactions per minute, but when a primary node went down it was a page and manually failing over to the spare then manually replacing the spare. The manual tooling was very finicky but at least it worked, no automated solution came even close. Lack of a good HA story is why I avoid self-managed Postgres as much as possible.
  • globular-toast 4mo ago
    Have you looked into things like CloudnativePG? https://cloudnative-pg.io/ https://cloudnative-pg.io/
    • nijave 4mo ago
      CNPG is quite nice and robust but I'd still be a bit reluctant to stack PG on k8s for really big clusters just because k8s ecosystem moves quite quickly and there's lots of patching/maintenance/churn which means more PG failovers so depends on how well your workload handles that (they're normally only a few seconds)
      • globular-toast 4mo ago
        Most K8s upgrades can happen independently of node reboots etc., you only need to update for OS updates really, but that would be true of anywhere you run PG, even RDS.
        • nijave 4mo ago
          >but that would be true of anywhere you run PG, even RDS It's a little easier to strip down userland if the machine is only running PG. Technically possible on k8s with distros like Talos, Bottlerocket, etc but you still have all the k8s deps on top of PG. It's also a little easier to do defense-in-depth on a dedicated PG machine which means you might have mitigating controls in place to skip security patches (minimal kernel modules, selinux)--possible on k8s but now you're fighting through a 2nd layer of configuration RDS is a bit of a special case because you also have AWS curating and prioritizing updates. You can do that yourself but it's a bit of a time sink scrutinizing every upgrade to see if you _really_ need it. Our RDS instances tend to go 3+ months without restarts
• aejm 4mo ago
  I notice there is an Enterprise Edition, can you please specify which features are not open source? Do you predict new features you add will be ee licensed as a way to pay back your VC funders?
  • levkk 4mo ago
    Two big ones: 1. Control plane to manage multi-node deployments; "works out of the box" experience to make PgDog easy to deploy and use 2. QoS (quality of service): automatically block bad queries from taking down the database Last but not least, you get SLA-backed support from us (up to P0). New features are broken down into two categories: 1. Sharding / running Postgres at scale: always open source. 2. Infra management / making it easy to run PgDog at scale: enterprise.
    • aejm 4mo ago
      Thank you for the clear response!
    • underdeserver 4mo ago
      This is a remarkably open-source friendly business model. I hope it works out similarly remarkably well for you!
• redmonduser 4mo ago
  How is this different from Citus?
• gen220 4mo ago
  Is there an explainer for people who are broadly familiar with the DB space? It sounds like you're building an equivalent to Vitesse for Postgres, but it's not super clear from the article (which I know is not the point of this, but still :) ). Edit: It also might be interesting to point out how your solution differs from what the folks at Planetscale are building https://planetscale.com/neki https://planetscale.com/neki
  • parthdesai 4mo ago
    There's multiple solutions coming up in this space: 1. Neki as you mentioned 2. PgDog 3. Multigres, headed by original creator of Vitesse
    • frollogaston 4mo ago
      Citus is an older one that does something like this, right? But it's an extension, not a proxy.
      • parthdesai 4mo ago
        I could be wrong, but with Citus, for most use cases, you can only have one co-ordinator node which fans out requests. So theoretically, you still can run into bottle necks at some point if 1 coordinator node is not enough. With proxies like pgdog, multigres, and eventually Neki, these can scale out horizontally, so you get true unlimited scale.
        • frollogaston 4mo ago
          Doesn't PgDog only have one proxy, or can you have multiple? I imagine they'd need to coordinate on sharding rules somehow.
          • levkk 4mo ago
            You can have multiple. All sharding is config-based, so no real-time synchronization is required.
• exabrial 4mo ago
  > The reason DBs like Mongo or Dynamo exist is because Not quite. The reason "DBs" like those exist is purely due to fashion. Lets not kid ourselves into thinking they do anything better, save the exception of making data hard to access, which might be a project goal in some cases.
  • inigyou 4mo ago
    Dynamo definitely scales better than anything else at the tradeoff of not guaranteeing durability in the case of enough node failures and (like most distributed databases) not allowing interaction between different pieces of data.
    • exabrial 4mo ago
      > Tradeoff of not being able to query your data Fixed it.
      • inigyou 4mo ago
        Dynamo is not write-only.
        • exabrial 4mo ago
          I mean neither is /dev/null
• dzonga 4mo ago
  I us pg. not that I know much about database internals, besides the 'b-tree' stuff we learned in college. I don't know how the pg scaling story gets fixed unless certain things are rewritten. that's my fear of going all in pg. mysql has vitess etc & even upgrades are easier. though pg is more extensible.
  • inigyou 4mo ago
    Any strongly consistent database is going to be limited by a single machine's throughput. That's just what you trade for strong consistency. You can shard it yourself but then the DBMS isn't giving you consistency so you'd better be very careful. You can use a tool like PgDog to aid with sharding but it's not doing magic, you still have to be aware how it works and the limitations of sharding. However 95% of projects are going to be fine with a normal single-machine database and another 4% are going to be well served by upgrading the hell out of that machine. Only the absolute busiest projects actually need a distributed database and you can cross that bridge when you actually get to it. They say Amazon processes 20k orders per second. That seems not unachievable for postgres with fast SSDs and careful query optimization, though they don't choose do it that way. You're not Amazon, you have at most 20 orders per second and that's nothing.
• SamInTheShell 4mo ago
  Scratching my head. Wondering why I would reach for this over just running a Yugabyte cluster.
• octernion 4mo ago
  congrats, lev! brings back fond memories of database fires. i'm sure you'll get 100x comments about "why not just have one fast SSD? it can do 2000 trillion writes/s"
  • levkk 4mo ago
    Thanks! Yup...to be expected. If you know, you know, and have the scars to prove it :)
• BowBun 4mo ago
  I really wish they'd acknowledge the prior art and name that they've taken inspiration from - https://github.com/postgresml/pgcat https://github.com/postgresml/pgcat Don't pay a startup for your DB proxy, you should own that layer yourself inside of your infrastructure.
  • xyzzy_plugh 4mo ago
    The creator of pgdog is also the creator of pgcat, so I think they probably don't need to do this.
    • levkk 4mo ago
      This reminds me of college. We had to cite our own papers from prior semesters or risk getting kicked out for plagiarism. I don't miss those days :)
    • BowBun 4mo ago
      I disagree, because now I am suspicious as to why there's a glaring omission like that. Never the mind looking at contribution timelines.
      • apsurd 4mo ago
        "it's not that deep" as the kids say. In fact postgresML took naming heat because Postgres is right there in the name and they weren't affiliated with the brand. "pg" is just two letters. like WP-engine (literally the name as they say it is "double U P engine"). And a cat and a dog is fun. don't think they're trying to get one over on you.
        • BowBun 4mo ago
          I don't think they're trying to get one over me, nor do I think it's _that deep_. One should simply acknowledge the prior projects that led to where you are today, even if they are your own. I 100% stand by my original statement and think that pgDog should mention that they're affiliated or a paid product on top of pgCat!
• snihalani 4mo ago
  I'd love to advocate for PgDog if there were more than 2 managed service providers. Adding a single company with no substitute in your supply chain feels hard
  • levkk 4mo ago
    I didn't realize there are _any_ managed providers of PgDog out there...do tell!
• karolist 4mo ago
  Love PgDog. I don't need it honestly, but using it in my on-prem k8s because I heard about you in Postgres FM podcast randomly when I had nothing to listen to on a hike in the woods and it picked up my interest. https://open.spotify.com/episode/6qgpfiW68KcvRASs6649Fb https://open.spotify.com/episode/6qgpfiW68KcvRASs6649Fb
  • levkk 4mo ago
    Thanks!
• afr0ck 4mo ago
  Is this vibe-coded?
  • rabidferret 4mo ago
    No
• gregaccount 4mo ago
  Fix the bad license.
• philippemnoel 4mo ago
  Let's go. Very bullish on PgDog. Lev understands this space better than anyone else. If you are sharding Postgres, you should talk to him.
• RedMagicBox 4mo ago
  [dead]
• bart3r 4mo ago
  We are still using Pgpool-II and it's been very solid, but would be interested in moving to PgDog. Would love to hear the advantages of moving to PgDog.
• frollogaston 4mo ago
  Reminds me of long ago, before Postgres even had things like parallel scan to utilize multiple CPU cores on a single machine, I used to have Python helpers to split up queries by ranges of IDs. If a query was complicated, I'd EXPLAIN it first then pick either the innermost or outermost index scan, and often get a linear speedup. But it was quite manual, required using temp tables for SELECTs, and ofc had no consistency.
• esafak 4mo ago
  I think sharding is the wrong approach; who wants to mess about with sharding logic? Distributed key-value stores are the way to go. But cockroach already offers that so I suppose you can try the other way.
• zadikian 4mo ago
  This is exciting. INSERT (SELECT ...) doesn't work though, right? The docs only mention VALUES inserts.
  • levkk 4mo ago
    Not yet, but actively working on this as we speak.
• andrey-g 4mo ago
  How does this compare to Aurora Serverless?
• TurdF3rguson 4mo ago
  let's say i have a primary with 100M rows of addresses and indexes on things like city, state, zip code (all in memory). I also have 3 read replicas that struggle to do 1000 lookups per minute each. Does PgDog help?
• sgt 4mo ago
  Is this like on prem RDS?
• floriferous 4mo ago
  Is this comparable to Supabase's just announced multigres?
• mijoharas 4mo ago
  Congrats on the funding Lev! Just to say we're happy pgdog users here! One feature we quite like (of the proxy) is the handling of different connection settings per connection (i.e. statement_timeout). When we investigated RDS proxy (ages ago) it wasn't supported, I think the same was true for pgbouncer so it required a bunch of application changes. With pgdog, it just works transparently.
  • levkk 4mo ago
    Thanks! Glad we made it relatively easy to migrate!
• s3cur3n3t 4mo ago
  This is just awsome
• ahachete 4mo ago
  I have mentioned this before, but here it goes again: I'm really happy that there's more options for Postgres sharding and I applaud Pgdog and the team's efforts and energy. Having said that, this makes it a no-go for me: > shard_number = hash(data) % num_shards https://docs.pgdog.dev/features/sharding/basics/#terminology https://docs.pgdog.dev/features/sharding/basics/#terminology Most sharding solutions distribute the hash value over linear ranges, that then split across "virtual shards", that are then placed on the physical shards or worker. This allows for shard replacement when needed. For example, Citus works this way, and even adds convenience functions for shard migration (using logical migration) in an automated way. That's all I'd need. Operationally, it's worlds apart. With modulo distribution the only way to replace data is to reshard everything --something you don't want to do however fast the operation may be.
  • levkk 4mo ago
    Yeah good callout. We'll add rendezvous soon enough. Until then, being compatible with Postgres partitions has been advantageous -- while we build everything out, people were able to migrate to PgDog for the query routing layer while doing the resharding in Postgres. Adding a sharding function in our architecture is relatively straightforward. We also support plugins which can control the flow (and direction) for queries, so our users can add their own (and they do!).
    • ahachete 4mo ago
      TBH I don't think it's that straightforward, I see it more of a notable architectural change. At a very high level, this means: * Adding a sharding function, as you say. * Developing an external service for metadata (shard placement) or alternatively have that metadata in one place and replicate (consistently!) to every query router. * Implementing functions/catalogs for the users to understand the placement and configure/alter it. * Implementing shard migration / rebalancing capabilities, possibly using Postgres logical replication (plus notable automation). Here's one idea if you follow this path, something that Citus doesn't have: make the sharding function pluggable and pick one by default which is well-known and available in many languages (e.g. xxhash). If you do so, and guarantee stability of those functions, they could be used externally (applications) to route queries / inserts especially to the appropriate shard. While it makes application more complex, it may allow (combined with access to the metadata service) for faster ingestion paths (this is often known as application assisted sharding), and its not exclusive of the query routers. Edit: formatting
• antonvs 4mo ago
  > we don’t think you would use anything else. This just seems like fanboyism to me. At the very least, you need to qualify what scenarios you think it's useful for. I don't doubt that Postgres is good for all the projects you've ever worked on. Generalizing from that, though, is hubristic.
• mohammedelkarsh 4mo ago
  [flagged]
• netswift 4mo ago
  We've run into so many issues with PgBouncer and Postgres that I wish we didn't have to deal with as a new growing company. Nice to see more options out there!
  • kstrauser 4mo ago
    Like what? I've never had problems related to PgBouncer, but apparently we have different use cases. I'd love to hear where the rough edges are so I can avoid them, or at least plan for them in advance.
• directionless 4mo ago
  We used `pgdog` as a proxy during a recent database backend migration (Heroku -> EC2 -> RDS) and it was much smoother than PgBouncer. Really nice seeing more things in this space, and having the team's work recognized.
  • levkk 4mo ago
    Awesome, glad it worked!
• sonixaep 4mo ago
  [dead]
• gertburger 4mo ago
  I see the top diagram on the frontpage shows 'dsql' support but it isn't mentioned in the documentation, is that correct?
• advertum 4mo ago
  [dead]
• hodgesrm 4mo ago
  What's the difference between pgdog and vitesse?
• edge_trader_41 3mo ago
  [dead]