6 ms·

This isn't just theory either, for example: Revolut is a bank that does all its event persistence and streaming on top of postgres. No traditional message queue

This isn't just theory either, for example: Revolut is a bank that does all its event persistence and streaming on top of postgres. No traditional message queues/brokers in their stack. https://medium.com/revolut/recording-more-events-but-where-will-we-store-them-4b1dad457cf5 https://medium.com/revolut/recording-more-events-but-where-w...

stackskipton 2mo ago
As SRE dealing with this at current company, a benefit of using well known software like Kafka is a lot of problems you will run into have solutions/guidance already available vs you having to explore solutions which a lot of time end with “Kafka could easily do this. “
- cyh555 2mo ago
  100% except when Kafka goes wrong, who maintains it?
  - striking 2mo ago
    Nose goes
  - ethbr1 2mo ago
    There are two sizes of companies: those that can afford '1+ dedicated ____-person' and those that can't. Which should filter through to technology choices more than it does.
    - erispoe 2mo ago
      It's not 1+ person when a system needs to operate 24/7. To have a proper on-call rotation you need 5 to 8 people.
      - CodesInChaos 2mo ago
        One specialist and a group of generalists is often enough. Especially if you are allowed to contact the specialist outside normal working hours in rare emergencies.
    - mooreds 2mo ago
      Often you start as the latter and grow toward the former. That transition can be super super painful as you don't quite have enough work for the dedicated person.
    - slfnflctd 2mo ago
      Or, you can do like one of my former bosses, and just rattle off a list of 60+ major projects which would require a team of 10 to make any reasonable progress on in the near future, then pin it on one underqualified person, refuse to provide a proper budget, and continually press them about why targeted deadlines are being missed.
      - ethbr1 1mo ago
        Organization, in that hypothetical, the individual also retains substantial power. Enough so that they should sit their boss down and say 'I have capacity for N of these projects, let's rank them in terms of priority.'
        slfnflctd 1mo ago
        You would think so! Perhaps in a more sane situation. We tried it and it did not help. Priorities shifted, facts on the ground changed, emergencies came up, and there was almost no chance to focus for long periods. Part of it was me not handling certain kinds of stress well, but a different management approach absolutely could have made a difference. To cut them a bit of slack, this particular boss had come from a prior position where they were managing hundreds of roles... to a small shop where they managed just a handful (and only one other dedicated IT person). They had trouble breaking away from the "throw everything at the wall and go full steam on it all" approach and dealing with scarce resources more carefully.
  - stackskipton 2mo ago
    I mean, with this custom thing, you have that question as well with downside is you cannot pick up the knowledge from off the street. I became the Kafka guy at my current company, it took me about a week of reading and every time I had further question, I didn't have to bother anyone, I could Google and get data I needed. When it's some NIH thing, you have to bother coworkers and knowledge is whatever is in YOUR company knowledge base with no ability to get knowledge from outside the company. EDIT: You could also leverage contractors or outside support if not homegrown software.
    - tyg13 2mo ago
      Yup, and every time it breaks, you've got to pester someone whose job probably isn't maintaining that thing actively. I worked at a startup with massive NIH syndrome, once. We even used our own in-house programming language, because it was "better than anything else out there on the market." It did have a lot of nifty features that others don't have: a pretty novel type system, programmatic macros, a built-in build system and other fun bells and whistles -- but also not-so-fun ones like having no syntax highlighter, LSP, or debugger, and having to constantly shuffle around your code to avoid ICEs in the compiler. The compiler wasn't the product, but we found ourselves fighting that thing more actively than any of the real problems our custom programming language was supposed to solve. The CTO found himself spending all his nights and weekends mostly trying to get the compiler to not explode. A few years later, after I had long left (for that reason, among many) I heard they switched to Python. Can't imagine how long it took them to get that all rewritten.
      - lelanthran 2mo ago
        > e even used our own in-house programming language, because it was "better than anything else out there on the market." It did have a lot of nifty features that others don't have: a pretty novel type system, programmatic macros, a built-in build system and other fun bells and whistles A DSL can work, but not for the features you list. Those features you already get from existing languages anyway! If you need general programming language features like excellent type system, programmatic macros, a build system (doesn't need to be built into the language), etc... then use a general purpose programming language. I have a DSL for backend/endpoints, and exactly none of those are in my feature list. What it has are things like easy way to specify access-control directives[1], the SQL query to execute, mapping request variables to SQL parameters, mapping SQL results-sets to response fields, etc. I have another DSL for a test program. Both of those DSLs have specs that's literally 2x screens of bullet points and examples. LLMs can output those DSL programs because the spec for the DSL is so small. For general purpose programming stuff (while loops, conditionals, etc) my DSLs break out to Python. A good indicator that you shouldn't be creating a new language for production is when you find yourself implementing conditionals, loops, etc. =========================== [1] Limit endpoint to specific roles, or members of the same team, or both, or even to the user itself - someone calling `/user/profile/update` should only be allowed if the profile they are updating is theirs, for example.
  - dwedge 2mo ago
    This problem doesn't go away with postgres. It's totally anecdotal but this is one thing that I've noticed different in mysql shops and postgres shops - with mysql there is usually at least one person on staff who knows MySQL DBA and scaling pretty well, with postgres it's rarely the case to have someone who knows the internals well - like you said, the person capable of maintaining it when it goes wrong. You could argue it's because postgres requires less poking though I would say you don't need the DBA for when things go right. Of course most people are just handing the management off to the cloud and that's potentially why, but it doesn't cover everything
    - sgarland 2mo ago
      MySQL will generally run fairly well with default tuning, assuming you've sized the buffer pool well relative to the amount of RAM you have (cloud providers do this automatically, but it's also not that hard to calculate). There are some knobs you can turn to eke out more performance in certain situations, and there are some defaults that are truly terrible (lock_wait_timeout is set to 1 year...), but all in all, it doesn't take a lot of care and feeding to run reasonably well. Postgres, on the other hand, has a million knobs, many of them interact, you'll find conflicting advice for some of them, and it can rapidly fall over if you aren't keeping a close eye on long-running transactions. It's also more performant than MySQL in _most_ situations (hello, clustered index), if you've tuned it correctly. It also of course has far more extensibility out of the box, with tons of index types that are extremely helpful, if you know how and when to use them. This difference is why I'm always frustrated when people parrot "just use Postgres" as though that solves all problems. It's an extremely powerful tool that can replace most of your stack, yes, but it also would really, really like you to RTFM. Not random Medium blog posts, the canonical documentation.
    - raverbashing 2mo ago
      > You could argue it's because postgres requires less poking though This is the myth people who parrot "just use Postgres" believe. It is false, obviously
  - raverbashing 2mo ago
    I love the naming Kafka. Either they knew what it stands for or they didn't. And the latter is the worse option.
andriy_koval 2mo ago
they likely have something on top of PG to distribute data across shards, which is still untrivial task I think and require ops overhead.
majormajor 2mo ago
If you start here, with the "Postgres will take you wherever you need to go" meme, without thinking extremely deeply about your schema and how you expect to evolve it in the future, you can easily paint yourself into a very difficult and expensive corner. It's easy to use Postgres poorly in ways that result in painful centralized bottlenecks. (Obviously this is largely true for anything, but I think that in 2026, where there's also a lot of more-specialized/less-fleible but much-easier-to-scale well-supported mature alternatives, you should be VERY wary of making everything have a single central SPOF. What are your users going to expect in terms of maintenance windows, etc.) I'd be cautious with articles that say things like "All cloud providers allow you to run (and scale!) PostgreSQL by clicking a single button." with no mention of how long that will take and what options should be set to make it faster, or the costs of those things.
- encoderer 2mo ago
  It doesn't take very long (because compute and storage are separate in most of them) but good lord does it get expensive. Every time you click that upgrade button you are doubling your cost. It's really painful when you have a spiky workload that is performing fine like 95% of the time but you are watching the p99 and need to double the cost of a very expensive infra component, only to improve the experience of the heaviest 4% of your workload. This is to say nothing of the gambit you then have to play with reservations/prepays.
  - majormajor 2mo ago
    I haven't seen a way to get guarantees of upscaling operations under like 30 seconds (with Multi-AZ RDS) with well-supported RDS stuff (leaving out active-active setups with logical replication because that's a whole other can of worms). If you know you're gonna be ok with that for a long time, go nuts. I'm just saying: think about it in advance! The cost pain for spikes is also a thing - some of Aurora's billing models look potentially promising but I haven't used them in practice - though it's also somethings that's harder to avoid with alternatives. Distributed DBs aren't generally super friendly to dynamic scaling IME.
- pbreit 2mo ago
dzonga 2mo ago
starling bank uk uses a similar kind of stack. both java based as revolut.
c0l0 2mo ago
During pgConf.eu in 2016(-ish, could have been one or two years later; I don't remember too well), a representative of payment processor Adyen told the audience that they were, essentially, one big postgres cluster in their backend, too ("cluster" used as per the postgres-native meaning of the term, as in, an installation on a single host with a data directory containing any number of databases).

stackskipton 2mo ago

As SRE dealing with this at current company, a benefit of using well known software like Kafka is a lot of problems you will run into have solutions/guidance already available vs you having to explore solutions which a lot of time end with “Kafka could easily do this. “

cyh555 2mo ago
100% except when Kafka goes wrong, who maintains it?
- striking 2mo ago
  Nose goes
- ethbr1 2mo ago
  There are two sizes of companies: those that can afford '1+ dedicated ____-person' and those that can't. Which should filter through to technology choices more than it does.
  - erispoe 2mo ago
    It's not 1+ person when a system needs to operate 24/7. To have a proper on-call rotation you need 5 to 8 people.
    - CodesInChaos 2mo ago
      One specialist and a group of generalists is often enough. Especially if you are allowed to contact the specialist outside normal working hours in rare emergencies.
  - mooreds 2mo ago
    Often you start as the latter and grow toward the former. That transition can be super super painful as you don't quite have enough work for the dedicated person.
  - slfnflctd 2mo ago
    Or, you can do like one of my former bosses, and just rattle off a list of 60+ major projects which would require a team of 10 to make any reasonable progress on in the near future, then pin it on one underqualified person, refuse to provide a proper budget, and continually press them about why targeted deadlines are being missed.
    - ethbr1 1mo ago
      Organization, in that hypothetical, the individual also retains substantial power. Enough so that they should sit their boss down and say 'I have capacity for N of these projects, let's rank them in terms of priority.'
      - slfnflctd 1mo ago
        You would think so! Perhaps in a more sane situation. We tried it and it did not help. Priorities shifted, facts on the ground changed, emergencies came up, and there was almost no chance to focus for long periods. Part of it was me not handling certain kinds of stress well, but a different management approach absolutely could have made a difference. To cut them a bit of slack, this particular boss had come from a prior position where they were managing hundreds of roles... to a small shop where they managed just a handful (and only one other dedicated IT person). They had trouble breaking away from the "throw everything at the wall and go full steam on it all" approach and dealing with scarce resources more carefully.
- stackskipton 2mo ago
  I mean, with this custom thing, you have that question as well with downside is you cannot pick up the knowledge from off the street. I became the Kafka guy at my current company, it took me about a week of reading and every time I had further question, I didn't have to bother anyone, I could Google and get data I needed. When it's some NIH thing, you have to bother coworkers and knowledge is whatever is in YOUR company knowledge base with no ability to get knowledge from outside the company. EDIT: You could also leverage contractors or outside support if not homegrown software.
  - tyg13 2mo ago
    Yup, and every time it breaks, you've got to pester someone whose job probably isn't maintaining that thing actively. I worked at a startup with massive NIH syndrome, once. We even used our own in-house programming language, because it was "better than anything else out there on the market." It did have a lot of nifty features that others don't have: a pretty novel type system, programmatic macros, a built-in build system and other fun bells and whistles -- but also not-so-fun ones like having no syntax highlighter, LSP, or debugger, and having to constantly shuffle around your code to avoid ICEs in the compiler. The compiler wasn't the product, but we found ourselves fighting that thing more actively than any of the real problems our custom programming language was supposed to solve. The CTO found himself spending all his nights and weekends mostly trying to get the compiler to not explode. A few years later, after I had long left (for that reason, among many) I heard they switched to Python. Can't imagine how long it took them to get that all rewritten.
    - lelanthran 2mo ago
      > e even used our own in-house programming language, because it was "better than anything else out there on the market." It did have a lot of nifty features that others don't have: a pretty novel type system, programmatic macros, a built-in build system and other fun bells and whistles A DSL can work, but not for the features you list. Those features you already get from existing languages anyway! If you need general programming language features like excellent type system, programmatic macros, a build system (doesn't need to be built into the language), etc... then use a general purpose programming language. I have a DSL for backend/endpoints, and exactly none of those are in my feature list. What it has are things like easy way to specify access-control directives[1], the SQL query to execute, mapping request variables to SQL parameters, mapping SQL results-sets to response fields, etc. I have another DSL for a test program. Both of those DSLs have specs that's literally 2x screens of bullet points and examples. LLMs can output those DSL programs because the spec for the DSL is so small. For general purpose programming stuff (while loops, conditionals, etc) my DSLs break out to Python. A good indicator that you shouldn't be creating a new language for production is when you find yourself implementing conditionals, loops, etc. =========================== [1] Limit endpoint to specific roles, or members of the same team, or both, or even to the user itself - someone calling `/user/profile/update` should only be allowed if the profile they are updating is theirs, for example.
- dwedge 2mo ago
  This problem doesn't go away with postgres. It's totally anecdotal but this is one thing that I've noticed different in mysql shops and postgres shops - with mysql there is usually at least one person on staff who knows MySQL DBA and scaling pretty well, with postgres it's rarely the case to have someone who knows the internals well - like you said, the person capable of maintaining it when it goes wrong. You could argue it's because postgres requires less poking though I would say you don't need the DBA for when things go right. Of course most people are just handing the management off to the cloud and that's potentially why, but it doesn't cover everything
  - sgarland 2mo ago
    MySQL will generally run fairly well with default tuning, assuming you've sized the buffer pool well relative to the amount of RAM you have (cloud providers do this automatically, but it's also not that hard to calculate). There are some knobs you can turn to eke out more performance in certain situations, and there are some defaults that are truly terrible (lock_wait_timeout is set to 1 year...), but all in all, it doesn't take a lot of care and feeding to run reasonably well. Postgres, on the other hand, has a million knobs, many of them interact, you'll find conflicting advice for some of them, and it can rapidly fall over if you aren't keeping a close eye on long-running transactions. It's also more performant than MySQL in _most_ situations (hello, clustered index), if you've tuned it correctly. It also of course has far more extensibility out of the box, with tons of index types that are extremely helpful, if you know how and when to use them. This difference is why I'm always frustrated when people parrot "just use Postgres" as though that solves all problems. It's an extremely powerful tool that can replace most of your stack, yes, but it also would really, really like you to RTFM. Not random Medium blog posts, the canonical documentation.
  - raverbashing 2mo ago
    > You could argue it's because postgres requires less poking though This is the myth people who parrot "just use Postgres" believe. It is false, obviously
- raverbashing 2mo ago
  I love the naming Kafka. Either they knew what it stands for or they didn't. And the latter is the worse option.

andriy_koval 2mo ago

they likely have something on top of PG to distribute data across shards, which is still untrivial task I think and require ops overhead.

majormajor 2mo ago

If you start here, with the "Postgres will take you wherever you need to go" meme, without thinking extremely deeply about your schema and how you expect to evolve it in the future, you can easily paint yourself into a very difficult and expensive corner. It's easy to use Postgres poorly in ways that result in painful centralized bottlenecks. (Obviously this is largely true for anything, but I think that in 2026, where there's also a lot of more-specialized/less-fleible but much-easier-to-scale well-supported mature alternatives, you should be VERY wary of making everything have a single central SPOF. What are your users going to expect in terms of maintenance windows, etc.) I'd be cautious with articles that say things like "All cloud providers allow you to run (and scale!) PostgreSQL by clicking a single button." with no mention of how long that will take and what options should be set to make it faster, or the costs of those things.

encoderer 2mo ago
It doesn't take very long (because compute and storage are separate in most of them) but good lord does it get expensive. Every time you click that upgrade button you are doubling your cost. It's really painful when you have a spiky workload that is performing fine like 95% of the time but you are watching the p99 and need to double the cost of a very expensive infra component, only to improve the experience of the heaviest 4% of your workload. This is to say nothing of the gambit you then have to play with reservations/prepays.
- majormajor 2mo ago
  I haven't seen a way to get guarantees of upscaling operations under like 30 seconds (with Multi-AZ RDS) with well-supported RDS stuff (leaving out active-active setups with logical replication because that's a whole other can of worms). If you know you're gonna be ok with that for a long time, go nuts. I'm just saying: think about it in advance! The cost pain for spikes is also a thing - some of Aurora's billing models look potentially promising but I haven't used them in practice - though it's also somethings that's harder to avoid with alternatives. Distributed DBs aren't generally super friendly to dynamic scaling IME.
pbreit 2mo ago

dzonga 2mo ago

starling bank uk uses a similar kind of stack. both java based as revolut.

c0l0 2mo ago

During pgConf.eu in 2016(-ish, could have been one or two years later; I don't remember too well), a representative of payment processor Adyen told the audience that they were, essentially, one big postgres cluster in their backend, too ("cluster" used as per the postgres-native meaning of the term, as in, an installation on a single host with a data directory containing any number of databases).