8 ms·

I often go down rabbit holes like this, trying to collapse and simplify the application stack. But inevitably, as an application grows in complexity, you start

I often go down rabbit holes like this, trying to collapse and simplify the application stack. But inevitably, as an application grows in complexity, you start to realize _why_ there's a stack, rather than just a single technology to rule them all. Trying to cram everything into Postgres (or lambdas, or S3, or firebase, or whatever other tech you're trying to consolidate on) starts to get really uncomfortable. That said, sometimes stretching your existing tech is better than adding another layer to the stack. E.g. using postgres as a message queue has worked very well for me, and is much easier to maintain than having a totally separate message queue. I think the main takeaway here is that postgres is wildly extensible as databases go, which makes it a really fun technology to build on.

nextaccountic 3y ago
> But inevitably, as an application grows in complexity, Some applications never grow that much
- ejb999 3y ago
  I would go further and even say 'most' applications never grow that much.
- LaGrange 3y ago
  But it surely will! It will! See, now that we're profitable, we're gonna become a _scale up_, go international, hire 20 developers, turn everything into microservices, rewrite the UI our customers love, _not_ hire more customer service, get more investors, get pressured by investors, hire extra c-levels, lay-off 25 developers and the remaining customer service, write a wonderful journey post. The future is so bright!
  - zwnow 3y ago
    Honestly, that's one of the reasons I never want to monetize my work and stay miles away from the software industry. Modern world is all web apps that require you to subscribe to 20 different 3rd party services to even build your app. So you rack up bills before your product is even remotely lucrative... Building an app with no third party dependencies seems impossible nowadays. At least if you plan to compete.
    - LaGrange 3y ago
      I mean, you _can_ host your staging environment on a Minisforum PC hidden in your closet and then deploy to Hetzner, and probably save a _ton_ unless your service benefits from things like, say, autoscaling or global low-latency access. Niches where you can get away with that are limited, not just by technical challenges but because large parts of the social ecosystem of IT won't like that. But they do exist. There's also still things that aren't webapps _at all_, there's software that has to run without internet access. It's all far apart and often requires specialized knowledge, but it exists.
      - zwnow 3y ago
        Yea I mostly learned web dev so far but wanted to get into IoT stuff so I might find something cool to do in there.
twosdai 3y ago
The same argument of UNIX design patterns (Single responsibility, well defined interfaces and communication protocals) vs Monolithic design patterns comes up a lot. I think that its mainly because both are effective at producing products, its just that they both have downsides.
- macksd 3y ago
  I read a meme yesterday about how you can just interject "it's all about finding that balance" into any meeting and people will just agree with you. I'm gonna say it here. Sometimes a flexible tool fits the bill well. Sometimes a specialized tool does. It's all about finding that balance. Thank you for coming to my TED talk.
- e12e 3y ago
  Just noting that sometimes one can do both: seperate postgres DBs/clusters for different use-case, seperate instances of a web server for TLS termination, caching, routing/rewriting, Ed:static asset serving. Benefit is orderly architecture, and fewer different dependencies.
diggan 3y ago
I think a lot of the industry struggles with the idea that maybe there is no "one size fits all", and what makes sense when you're a one person company with 100 customer probably doesn't make sense when you're a 1000 people company with millions of customers. If you use a stack meant for a huge userbase (with all the tradeoffs that comes with it) but you're still trying to find market fit, you're in for a disappointment Similarly, if you use a stack meant for smaller projects while having thousands of users relying on you, you're also in for a disappointment. It's OK to make a choice in the beginning based on the current context and environment, and then change when it no longer makes sense. Doesn't even have to be "technical debt", just "the right choice at that moment".
- davidw 3y ago
  > It's OK to make a choice in the beginning based on the current context and environment, and then change when it no longer makes sense. Yep. And Postgres is a really good choice to start with. Plenty of people won't outgrow it. Those who do find it's not meeting some need will, by the time they need to replace it, have a really good understanding of what that replacement looks like in detail, rather than just some hand-wavy "web scale".
  - tracker1 3y ago
    True enough and with modern hardware that barrier is relatively high. IIRC Stack overflow was handling several million users in a single database server over a decade ago... We've got over 8x the compute power and memory now. Still need to understand the data model and effects on queues though.
wg0 3y ago
I think SQS is cheap enough to build on as a messaging queue even if you're not hosting within AWS. Out of the widely underrated AWS services include SNS and SES and they are not a bad choice even if you're not using AWS for compute and storage.
- qaq 3y ago
  SQS is at least once PG can give you exactly once
  - qaq 3y ago
    Not sure why this is making people upset.
    - williamdclt 3y ago
      Because it's incorrect. If you have any non-postgres side-effect, you can't have exactly-once (unless you do 2PC or something like that). There isn't any technology that gives you "exactly once" in the general case.
      - qaq 3y ago
        That's not how exactly once is defined for queue. We are talking about semantics of what queue systems is providing.
        silon42 3y ago
        Nobody will understand it like that.
        qaq 3y ago
        Anyone who has ever selected queue service/product will understand it like that. Because thats one of the most prominent features that gets highlighted by those products: SQS Standard queues support at-least-once message delivery. NATS offers "at-most-once" delivery etc.
TheCapeGreek 3y ago
On top of that, a lot of discourse seems to happen with an assumption that you only make the tech/stack choice once. For the majority of apps, just doing basic CRUD with a handful of data types, is it that hard to just move to another DB? Especially if you're in framework land with an ORM that abstracts some of the differences, since your app code will largely stay the same.
- philihp 3y ago
  The way you've trivialized a database vendor swap makes me curious how often you do this.
brightball 3y ago
PG works really well as a message queue and there's several excellent implementations on top of it. Most systems are still going to need Redis involved just as a coordinator for other pub/sub related work unless you're using a stack that can handle it some other way (looking at BEAM here). But there are always going to be scenarios as an application grows where you'll find a need to scale specific pieces. Otherwise though, PostgreSQL by itself can get you very, very far.
- dfee 3y ago
  Worth noting that Postgres has a pubsub implementation built in: listen/notify. https://www.postgresql.org/docs/current/sql-notify.html https://www.postgresql.org/docs/current/sql-notify.html
  - brightball 3y ago
    Oh yea, definitely aware of it. I believe many of the queuing solutions utilize it as well. I've ready a lot of reports (on here) that it comes with several unexpected footguns if you really lean on it though.
  - cpursley 3y ago
    Yep, I should add that. One of the libraries in my list (that I maintain) is WalEx: https://github.com/cpursley/walex/issues https://github.com/cpursley/walex/issues It subscribes to the Postgres WAL and let you do the same sort of thing you can do with listen/notify, but without the drawbacks like need for triggers or character limits.
    - brightball 3y ago
      Going to add this to my research list.
      - cpursley 3y ago
        Ping me if you have any questions. Long time fan of your blog.
legohead 3y ago
My saying has always been: be nice to the DB Don't use it anymore than you have to for your application. Other than network IO it's the slowest part of your stack.
- niels_bom 3y ago
  Would you say it's slower than file IO too?
  - alternatex 3y ago
    Kind of irrelevant since a DB provides some guarantees that a simple file does not by default.
    - samtheprogram 3y ago
      GP was responding to a comment comparing it to network IO in terms of bottlenecks in your application stack ...?
  - Too 3y ago
    It’s not slow by itself. It’s a single point of bottleneck that will inevitably become slow as you cram everything into it.
    - dagss 3y ago
      ...but by trying to avoid the bottleneck and moving things to backend, you make things 10x worse resource wise for the DB. So it is not a easy tradeoff. Take any computation you can do in SQL like "select sum(..) ...". Should you do that in the database, or move each item over the network and sum them in the backend? Summing in the database uses a lot less resources FOR THE DB than the additional load the DB would get from "offloading" this to backend. More complex operations would typically also use 10x-100x less resources if you operate on sets and amortize the B-tree lookups over 1000 items. The answer is "it depends" and "understand what you are doing"; nothing about it is "inevitable". Trying to avoid computing in the DB is a nice way of thinking you maxed out the DB ...on 10% of what it should be capable of.
deathanatos 3y ago
I have certain experience with some technologies, e.g., SQS and Postgres. Say I'm on your team, and you're an application developer, and you need a queue. If you're taking the "we're small, this queue is small, just do it in PG for now and see if we ever grow out of that" — that's fine. "Let's use SQS, it's a well-established thing for this and we're already in AWS" — that's fine, I know SQS too. I've seen both of these decisions get made. (And both worked: the PG queue was never grown out of, and generally SQS was easy to work with & reliable.) But what I've also seen is "Let's introduce bespoke tech that nobody on the team, including the person introducing it, has experience in, for a queue that isn't even the main focus of what we're building" — this I'm less fine with. There needs to be a solid reason why we're doing that, and that we're going to get some real benefit, vs. something that the team does have experience in, like SQS or PG. Instead, this … thing … crashes on the regular, uses its own bespoke terminology, and you find out the documentation is … very empty. This does not make for a happy SRE.
- cnity 3y ago
  This desire can sometimes be so strong that people insist on truly wacky decisions. I have before demonstrated that Postgres performs perfectly well (and in fact exceeds) compared with a niche graph database, and heard some very strange reasons for why this approach should be avoided. A lot of the time you hear that it's engineers who chase shiny technology, but I've seen first hand what can happen when it's leadership.
  - tracker1 3y ago
    I've been on both sides of this.. Rabbit MQ and Elastic Search for a public facing site. The dedicated queue for workers to denormalize and push updates. To elastic. Why, because the $10k/month RDBMS servers couldn't handle the search load and were overly normalized. Definitely a hard sell. I've also seen literally hundreds of lambda functions connecting to dozens of dynamo databases. I'm firmly in the camp of use an RDBMS (PostgreSQL my first choice) for most things in most apps. A lot of times you can simply apply the lessons from other databases at scale in pg rather than something completely different. I'm also more than okay leveraging a cloud's own MQ option, it's usually easy enough to swap out as/if needed.
brlewis 3y ago
> you start to realize _why_ there's a stack, rather than just a single technology to rule them all Architecturally, there are other cases besides message queues where there's no reason for introducing another layer in the stack, once you have a database, other than just because SQL isn't anybody's favorite programming language. And that's the real reason there's a stack.
wvh 3y ago
That's a nicely balanced view. I've been working on the intersection between dev, sec and ops for many, many years and one of the most important lessons has been that every dependency is a liability. That liability is either complexity, availability, security, wasting resources or projects or key people disappearing. Do anything to avoid adding more service, library or technology dependencies; if necessary, let people have their side projects and technological playgrounds to distil future stacks out of. There are good reasons to go OLAP or graph for certain kinds of problems, but think carefully before adding more services and technologies because stuff has a tendency to go in easily but nothing ever leaves a project and you will inevitably end up with a bloated juggernaut that nobody can tame. And it's usually those people pushing the hardest for new technologies that are jumping into new projects when shit starts hitting the fan. If a company survives long enough (or cough government), a substantial and ever increasing amount of time, money and sec/ops effort will go into those dependencies and complexity cruft.
tetha 3y ago
This is very much the way I'm pushing in our internal development platform: I want to offer as little middlewares as possible, but as many as necessary. And ideally these systems are boring, established tech covering a lot of use cases. From there, Postgres ended up being our relational storage for the platform. It is a wonderful combination of supporting teams by being somewhat strict (in a flexible way) as well as supporting a large variety of use cases. And after some grumbling (because some teams had to migrate off of SQL Server, or off of MariaDB, and data migrations were a bit spicy), agreement is growing that it's a good decision to commit on a DB like this. We as the DB-Operators are accumulating a lot of experience running this lady and supporting the more demanding teams. And a lot of other teams can benefit from this, because many of the smaller applications either don't cause enough load on the Postgres Clusters to be even noticeable or we and the trailblazer teams have seen many of their problems already and can offer internally proven and understood solutions. And like this, we offer a relational storage, file storage, object storage and queues and that seems to be enough for a lot of applications. We're only now adding in Opensearch after a few years as a service now for search, vector storage and similar use cases.
agumonkey 3y ago
The more I do fullstack work the more I see an obesity crisis. I under the need to modularize (I dearly think I do) but god you have relational model, reimplemented in your framework, reencoded as a middleware to handle url parsing, the one more layer to help integrate things client side. I find that insane. And Postgrest was a refreshing idea.
- cpursley 3y ago
  Seriously. There's like 7000 duplicates of the very same data layer in a single stack: database, back-end ORM/data mapper, front end and various caching things in between. Things like PostgREST and Hasura area great pared with fluent clients.
  - agumonkey 3y ago
    And then there's the failed microservice case.. what some people describe a distributed monolith where data has to be passed around through every layer, with domain logic replicated here and there.

nextaccountic 3y ago

> But inevitably, as an application grows in complexity, Some applications never grow that much

ejb999 3y ago
I would go further and even say 'most' applications never grow that much.
LaGrange 3y ago
But it surely will! It will! See, now that we're profitable, we're gonna become a _scale up_, go international, hire 20 developers, turn everything into microservices, rewrite the UI our customers love, _not_ hire more customer service, get more investors, get pressured by investors, hire extra c-levels, lay-off 25 developers and the remaining customer service, write a wonderful journey post. The future is so bright!
- zwnow 3y ago
  Honestly, that's one of the reasons I never want to monetize my work and stay miles away from the software industry. Modern world is all web apps that require you to subscribe to 20 different 3rd party services to even build your app. So you rack up bills before your product is even remotely lucrative... Building an app with no third party dependencies seems impossible nowadays. At least if you plan to compete.
  - LaGrange 3y ago
    I mean, you _can_ host your staging environment on a Minisforum PC hidden in your closet and then deploy to Hetzner, and probably save a _ton_ unless your service benefits from things like, say, autoscaling or global low-latency access. Niches where you can get away with that are limited, not just by technical challenges but because large parts of the social ecosystem of IT won't like that. But they do exist. There's also still things that aren't webapps _at all_, there's software that has to run without internet access. It's all far apart and often requires specialized knowledge, but it exists.
    - zwnow 3y ago
      Yea I mostly learned web dev so far but wanted to get into IoT stuff so I might find something cool to do in there.

twosdai 3y ago

The same argument of UNIX design patterns (Single responsibility, well defined interfaces and communication protocals) vs Monolithic design patterns comes up a lot. I think that its mainly because both are effective at producing products, its just that they both have downsides.

macksd 3y ago
I read a meme yesterday about how you can just interject "it's all about finding that balance" into any meeting and people will just agree with you. I'm gonna say it here. Sometimes a flexible tool fits the bill well. Sometimes a specialized tool does. It's all about finding that balance. Thank you for coming to my TED talk.
e12e 3y ago
Just noting that sometimes one can do both: seperate postgres DBs/clusters for different use-case, seperate instances of a web server for TLS termination, caching, routing/rewriting, Ed:static asset serving. Benefit is orderly architecture, and fewer different dependencies.

diggan 3y ago

I think a lot of the industry struggles with the idea that maybe there is no "one size fits all", and what makes sense when you're a one person company with 100 customer probably doesn't make sense when you're a 1000 people company with millions of customers. If you use a stack meant for a huge userbase (with all the tradeoffs that comes with it) but you're still trying to find market fit, you're in for a disappointment Similarly, if you use a stack meant for smaller projects while having thousands of users relying on you, you're also in for a disappointment. It's OK to make a choice in the beginning based on the current context and environment, and then change when it no longer makes sense. Doesn't even have to be "technical debt", just "the right choice at that moment".

davidw 3y ago
> It's OK to make a choice in the beginning based on the current context and environment, and then change when it no longer makes sense. Yep. And Postgres is a really good choice to start with. Plenty of people won't outgrow it. Those who do find it's not meeting some need will, by the time they need to replace it, have a really good understanding of what that replacement looks like in detail, rather than just some hand-wavy "web scale".
- tracker1 3y ago
  True enough and with modern hardware that barrier is relatively high. IIRC Stack overflow was handling several million users in a single database server over a decade ago... We've got over 8x the compute power and memory now. Still need to understand the data model and effects on queues though.

wg0 3y ago

I think SQS is cheap enough to build on as a messaging queue even if you're not hosting within AWS. Out of the widely underrated AWS services include SNS and SES and they are not a bad choice even if you're not using AWS for compute and storage.

qaq 3y ago
SQS is at least once PG can give you exactly once
- qaq 3y ago
  Not sure why this is making people upset.
  - williamdclt 3y ago
    Because it's incorrect. If you have any non-postgres side-effect, you can't have exactly-once (unless you do 2PC or something like that). There isn't any technology that gives you "exactly once" in the general case.
    - qaq 3y ago
      That's not how exactly once is defined for queue. We are talking about semantics of what queue systems is providing.
      - silon42 3y ago
        Nobody will understand it like that.
        qaq 3y ago
        Anyone who has ever selected queue service/product will understand it like that. Because thats one of the most prominent features that gets highlighted by those products: SQS Standard queues support at-least-once message delivery. NATS offers "at-most-once" delivery etc.

TheCapeGreek 3y ago

On top of that, a lot of discourse seems to happen with an assumption that you only make the tech/stack choice once. For the majority of apps, just doing basic CRUD with a handful of data types, is it that hard to just move to another DB? Especially if you're in framework land with an ORM that abstracts some of the differences, since your app code will largely stay the same.

philihp 3y ago
The way you've trivialized a database vendor swap makes me curious how often you do this.

brightball 3y ago

PG works really well as a message queue and there's several excellent implementations on top of it. Most systems are still going to need Redis involved just as a coordinator for other pub/sub related work unless you're using a stack that can handle it some other way (looking at BEAM here). But there are always going to be scenarios as an application grows where you'll find a need to scale specific pieces. Otherwise though, PostgreSQL by itself can get you very, very far.

dfee 3y ago
Worth noting that Postgres has a pubsub implementation built in: listen/notify. https://www.postgresql.org/docs/current/sql-notify.html https://www.postgresql.org/docs/current/sql-notify.html
- brightball 3y ago
  Oh yea, definitely aware of it. I believe many of the queuing solutions utilize it as well. I've ready a lot of reports (on here) that it comes with several unexpected footguns if you really lean on it though.
- cpursley 3y ago
  Yep, I should add that. One of the libraries in my list (that I maintain) is WalEx: https://github.com/cpursley/walex/issues https://github.com/cpursley/walex/issues It subscribes to the Postgres WAL and let you do the same sort of thing you can do with listen/notify, but without the drawbacks like need for triggers or character limits.
  - brightball 3y ago
    Going to add this to my research list.
    - cpursley 3y ago
      Ping me if you have any questions. Long time fan of your blog.

legohead 3y ago

My saying has always been: be nice to the DB Don't use it anymore than you have to for your application. Other than network IO it's the slowest part of your stack.

niels_bom 3y ago
Would you say it's slower than file IO too?
- alternatex 3y ago
  Kind of irrelevant since a DB provides some guarantees that a simple file does not by default.
  - samtheprogram 3y ago
    GP was responding to a comment comparing it to network IO in terms of bottlenecks in your application stack ...?
- Too 3y ago
  It’s not slow by itself. It’s a single point of bottleneck that will inevitably become slow as you cram everything into it.
  - dagss 3y ago
    ...but by trying to avoid the bottleneck and moving things to backend, you make things 10x worse resource wise for the DB. So it is not a easy tradeoff. Take any computation you can do in SQL like "select sum(..) ...". Should you do that in the database, or move each item over the network and sum them in the backend? Summing in the database uses a lot less resources FOR THE DB than the additional load the DB would get from "offloading" this to backend. More complex operations would typically also use 10x-100x less resources if you operate on sets and amortize the B-tree lookups over 1000 items. The answer is "it depends" and "understand what you are doing"; nothing about it is "inevitable". Trying to avoid computing in the DB is a nice way of thinking you maxed out the DB ...on 10% of what it should be capable of.

deathanatos 3y ago

I have certain experience with some technologies, e.g., SQS and Postgres. Say I'm on your team, and you're an application developer, and you need a queue. If you're taking the "we're small, this queue is small, just do it in PG for now and see if we ever grow out of that" — that's fine. "Let's use SQS, it's a well-established thing for this and we're already in AWS" — that's fine, I know SQS too. I've seen both of these decisions get made. (And both worked: the PG queue was never grown out of, and generally SQS was easy to work with & reliable.) But what I've also seen is "Let's introduce bespoke tech that nobody on the team, including the person introducing it, has experience in, for a queue that isn't even the main focus of what we're building" — this I'm less fine with. There needs to be a solid reason why we're doing that, and that we're going to get some real benefit, vs. something that the team does have experience in, like SQS or PG. Instead, this … thing … crashes on the regular, uses its own bespoke terminology, and you find out the documentation is … very empty. This does not make for a happy SRE.

cnity 3y ago
This desire can sometimes be so strong that people insist on truly wacky decisions. I have before demonstrated that Postgres performs perfectly well (and in fact exceeds) compared with a niche graph database, and heard some very strange reasons for why this approach should be avoided. A lot of the time you hear that it's engineers who chase shiny technology, but I've seen first hand what can happen when it's leadership.
- tracker1 3y ago
  I've been on both sides of this.. Rabbit MQ and Elastic Search for a public facing site. The dedicated queue for workers to denormalize and push updates. To elastic. Why, because the $10k/month RDBMS servers couldn't handle the search load and were overly normalized. Definitely a hard sell. I've also seen literally hundreds of lambda functions connecting to dozens of dynamo databases. I'm firmly in the camp of use an RDBMS (PostgreSQL my first choice) for most things in most apps. A lot of times you can simply apply the lessons from other databases at scale in pg rather than something completely different. I'm also more than okay leveraging a cloud's own MQ option, it's usually easy enough to swap out as/if needed.

brlewis 3y ago

> you start to realize _why_ there's a stack, rather than just a single technology to rule them all Architecturally, there are other cases besides message queues where there's no reason for introducing another layer in the stack, once you have a database, other than just because SQL isn't anybody's favorite programming language. And that's the real reason there's a stack.

wvh 3y ago

That's a nicely balanced view. I've been working on the intersection between dev, sec and ops for many, many years and one of the most important lessons has been that every dependency is a liability. That liability is either complexity, availability, security, wasting resources or projects or key people disappearing. Do anything to avoid adding more service, library or technology dependencies; if necessary, let people have their side projects and technological playgrounds to distil future stacks out of. There are good reasons to go OLAP or graph for certain kinds of problems, but think carefully before adding more services and technologies because stuff has a tendency to go in easily but nothing ever leaves a project and you will inevitably end up with a bloated juggernaut that nobody can tame. And it's usually those people pushing the hardest for new technologies that are jumping into new projects when shit starts hitting the fan. If a company survives long enough (or cough government), a substantial and ever increasing amount of time, money and sec/ops effort will go into those dependencies and complexity cruft.

tetha 3y ago

This is very much the way I'm pushing in our internal development platform: I want to offer as little middlewares as possible, but as many as necessary. And ideally these systems are boring, established tech covering a lot of use cases. From there, Postgres ended up being our relational storage for the platform. It is a wonderful combination of supporting teams by being somewhat strict (in a flexible way) as well as supporting a large variety of use cases. And after some grumbling (because some teams had to migrate off of SQL Server, or off of MariaDB, and data migrations were a bit spicy), agreement is growing that it's a good decision to commit on a DB like this. We as the DB-Operators are accumulating a lot of experience running this lady and supporting the more demanding teams. And a lot of other teams can benefit from this, because many of the smaller applications either don't cause enough load on the Postgres Clusters to be even noticeable or we and the trailblazer teams have seen many of their problems already and can offer internally proven and understood solutions. And like this, we offer a relational storage, file storage, object storage and queues and that seems to be enough for a lot of applications. We're only now adding in Opensearch after a few years as a service now for search, vector storage and similar use cases.

agumonkey 3y ago

The more I do fullstack work the more I see an obesity crisis. I under the need to modularize (I dearly think I do) but god you have relational model, reimplemented in your framework, reencoded as a middleware to handle url parsing, the one more layer to help integrate things client side. I find that insane. And Postgrest was a refreshing idea.

cpursley 3y ago
Seriously. There's like 7000 duplicates of the very same data layer in a single stack: database, back-end ORM/data mapper, front end and various caching things in between. Things like PostgREST and Hasura area great pared with fluent clients.
- agumonkey 3y ago
  And then there's the failed microservice case.. what some people describe a distributed monolith where data has to be passed around through every layer, with domain logic replicated here and there.