4 ms·
I think both of these are missing the fundamental question of: What does the business, or what do the product teams need? And what is the business willing to pa
by tetha 4y ago
I think both of these are missing the fundamental question of: What does the business, or what do the product teams need? And what is the business willing to pay for it?
We have a whole bunch of simple systems with very lax SLAs around. Those are just running as a single container in the orchestration, because upon failure, they just restart with a minute of downtime or five and it's available enough. For these systems it wouldn't make sense to really think about HA postgres clusters for example. It would be entirely fine to have 2 postgres instances with replication and alerting so the admin can trigger a failover as needed.
However, we also have systems with rather strict and ambitious SLAs, and we're being paid well for the availability of these systems. And at that point, the buisness decisions start to accrue - the company considers it a selling point to be self-hosting, we have ambitious SLAs, we have a lot of products settling on postgres.
At that point, it makes sense for the company to have 1-2 engineers focusing on an HA postgres setup and a couple more engineers who can handle it during an oncall incident. It took us like 2 years and a lot of head scratching to get where we are (which would be an unacceptable investment of time and money to a startup), but now these rock-solid database clusters are turning into an actual asset for sales and product development.
This has been my learning over the last 2-4 years overall: You kinda have to do the smallest and simplest thing to make the business requirement work. Oftentimes, a simple single node solution or a 2-node setup with some defined manual emergency handling works surprisingly well and you don't need any fully scalable auto-failover setup. In other times, big requirements require the big hammer, but that one doesn't come cheap.
- didgetmaster 4y agoCompanies that anticipate scaling needs well in advance and plan accordingly tend to make smoother transitions than if they are always reacting to the latest crisis. Too many wait until it hits the fan before they do what is necessary. Some small businesses will try to run a set of Excel spreadsheets as the 'company database' until it absolutely breaks on them once the load reaches a certain point. Similarly, companies that opt for a lighter-weight solution such as SQLite might have to scramble when they have to port all their data to Postgres or other database. In today's environment where a company can go from nothing to a massive data overload in a short period of time, it can be particularly challenging to anticipate what is coming their way. This is one reason why cloud computing is so popular.