6 ms·

Yes, perfs are better on bare metal, and it's cheaper too. In fact, even a cheap VPS will perform better than the cloud for much less money, and scale verticall

Yes, perfs are better on bare metal, and it's cheaper too. In fact, even a cheap VPS will perform better than the cloud for much less money, and scale vertically to incredible highs. You do have to do a lot of things manually, and even doing that, you won't get the high availability and elasticity you have the potential to get from a cloud offer. Potential being the key word though. But honestly, most projects don't need it. Crashing used to terrify me. One day I started to work for a client that had his service down once a month. Guess what ? Nothing happened. No customers ever complained. No money was lost, in fact, the cash flow kept growing. Most services are not so important they can't get switched off once in a while. Not to mention, monoliths are more robust they are given credit for.

omegalulw 5y ago
One thing I don't get is why not have an on prem solution with cloud fallback? It's hard but not super hard. You would just need a cloud data store. And depending on your app, you can have on prem data store that periodically backs up (and no egress under normal operation) if you can design with an eventual consistency models.
Nextgrid 5y ago
> you won't get the high availability Debatable. I've seen a fair amount of outages caused by the extra complexity brought on by making a system distributed for the purposes of high availability. Hardware is actually quite reliable nowadays, and I'll trust a hardware single-point of failure running a monolithic application more than a distributed microservice-based system with lots of moving parts. Sure, in theory, the distributed system should win, but in practice, it fails more often (due to operator error or unforeseen bugs) than hardware.
- KronisLV 5y ago
  > Sure, in theory, the distributed system should win, but in practice, it fails more often (due to operator error or unforeseen bugs) than hardware. Isn't this because of the rampant introduction of accidental complexity whenever you attempt to make a system horizontally scalable - e.g. for whatever reason the developers or the people in charge suddenly try to cram in as many technological solutions as possible because apparently that's what the large companies are doing? There's no reason why you couldn't think about which data can or cannot be shared, and develop your system as one that's scalable, possibly modular, but with the codebase still being largely monolithic in nature. I'd argue that there's a large gray area between the the opposite ends of that spectrum - monoliths vs microservices. I actually wrote down some thoughts in a blog post of mine, called "Moduliths: because we need to scale, but we also cannot afford microservices": https://blog.kronis.dev/articles/modulith-because-we-need-to-scale-but-we-also-cannot-afford-micro-services https://blog.kronis.dev/articles/modulith-because-we-need-to...
  - Nextgrid 5y ago
    I think another issue is that there are plenty of robust, battle-tested building blocks such as databases for monolithic, vertically-scalable applications, but much fewer for horizontally-scalable ones, meaning you'll need to roll your own (at the application layer) and most likely screw it up.
    - KronisLV 5y ago
      I do agree with you in that regard, however, that's also a dangerous line of thinking. There are attempts to provide horizontal scalability for RDBMSes in a transparent way, like TiDB https://pingcap.com/ https://pingcap.com/ (which is compatible with the MySQL 5.7 drivers), however, the list of functionality that's sacrificed to achieve easily extensible clusters is a long one: https://docs.pingcap.com/tidb/stable/mysql-compatibility https://docs.pingcap.com/tidb/stable/mysql-compatibility There are other technologies, like MongoDB, which sometimes are more successful at a clustered configuration, however most of the traditional RDBMSes work best in a leader-follower type of replication scenario, because even those aforementioned systems oftentimes have data consistency issues that may eventually pop up. Essentially, my argument is that the lack of good horizontally scalable databases or other data storage solutions is easily explainable by the fact that the problem itself isn't solvable in any easy way, apart from adopting eventual consistency, which is probably going to create more problems than it will solve in case of any pre-existing code that makes assumptions about what ways it'll be able to access data and operate on it: https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu... To that end, i'd perhaps like to suggest an alternative: use a single vertically scalable RDBMS instance when possible, with a hot standby if you have the resources for that. Let the architecture around it be horizontally scalable instead, and let it deal with the complexities of balancing the load and dealing with backpressure - introduce a message queue if you must, maybe even an in-memory one for simplicity's sake, or consider an event based architecture where "what needs to be done" is encapsulated within a data structure that can be passed around and applied whenever possible. In my eyes, such solutions can in many cases be better than losing the many benefits of having a single source of truth. Alternatively, when you actually hit issues with the above approach (and only then), consider sharding as a possibility, or, alternatively, do some domain driven design, figure out where to draw some boundaries and split your service into multiple ones that cover the domain with which you need to work with. Then you have one DB for sales, one for account management, one for reports and so on, all with services that are separated by something as simple as REST interfaces and with rate limits or any of the other mechanisms. If, however, neither of those two groups of approaches don't seem to be suitable for the loads that you're dealing with, then you probably have a team of very smart people and a large amount of resources to figure out what will work best. To sum up, if there are no good solutions in the space, perhaps that's because the problems themselves haven't been solved yet. Thus, sooner or later, they'll need to be sidestepped and their impact mitigated in whatever capacity is possible. Not all components can or should scale horizontally.
- BiteCode_dev 5y ago
  Yeah, I edited my answer because that's fair. Even AS 430 main frames with crazy uptimes are not unheard of after all.
- Too 5y ago
  The biggest benefit of HA architectures is IMO not resilience from crashing or overloaded systems, but more that it is often a prerequisite for doing zero-downtime updates and partial rollout of updates. No more emergency revert to last version because the new version didn't start up, only to realize there was a schema update so you also must restore your data from a snapshot, which nobody knows how to do. All this under stress from the service being completely down.
atoav 5y ago
Oh like that one system I saw once with an uptime of 10 years that was happily chugging away data (not web facing tho). Bare metal servers with a propper fallback and a working monitoring/notification system can be incredibly reliable and for most purposes definitely enough.
FpUser 5y ago
>"You do have to do a lot of things manually, and even doing that, you won't get the high availability and elasticity you get from a cloud offer." I run my things on bare metal. In case of hardware failure it takes less than 1 hour to restore servers from backup and there was exactly zero hardware failures for all its lifetime anyways. I also have configurations with standby servers but I am questioning it now as again there was not a single time (bar testing) when standby was needed. As for performance - I use native C++ backends and modern multicore CPU with lots of RAM. Those babies can process thousands of requests per second sustainably without ever breaking a sweat. It is more than enough for any reasonable business. All while costing fractions of a cloudy stuff.
- Nextgrid 5y ago
  With ZFS or even RAID, there should ideally never be a need to "restore from backup" because of a conventional hardware failure; storage drive malfunctions nowadays can and (IMO) should be resolved online. This is of course not a reason to avoid backups, but nowadays "restoring from backups" should be because of operator error or large-scale disaster (fire, etc), not because of storage drive failure. Nowadays I'd be more worried about compute hardware failure - think CPU, RAM or the system board. Storage redundancy is IMO a long-solved problem provided you don't cheap out.
  - BiteCode_dev 5y ago
    Can you educate me on the typical setup you would use to achieve that goal?
    - Nextgrid 5y ago
      Either ZFS or some form of RAID with redundancy cranked up to the max, so that it can tolerate a high number of drives failing and still continue operating. ZFS configured in RAIDZ3 mode for example can tolerate 3 drives failing and still be able to operate (and rebuild itself once the failed drives are replaced). You are very unlikely to have 3 drives fail at the same time, so you're pretty much immune to storage hardware failures. Again, this is not an argument against backups (they're useful for other reasons), but with this amount of redundancy on the storage level, I'd expect something else in the system to die before the storage layer fails catastrophically.
hinkley 5y ago
I have a service at work that only needs to be up during a couple of 4 hour intervals every week. It's one of the least stressful elements of my job. There's a cost to always-on systems, and I don't think we're accounting those costs properly (externalities). It's likely that in many cases the benefits do not outweigh the costs. I think it comes down to your ability to route around a problem. If the budgeting or ordering system is down, there are any number of other tasks most of your people can be doing. They can make a note, and stay productive for hours or days. If you put all of your software on one server or one provider, and that goes down, then your people are pretty much locked out of everything. Partial availability needs to be prioritized over total availability, because of the Precautionary Principle. The cost of everything being down is orders of magnitude higher than the cost of having some things down.
PragmaticPulp 5y ago
> ...and it's cheaper too... > You do have to do a lot of things manually... Doing those manual things has a cost, though. In most cases, it's only cheaper if you value engineering time at $0. Fine for a personal project, but the math changes when you're paying a fully-loaded cost of $100-200/hr for engineers and you have a long backlog of more valuable things for them to work on. That's the real reason companies use cloud services: Not because it's cheaper or faster, but because it lets the engineers focus their efforts on the things that differentiate the company. From another perspective: I can change the oil in my car cheaper and faster than driving to a shop and having someone else do it. However, if my time is better spent doing an hour of freelancing work then I'll gladly pay the shop to change my oil while I work.
- colechristensen 5y ago
  Eh, everything can be automated just as well on bare metal. "The cloud" tends to add complexity not remove it, or at best replace one kind of complexity with another. Bare metal tooling is less familiar and could use some advancement, but basically anything that a cloud compute provider can do can be done on bare metal. A lot of orgs just never bothered to learn how to automate bare metal and ended up doing a lot of manual work.
  - PragmaticPulp 5y ago
    > Eh, everything can be automated just as well on bare metal. Right, but that's not free. Anything can be automated to match the level of commercial cloud providers given sufficient time and money, but it's far from free. It's easy to underestimate engineering costs when they're hidden behind a fixed annual salary, but once you're looking at the cost/benefit analysis and actually accounting for all the time spent managing bare metal, it's not a good tradeoff for small or even medium size companies.
    - colechristensen 5y ago
      People universally build big complicated systems around cloud providers which are equally not free. Unless you are doing something extremely basic, this is what gets done, with a good deal of experience it seems the complexity of what people do to manage and automate cloud providers is more or less equal to what they do to manage bare metal hardware well.

omegalulw 5y ago

One thing I don't get is why not have an on prem solution with cloud fallback? It's hard but not super hard. You would just need a cloud data store. And depending on your app, you can have on prem data store that periodically backs up (and no egress under normal operation) if you can design with an eventual consistency models.

Nextgrid 5y ago

> you won't get the high availability Debatable. I've seen a fair amount of outages caused by the extra complexity brought on by making a system distributed for the purposes of high availability. Hardware is actually quite reliable nowadays, and I'll trust a hardware single-point of failure running a monolithic application more than a distributed microservice-based system with lots of moving parts. Sure, in theory, the distributed system should win, but in practice, it fails more often (due to operator error or unforeseen bugs) than hardware.

KronisLV 5y ago
> Sure, in theory, the distributed system should win, but in practice, it fails more often (due to operator error or unforeseen bugs) than hardware. Isn't this because of the rampant introduction of accidental complexity whenever you attempt to make a system horizontally scalable - e.g. for whatever reason the developers or the people in charge suddenly try to cram in as many technological solutions as possible because apparently that's what the large companies are doing? There's no reason why you couldn't think about which data can or cannot be shared, and develop your system as one that's scalable, possibly modular, but with the codebase still being largely monolithic in nature. I'd argue that there's a large gray area between the the opposite ends of that spectrum - monoliths vs microservices. I actually wrote down some thoughts in a blog post of mine, called "Moduliths: because we need to scale, but we also cannot afford microservices": https://blog.kronis.dev/articles/modulith-because-we-need-to-scale-but-we-also-cannot-afford-micro-services https://blog.kronis.dev/articles/modulith-because-we-need-to...
- Nextgrid 5y ago
  I think another issue is that there are plenty of robust, battle-tested building blocks such as databases for monolithic, vertically-scalable applications, but much fewer for horizontally-scalable ones, meaning you'll need to roll your own (at the application layer) and most likely screw it up.
  - KronisLV 5y ago
    I do agree with you in that regard, however, that's also a dangerous line of thinking. There are attempts to provide horizontal scalability for RDBMSes in a transparent way, like TiDB https://pingcap.com/ https://pingcap.com/ (which is compatible with the MySQL 5.7 drivers), however, the list of functionality that's sacrificed to achieve easily extensible clusters is a long one: https://docs.pingcap.com/tidb/stable/mysql-compatibility https://docs.pingcap.com/tidb/stable/mysql-compatibility There are other technologies, like MongoDB, which sometimes are more successful at a clustered configuration, however most of the traditional RDBMSes work best in a leader-follower type of replication scenario, because even those aforementioned systems oftentimes have data consistency issues that may eventually pop up. Essentially, my argument is that the lack of good horizontally scalable databases or other data storage solutions is easily explainable by the fact that the problem itself isn't solvable in any easy way, apart from adopting eventual consistency, which is probably going to create more problems than it will solve in case of any pre-existing code that makes assumptions about what ways it'll be able to access data and operate on it: https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu... To that end, i'd perhaps like to suggest an alternative: use a single vertically scalable RDBMS instance when possible, with a hot standby if you have the resources for that. Let the architecture around it be horizontally scalable instead, and let it deal with the complexities of balancing the load and dealing with backpressure - introduce a message queue if you must, maybe even an in-memory one for simplicity's sake, or consider an event based architecture where "what needs to be done" is encapsulated within a data structure that can be passed around and applied whenever possible. In my eyes, such solutions can in many cases be better than losing the many benefits of having a single source of truth. Alternatively, when you actually hit issues with the above approach (and only then), consider sharding as a possibility, or, alternatively, do some domain driven design, figure out where to draw some boundaries and split your service into multiple ones that cover the domain with which you need to work with. Then you have one DB for sales, one for account management, one for reports and so on, all with services that are separated by something as simple as REST interfaces and with rate limits or any of the other mechanisms. If, however, neither of those two groups of approaches don't seem to be suitable for the loads that you're dealing with, then you probably have a team of very smart people and a large amount of resources to figure out what will work best. To sum up, if there are no good solutions in the space, perhaps that's because the problems themselves haven't been solved yet. Thus, sooner or later, they'll need to be sidestepped and their impact mitigated in whatever capacity is possible. Not all components can or should scale horizontally.
BiteCode_dev 5y ago
Yeah, I edited my answer because that's fair. Even AS 430 main frames with crazy uptimes are not unheard of after all.
Too 5y ago
The biggest benefit of HA architectures is IMO not resilience from crashing or overloaded systems, but more that it is often a prerequisite for doing zero-downtime updates and partial rollout of updates. No more emergency revert to last version because the new version didn't start up, only to realize there was a schema update so you also must restore your data from a snapshot, which nobody knows how to do. All this under stress from the service being completely down.

atoav 5y ago

Oh like that one system I saw once with an uptime of 10 years that was happily chugging away data (not web facing tho). Bare metal servers with a propper fallback and a working monitoring/notification system can be incredibly reliable and for most purposes definitely enough.

FpUser 5y ago

>"You do have to do a lot of things manually, and even doing that, you won't get the high availability and elasticity you get from a cloud offer." I run my things on bare metal. In case of hardware failure it takes less than 1 hour to restore servers from backup and there was exactly zero hardware failures for all its lifetime anyways. I also have configurations with standby servers but I am questioning it now as again there was not a single time (bar testing) when standby was needed. As for performance - I use native C++ backends and modern multicore CPU with lots of RAM. Those babies can process thousands of requests per second sustainably without ever breaking a sweat. It is more than enough for any reasonable business. All while costing fractions of a cloudy stuff.

Nextgrid 5y ago
With ZFS or even RAID, there should ideally never be a need to "restore from backup" because of a conventional hardware failure; storage drive malfunctions nowadays can and (IMO) should be resolved online. This is of course not a reason to avoid backups, but nowadays "restoring from backups" should be because of operator error or large-scale disaster (fire, etc), not because of storage drive failure. Nowadays I'd be more worried about compute hardware failure - think CPU, RAM or the system board. Storage redundancy is IMO a long-solved problem provided you don't cheap out.
- BiteCode_dev 5y ago
  Can you educate me on the typical setup you would use to achieve that goal?
  - Nextgrid 5y ago
    Either ZFS or some form of RAID with redundancy cranked up to the max, so that it can tolerate a high number of drives failing and still continue operating. ZFS configured in RAIDZ3 mode for example can tolerate 3 drives failing and still be able to operate (and rebuild itself once the failed drives are replaced). You are very unlikely to have 3 drives fail at the same time, so you're pretty much immune to storage hardware failures. Again, this is not an argument against backups (they're useful for other reasons), but with this amount of redundancy on the storage level, I'd expect something else in the system to die before the storage layer fails catastrophically.

hinkley 5y ago

I have a service at work that only needs to be up during a couple of 4 hour intervals every week. It's one of the least stressful elements of my job. There's a cost to always-on systems, and I don't think we're accounting those costs properly (externalities). It's likely that in many cases the benefits do not outweigh the costs. I think it comes down to your ability to route around a problem. If the budgeting or ordering system is down, there are any number of other tasks most of your people can be doing. They can make a note, and stay productive for hours or days. If you put all of your software on one server or one provider, and that goes down, then your people are pretty much locked out of everything. Partial availability needs to be prioritized over total availability, because of the Precautionary Principle. The cost of everything being down is orders of magnitude higher than the cost of having some things down.

PragmaticPulp 5y ago

> ...and it's cheaper too... > You do have to do a lot of things manually... Doing those manual things has a cost, though. In most cases, it's only cheaper if you value engineering time at $0. Fine for a personal project, but the math changes when you're paying a fully-loaded cost of $100-200/hr for engineers and you have a long backlog of more valuable things for them to work on. That's the real reason companies use cloud services: Not because it's cheaper or faster, but because it lets the engineers focus their efforts on the things that differentiate the company. From another perspective: I can change the oil in my car cheaper and faster than driving to a shop and having someone else do it. However, if my time is better spent doing an hour of freelancing work then I'll gladly pay the shop to change my oil while I work.

colechristensen 5y ago
Eh, everything can be automated just as well on bare metal. "The cloud" tends to add complexity not remove it, or at best replace one kind of complexity with another. Bare metal tooling is less familiar and could use some advancement, but basically anything that a cloud compute provider can do can be done on bare metal. A lot of orgs just never bothered to learn how to automate bare metal and ended up doing a lot of manual work.
- PragmaticPulp 5y ago
  > Eh, everything can be automated just as well on bare metal. Right, but that's not free. Anything can be automated to match the level of commercial cloud providers given sufficient time and money, but it's far from free. It's easy to underestimate engineering costs when they're hidden behind a fixed annual salary, but once you're looking at the cost/benefit analysis and actually accounting for all the time spent managing bare metal, it's not a good tradeoff for small or even medium size companies.
  - colechristensen 5y ago
    People universally build big complicated systems around cloud providers which are equally not free. Unless you are doing something extremely basic, this is what gets done, with a good deal of experience it seems the complexity of what people do to manage and automate cloud providers is more or less equal to what they do to manage bare metal hardware well.