4 ms·

The thing that worries me about stories like this is that there is frequently (as is the case here) no mention any sort of HA or backups. No details on what dis

The thing that worries me about stories like this is that there is frequently (as is the case here) no mention any sort of HA or backups. No details on what disaster recovery looks like. Those are business critical considerations that cost money, and just disappear from the discussion when people say "hey i saved all this money dropping everything down to a single server!"

FpUser 7y ago
I am in the same boat (writing native servers). I will also "disappear" if you start asking me about HA/backups/etc. A particular solutions are very much case specific and can depend on business conducting rules just as much as on pure tech factors. Properly answering your question requires way too much writing and hardly a subject of a single post. I have HA solutions for the products I built but this post is the extent I am willing to talk about the subject ;)
- cthalupa 7y ago
  I'm not asking about the specifics, and I don't really care about them. But the fact of the matter is quite simply is that any single physical server solution will never be satisfactory for backups or HA purposes. You can't store your backups in the same place as your data and call it good - what do you do when you have multiple disks fail and your RAID can't be rebuilt? This happens. What do you do when operator error accidentally destroys the array? This happens. What do you do when there's a datacenter fire and the server burns up? This happens. None of these things should be a business ending event, but if you only have a single server handling quintuple duty, that's what it has a real chance of being. If you need HA, a single server isn't good enough even if you've got multiple VMs running the service. What do you do when the utility power is out and the generator fails to kick on properly or they run out of fuel? Both of those happen pretty frequently. What do you do when there's a network outage at the DC? This happens. When someone fucks up BGP somewhere and now the prefix you used is being routed to god knows where? This happens. When you have any sort of physical server failure that brings your single box down? This happens. Any of those situations will take you offline and render your HA meaningless. I'm not saying that any of these things are impossible to do when colocating your hardware - but they're not free, and they're not mentioned even at a high level in this story. And since we're not talking about running benchmarks on price/performance and instead talking about a service that a business needs to keep available to their customers to make money, these are important aspects to talk about. Everywhere I've worked, keeping our services available and being able to recover from hardware failure are far more important priorities than being able to optimize performance. And doing those things properly takes more than one physical server.
  - bsenftner 7y ago
    I can't speak for FpUser, but you misunderstand the idea of creating a single server: one does not just run one of them, they know about and expect multiple copies of themselves to be running at different IP addresses, and they synchronize with one another, as well as maintain individual backups that additional background processes validate between different instances. Each and every one of your disaster scenarios is handled by the architecture. Each an every one of your disaster scenarios has happened and we've lived through them, as well as after the fact reviewed and optimized how we handled the events. As you are, we're professionals.
bsenftner 7y ago
Well, in my case described above, single server solutions include an automated backup sub-system, and my servers expect multiple instances of itself to running on the client network, and these multiple instances synchronize with one another via additional endpoints specific for the purpose. The whole issue of HA and backups is critical and one of the areas my approach shines.
- cthalupa 7y ago
  You're still not actually answering the question of how you are HA and backing things up with only a single physical server and nothing else. If you're backing things up to the same server, that's not enough. If the HA instances are running in the same server, that's not enough. If there are other things besides that one physical server and it's power/network, you didn't include them in the cost, so the comparison is disingenuous there.
  - bsenftner 7y ago
    I do say the deployed system ends up being multiple instances of my single server, which synchronize with one another. Those are separate physical devices each running my one server. Additionally, when a backup runs the data is stored locally as well as on a physical storage device separate from the hardware it is running. Typically clients already have a firewall/router which is used to distribute requests to the various instances. My deployed systems are not one server, they become a server mesh.
    - cthalupa 7y ago
      >. Those are separate physical devices each running my one server. Additionally, when a backup runs the data is stored locally as well as on a physical storage device separate from the hardware it is running. Typically clients already have a firewall/router which is used to distribute requests to the various instances. Awesome! Really glad to hear this is the case. But those are all added costs beyond the single physical server you gave the price of. Server cost * number of physical servers you have deployed Cost of your off-server storage Cost of your network appliance doing load balancing It's still probably way less than the AWS bill, but it's not really fair to compare the total price of infrastructure in one environment vs. just a portion of the other.

FpUser 7y ago

I am in the same boat (writing native servers). I will also "disappear" if you start asking me about HA/backups/etc. A particular solutions are very much case specific and can depend on business conducting rules just as much as on pure tech factors. Properly answering your question requires way too much writing and hardly a subject of a single post. I have HA solutions for the products I built but this post is the extent I am willing to talk about the subject ;)

cthalupa 7y ago
I'm not asking about the specifics, and I don't really care about them. But the fact of the matter is quite simply is that any single physical server solution will never be satisfactory for backups or HA purposes. You can't store your backups in the same place as your data and call it good - what do you do when you have multiple disks fail and your RAID can't be rebuilt? This happens. What do you do when operator error accidentally destroys the array? This happens. What do you do when there's a datacenter fire and the server burns up? This happens. None of these things should be a business ending event, but if you only have a single server handling quintuple duty, that's what it has a real chance of being. If you need HA, a single server isn't good enough even if you've got multiple VMs running the service. What do you do when the utility power is out and the generator fails to kick on properly or they run out of fuel? Both of those happen pretty frequently. What do you do when there's a network outage at the DC? This happens. When someone fucks up BGP somewhere and now the prefix you used is being routed to god knows where? This happens. When you have any sort of physical server failure that brings your single box down? This happens. Any of those situations will take you offline and render your HA meaningless. I'm not saying that any of these things are impossible to do when colocating your hardware - but they're not free, and they're not mentioned even at a high level in this story. And since we're not talking about running benchmarks on price/performance and instead talking about a service that a business needs to keep available to their customers to make money, these are important aspects to talk about. Everywhere I've worked, keeping our services available and being able to recover from hardware failure are far more important priorities than being able to optimize performance. And doing those things properly takes more than one physical server.
- bsenftner 7y ago
  I can't speak for FpUser, but you misunderstand the idea of creating a single server: one does not just run one of them, they know about and expect multiple copies of themselves to be running at different IP addresses, and they synchronize with one another, as well as maintain individual backups that additional background processes validate between different instances. Each and every one of your disaster scenarios is handled by the architecture. Each an every one of your disaster scenarios has happened and we've lived through them, as well as after the fact reviewed and optimized how we handled the events. As you are, we're professionals.

bsenftner 7y ago

Well, in my case described above, single server solutions include an automated backup sub-system, and my servers expect multiple instances of itself to running on the client network, and these multiple instances synchronize with one another via additional endpoints specific for the purpose. The whole issue of HA and backups is critical and one of the areas my approach shines.

cthalupa 7y ago
You're still not actually answering the question of how you are HA and backing things up with only a single physical server and nothing else. If you're backing things up to the same server, that's not enough. If the HA instances are running in the same server, that's not enough. If there are other things besides that one physical server and it's power/network, you didn't include them in the cost, so the comparison is disingenuous there.
- bsenftner 7y ago
  I do say the deployed system ends up being multiple instances of my single server, which synchronize with one another. Those are separate physical devices each running my one server. Additionally, when a backup runs the data is stored locally as well as on a physical storage device separate from the hardware it is running. Typically clients already have a firewall/router which is used to distribute requests to the various instances. My deployed systems are not one server, they become a server mesh.
  - cthalupa 7y ago
    >. Those are separate physical devices each running my one server. Additionally, when a backup runs the data is stored locally as well as on a physical storage device separate from the hardware it is running. Typically clients already have a firewall/router which is used to distribute requests to the various instances. Awesome! Really glad to hear this is the case. But those are all added costs beyond the single physical server you gave the price of. Server cost * number of physical servers you have deployed Cost of your off-server storage Cost of your network appliance doing load balancing It's still probably way less than the AWS bill, but it's not really fair to compare the total price of infrastructure in one environment vs. just a portion of the other.