4 ms·

Is it actually making things less reliable to run things on a single instance though? In my experience, why things go down tend to be because humans deploy chan

Is it actually making things less reliable to run things on a single instance though? In my experience, why things go down tend to be because humans deploy changes, not because one instance went down because of hardware failures or connectivity issues. I've outlined more of this thought here too: https://news.ycombinator.com/item?id=33593361 https://news.ycombinator.com/item?id=33593361

salamander014 4y ago
It's more about what happens once there is an issue. I agree if something works and remains unchanged it will probably stay reliable, the issue is once you introduce a change, even nowadays you can possibly break something that isn't easy to undo (or troubleshoot without taking services down). For low traffic stuff, might seem OK, but no redundancy for a service that users pay for seems irresponsible, however Gitea might just be for them to track internal stuff, so maybe a little less so.
- capableweb 4y ago
  > It's more about what happens once there is an issue But again, when I've seen distributed infrastructure, it's very uncommon to see issues hitting one instance while others are not being hit by the very same issue. When you introduce change, you introduce it to all running instances, all at once or over a period of time. You might be in luck if you're doing blue/green deployment, but you can achieve the same thing with a staging + production environment mostly.
  - markstos 4y ago
    Hardware failures will hit one twin and not the other. Same for some kinds of high load triggers. Major changes like OS upgrades can be applied to one twin and then the other.
    - capableweb 4y ago
      How often to do hit actual hardware issues on dedicated instances for small to mid size applications? Usually you see those issues the first time you use it, or it get replaced data center wide and you get forewarning so you can move stuff over first, with minimal downtime. Same with OS upgrades, I can't remember the last time a OS upgrade completely borked anything I have deployed. At worst, the kernel didn't boot but recover from a backup was fast enough.
adql 4y ago
Running on one instance ties you to reliability of the hardware Running on multiple instance ties you to reliability of the hardware and reliability of the whatever software you use for cluster, which is also heavily dependant on your skill with that. So it will entirely depend on what you run. If you say run a cache server (Varnish or whatever else) behind some LB instead of single Varnish instance behind same LB, skill required to set it up is zero, extra software is zero so you end up with more reliability. We had exact zero problem with those kinds of setup for decade+ and it never caused any failure. On other hand, if you use something like Pacemaker (our usual config was pair of DRBD + DB nodes managed by Pacemaker) it is absolute nightmare to "get it right" and every mistake on your part in configuring it will make it less reliable, and there is plenty of edge cases to think of and handle, some might not even exist when you start configuring it. For example we had a hiccup at one point where initial config had too short timeouts to shut down database and it assumed it hanged, then killed it. Other times DB occasionally failed to start because pacemaker was (back in SysV days, no systemd) running /etc/init.d/db start & status in quick succession, and Java didn't managed to write PID yet which made status return "db is down", confusing the resource manager. We now have a bunch of setups like that also running rock solid but it took a bunch of time and setup to get it right. No extra cost to set it up really coz we have that in CM but for a time it definitely was lower actual uptime than "just a node with DB" And there are elasticsearch clusters that are just more stable with 3 nodes than one just because sometimes index could shit itself on crash...

salamander014 4y ago

It's more about what happens once there is an issue. I agree if something works and remains unchanged it will probably stay reliable, the issue is once you introduce a change, even nowadays you can possibly break something that isn't easy to undo (or troubleshoot without taking services down). For low traffic stuff, might seem OK, but no redundancy for a service that users pay for seems irresponsible, however Gitea might just be for them to track internal stuff, so maybe a little less so.

capableweb 4y ago
> It's more about what happens once there is an issue But again, when I've seen distributed infrastructure, it's very uncommon to see issues hitting one instance while others are not being hit by the very same issue. When you introduce change, you introduce it to all running instances, all at once or over a period of time. You might be in luck if you're doing blue/green deployment, but you can achieve the same thing with a staging + production environment mostly.
- markstos 4y ago
  Hardware failures will hit one twin and not the other. Same for some kinds of high load triggers. Major changes like OS upgrades can be applied to one twin and then the other.
  - capableweb 4y ago
    How often to do hit actual hardware issues on dedicated instances for small to mid size applications? Usually you see those issues the first time you use it, or it get replaced data center wide and you get forewarning so you can move stuff over first, with minimal downtime. Same with OS upgrades, I can't remember the last time a OS upgrade completely borked anything I have deployed. At worst, the kernel didn't boot but recover from a backup was fast enough.

adql 4y ago

Running on one instance ties you to reliability of the hardware Running on multiple instance ties you to reliability of the hardware and reliability of the whatever software you use for cluster, which is also heavily dependant on your skill with that. So it will entirely depend on what you run. If you say run a cache server (Varnish or whatever else) behind some LB instead of single Varnish instance behind same LB, skill required to set it up is zero, extra software is zero so you end up with more reliability. We had exact zero problem with those kinds of setup for decade+ and it never caused any failure. On other hand, if you use something like Pacemaker (our usual config was pair of DRBD + DB nodes managed by Pacemaker) it is absolute nightmare to "get it right" and every mistake on your part in configuring it will make it less reliable, and there is plenty of edge cases to think of and handle, some might not even exist when you start configuring it. For example we had a hiccup at one point where initial config had too short timeouts to shut down database and it assumed it hanged, then killed it. Other times DB occasionally failed to start because pacemaker was (back in SysV days, no systemd) running /etc/init.d/db start & status in quick succession, and Java didn't managed to write PID yet which made status return "db is down", confusing the resource manager. We now have a bunch of setups like that also running rock solid but it took a bunch of time and setup to get it right. No extra cost to set it up really coz we have that in CM but for a time it definitely was lower actual uptime than "just a node with DB" And there are elasticsearch clusters that are just more stable with 3 nodes than one just because sometimes index could shit itself on crash...