3 ms·

Availability?

by anonymousDan 5y ago

Availability?

javajosh 5y ago
Why isn't "we're going to risk downtime to speed up our BTD loop for cheap" a good answer?
- NomDePlum 5y ago
  I've made that very decision. Partly forced due to internal resource constraints. However, swapping getting a working system out and tested with real users instead of waiting an undefined time to get high availability didn't lose me any sleep. It's also the case that some systems can withstand a degree of downtime that others can't, or it's not worth paying the cost for the perceived benefit gained.
JohnBooty 5y ago
Devops is not my area of expertise to put it mildly, but "a single big-ass server" scenarios are often pretty well covered by "a second big-ass server configured as a mirror/hot-spare", right? Depends on how many nines of uptime you need, of course, and other such things. (Obviously, there are reams of cases where this isn't sufficient)
- senorsmile 5y ago
  In 2009 I had a pair of (what I thought at the time) very cleverly architected VM servers with complete redundancy between them. Either one could be pulled and there'd maybe be a few seconds of data loss at the worst. One day lightning struck the building and even managed to "jump" the grounded pair of UPS's and fried all hard disks in both nodes. Thankfully I had both onsite and offsite backups; but it took a couple of weeks to order the new equipment, install it and set everything up again. Now I "devops" all the things and always have the ability to spin up all the services and restore from backups within minutes. This is the new standard in my opinion.
  - maccolgan 5y ago
    I think it's always better to have the mirror be located in a separate datacenter and then connect both via a SDN.
    - pclmulqdq 5y ago
      Why not both?
  - JohnBooty 5y ago
    Yeah, there definitely needs to be an offsite spare for sure.
  - exikyut 5y ago
    Two questions: 1. How would power conditioners and lightning arresters handle these sorts of situations? 2. Was this using VMWare fault tolerant VMs, just out of curiosity? (It continually live-migrates one VM to a hot spare over a 10G link so either can disappear and the VM continues to run.) Or was this a bespoke application architecture implementing effectively the same thing?

Availability?

javajosh 5y ago

Why isn't "we're going to risk downtime to speed up our BTD loop for cheap" a good answer?

NomDePlum 5y ago
I've made that very decision. Partly forced due to internal resource constraints. However, swapping getting a working system out and tested with real users instead of waiting an undefined time to get high availability didn't lose me any sleep. It's also the case that some systems can withstand a degree of downtime that others can't, or it's not worth paying the cost for the perceived benefit gained.

JohnBooty 5y ago

Devops is not my area of expertise to put it mildly, but "a single big-ass server" scenarios are often pretty well covered by "a second big-ass server configured as a mirror/hot-spare", right? Depends on how many nines of uptime you need, of course, and other such things. (Obviously, there are reams of cases where this isn't sufficient)

senorsmile 5y ago
In 2009 I had a pair of (what I thought at the time) very cleverly architected VM servers with complete redundancy between them. Either one could be pulled and there'd maybe be a few seconds of data loss at the worst. One day lightning struck the building and even managed to "jump" the grounded pair of UPS's and fried all hard disks in both nodes. Thankfully I had both onsite and offsite backups; but it took a couple of weeks to order the new equipment, install it and set everything up again. Now I "devops" all the things and always have the ability to spin up all the services and restore from backups within minutes. This is the new standard in my opinion.
- maccolgan 5y ago
  I think it's always better to have the mirror be located in a separate datacenter and then connect both via a SDN.
  - pclmulqdq 5y ago
    Why not both?
- JohnBooty 5y ago
  Yeah, there definitely needs to be an offsite spare for sure.
- exikyut 5y ago
  Two questions: 1. How would power conditioners and lightning arresters handle these sorts of situations? 2. Was this using VMWare fault tolerant VMs, just out of curiosity? (It continually live-migrates one VM to a hot spare over a 10G link so either can disappear and the VM continues to run.) Or was this a bespoke application architecture implementing effectively the same thing?