9 ms·

We might want to regularly keep track of how important each server is

226 points by pabs3 3y ago

tlb 3y ago
Or you might want to have redundant cooling. Cooling system prices seem to scale fairly linearly with the cooling power above a few kW, so instead of one 100 kW system you could buy four 25 kW systems so a single failure won't be a disaster.
- nine_k 3y ago
  Won't 4 x 25 kW systems mean also 4x the installation cost?
  - mdekkers 3y ago
    What the % of install cost over the projected lifetime of the system?
  - dist-epoch 3y ago
    Probably less if you install them all at the same time.
  - bluGill 3y ago
    Maybe, but costs are not linear and the nonlinear goes different ways for different parts of the install. The costs of the smaller systems installed could be cheaper than just the large system not installed if the smaller systems are standard parts.
- throw0101a 3y ago
  > Or you might want to have redundant cooling. Can you provide a cost centre or credit card for which they can bill this to? In case you didn't notice the domain, it is UToronto: academic departments aren't generally flush with cash. Further, you have to have physical space to fit the extra cooling equipment and pipes: not always easy or possible to do in old university buildings.
  - wongarsu 3y ago
    If you designed the system like this from the start or when replacing it anyways, N+1 redundancy might not me much more expensive than one big cooling unit. The systems can mostly share their ductwork and just have redundancy in the active components, so mostly the chillers. Of course these systems only get replaced every couple decades, if ever, so they are pretty much stuck with the setup they have.
    - throw0101b 3y ago
      > If you designed […] University department IT is not designed, it grows over decades. At some point some benefactor may pay for a new building and the department will move, so that could be a chance to actually design. But modern architecture and architects don't really go well with hosting lots of servers in what is ostensibly office space. I've been involved in the build-out of buildings/office space on three occasions in my career, and trying to get a decent IT space pencilled has always been like pulling teeth.
    - bluGill 3y ago
      > Of course these systems only get replaced every couple decades, if ever This is despite the massive energy savings they could get if they replaced those older systems. Universities often are full of old buildings with terrible insulation heated/cooled by very old/inefficient systems. In 20 years they would be money ahead by tearing down most buildings on campus and rebuilding to modern standards - assuming energy costs don't go up which seems unlikely) But they consider all those old buildings historic and so won't.
      - light_hue_1 3y ago
        > In 20 years they would be money ahead by tearing down most buildings on campus and rebuilding to modern standards - assuming energy costs don't go up which seems unlikely) But they consider all those old buildings historic and so won't. It has nothing to do with considering those building historic. The problem is unless someone wants to donate a $50-100M, new buildings don't happen. And big donors want to donate to massive causes "Build a new building to cure cancer!" not "This building is kind of crappy, let's replace it with a better one". It doesn't matter that over 50 years something could be cheaper if there's no money to fix it now.
- marcus0x62 3y ago
  With that model, you’d probably want 5 instead of 4 (N+1), but the other thing to consider is if you can duct the cold air to where it needs to go when one or more of the units has failed.
- dotancohen 3y ago
  This will mean just about 4 times the number of failures, too. And can 75% cooling still cool the server room anyway?
  - donkeyd 3y ago
    Maybe not, but some cooling means less servers to shut down.
  - bluGill 3y ago
    It means 5 times the number of failures as you intentionally put in an extra unit so that one can be taken offline at any time for maintenance (which itself will keep the whole system more reliable), and if one fails the whole keeps up. The cost is only slightly more to do this when there are 5 smaller units. Those smaller units could be standard off the shelf units as well, so it could be cheaper than a large unit that isn't mad in as large a quantity - this is a consideration that needs to be made case by case) Even if you cheap out and only install 4 units, odds are your failure doesn't happen on the hottest day of the year and so 3 can keep up just fine. It is only when you are unlikely that you need to shut anything down.
    - dotancohen 3y ago
      > Even if you cheap out and only install 4 units, odds are your failure doesn't happen on the hottest day of the year Actually, historically many HVAC failures happen when the system is running at max capacity.
  - evilduck 3y ago
    Four service degradations vs. one huge outage event. Pick your poison.
cowsandmilk 3y ago
It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.
- throw0101a 3y ago
  > Author: proudly declares himself Unix herder, wants to keep track of which systems are important. Because not all environments are webapps which dozens or hundreds of systems configured in a cookie cutter matter. Plenty of IT environments have pets because plenty of IT environments are not Web Scale™. And plenty of IT environments have staff churn and legacy systems where knowledge can become murky (see reference about "archaeology"). An IMAP server is different than a web server is different than an NFS server, and there may also be inter-dependencies between them.
  - cookiemonster9 3y ago
    So much this. Not everything fits cattle/chicken/etc models. Even in cases where those models could fit, they are not necessarily the right choice, given staffing, expertise, budgets, and other factors.
  - LeonB 3y ago
    I work with one system where the main entity is a “sale” which is processed beginning to end in some fraction of a second. A different system I work with, the main “entity” is, conceptually, more like a murder investigation. Less than 200 of the main entity are created in a year. Many different pieces of information are painstakingly gathered and tracked over a long period of time, with input from many people and oversight from legal experts and strong auditing requirements. Trying to apply the best lessons and principles from one system to the other is rarely a good idea. These kind of characteristics of different systems make a lot of difference to their care and feeding.
- karmarepellent 3y ago
  I would argue even the "wider industry" still administrates systems that must be treated as pets because they were not designed to be treated as cattle.
karmarepellent 3y ago
It's generally a good idea to have some documentation that states what a machine is used for, the "service", and how important said service is relative to others. At my company we kind of enforce this by not operating machines that are not explicitly assigned to any service. However you have to anticipate that the quality of documentation still varies immensely which might result in you shutting down a service that is actually more important than stated. Fortunately documentation improves after every outage because service owners reiterate on their part of the documentation when their service was shut down as "it appeared unimportant". It's a process.
spydum 3y ago
Asset management is definitely a thing. Tag your environments, tag your apps, and provide your apps criticality ratings based on how important they are to running the business. Then it's a matter of a query to know which servers can be shut, and which absolutely must remain.
- ethbr1 3y ago
  > provide your apps criticality ratings based on how important they are to running the business In a decentralized, self-service model, you can add "deal with convincing a stakeholder their app is anything less than most-critical." Although it usually works itself out if higher-criticality imposes ongoing time commitments on them as well (aka stick).
  - j33zusjuice 3y ago
    That seems like a poorly run company. Idk. Maybe we’ve worked in very different environments, but devs have almost always been aware of the criticality of the app, so convincing people wasn’t hard. In most places, the answer hinges on “is it customer facing?” and/or “does it break a core part of our business?” If the answer is no to both, it’s not critical, and everyone understands that. There’s always some weird outlier, “well, this runs process B to report on process A, and sends a custom report to the CEO …”, but hopefully those exceptions are rare.
    - spydum 3y ago
      agree. ethbr1 is 100% right about this being a problem; if politics is driving your criticality rating, it's probably being done wrong. it should be as simple as your statement, being mindful of some of those downstream systems that aren't always obviously critical (until they are unavailable for $time) edit: whoops, maybe I read the meaning backward, but both issues exist!
    - hinkley 3y ago
      I maintain a couple of apps that are pretty much free to break or to revert back to an older build without much consequence, except for one day a week, when half the team uses them instead of just me. Any other day I can use them to test new base images, new coding or deployment techniques, etc. I just have to put things back by the end of our cycle.
marcus0x62 3y ago
Years ago during a week-long power outage, a telephone central office where we had some equipment suffered a generator failure. The telephone company had a backup plan (they kept one generator on a trailer in the city for such a contingency,) and they had battery capacity[0] for their critical equipment to last until the generator was hooked up. They did have to load shed, though: they just turned off the AC inverters. They figured anything critical in a central office was on DC power, and if you had something on AC, you were just going to have to wait until the backup-backup generator was installed. 0 - at the time, at least, CO battery backup was usually sized for 24 hours of runtime.
- Scoundreller 3y ago
  > had some equipment suffered a generator failure. The telephone company had a backup plan (they kept one generator on a trailer in the city for such a contingency Well that’s more forward thinking than AT&T’s Nashville CO when it got bombed. They just depended on natgas grid for their generators if grid electric went out. They underestimated the correlation between energy grids. When natgas got cutoff and the UPSs died, they had no ability to hook in roll-up generators and had to hastily install connection points (or hardline them). And no standby contracts for them.
  - marcus0x62 3y ago
    To be fair, the CO I described[0] is a local switching center. The Nashville “CO” was the area long lines/tandem/long distance site. Now, I’m not defending the lack of a backup plan, but tandems tend to be far larger than a typical local/class 5 CO. 0 - which like the Nashville long lines building also happens to be a pre-divestiture AT&T site, although at the time it was owned by BellSouth. It is also in Tennessee, although in Memphis, not Nashville.
yevpats 3y ago
Check out CloudQuery - https://github.com/cloudquery/cloudquery https://github.com/cloudquery/cloudquery for an easy cloud asset inventory.
znlfl2 3y ago
[flagged]
- Scubabear68 3y ago
  I just did a due diligence with a company that only had servers in one data center. They were supremely confident that there was no way a whole DC could be impacted by an event.
  - Retr0id 3y ago
    fwiw you're likely replying to someone's GPT bot
radiowave 3y ago
Tracking servers is one thing, but tracking the dependency relationships among them is likely at least as important.
- j33zusjuice 3y ago
  I’m really glad we realized that before disaster struck. We have a project in-progress to do exactly this. It’d even better if SWE wrote ADRs (or whatever) that document all this stuff up front, but … well, there are only so many battles anyone can fight, right?
- h2odragon 3y ago
  Writing down and graphing out these relationships is a good way to identify and normalize them. I once had a system with layers of functionality; lvl 0 services were the most critical; lvl 3+ was "user shit" that could be sloughed off at need. Had some stub servers at lvl 0 and 1 that did things like providing a file share of the same name as the lower level services, but not populated; so that inadvertent domain crossing dependencies weren't severe problems. There was a "DB server" stub that only returned "no results." The actual DB server for those queries was on the monster big rack SPARC with the 3dz disks that took 10min to spin up fully. When it came up it took over.
charcircuit 3y ago
Turning off servers seems like the wrong call instead of transitioning servers into a lower powered state which can be exited once the power budget is available again.
- bluGill 3y ago
  The right answer is turn them all off - anything important is in a redundant data center. But odds are they don't have that. If a redundant data center isn't an option, then you should put more into ensuring the system is resilient - fireproof room (if a server catches on fire it can't spread to the next - there are a lot of considerations here that I don't know about that you need to figure out), plenty of backup power, redundant HVAC, redundant connections to the internet - and you should brainstorm other things that I didn't think of.
giaour 3y ago
Cloud environments have an elegant solution in the form of "spot" or "pre-emptible" instances: if your workload can tolerate interruptions because it's not time sensitive or not terribly important, you can get a pretty steep discount.
jll29 3y ago
I suspect making a list of "I think these ones are not critical." is not sufficient. You may overlook some subtle forms of interdependence. To be sure, you need to test your documentation by actually switching off the "uncritical" assets.
effnorwood 3y ago
[dead]
throwawaaarrgh 3y ago
If your machines are all hypervisors you could migrate important VMs to a couple hosts and turn off the rest. You could also possibly throttle the vcpus, which would slow down the VMs but allow you to run the machines cooler, or more VMs per machine. Finally the ones with long running jobs could just be snapshotted and powered down and restored later, resuming their computation. There's a reason us old fogies were so excited when virtual machines got increasingly robust. We could use them to solve problems quickly that used to be nearly impossible.
oasisbob 3y ago
Good plan! I think this is a relatively common practice within some corners of the telecom world. At university (Western WA in B'ham), I worked for our campus resnet, which had extensive involvement with other networking groups on campus. They ran layers 3 and below on the resnets, we took DNS+DHCP, plus egress, and everything through to layers 8 and 9. The core network gear was co-located in a few musty basements along with the telephone switches. DC and backup power was available, but severely limited under certain failure scenarios. All of the racked networking gear in the primary space was labeled with red and green dots. Green was first to go in any load-shedding scenario. Think: redundant LAN switches, switches carrying local ports, network monitoring servers, other +1 redundant components, etc. I'm not sure if the scheme was ever required in real life, but do know it was based on hard-earned experiences like the author here.
- red-iron-pine 3y ago
  Used to run data centers for ISPs and such around NoVA. This was built into the building plan by room, with most rooms going down first and Meet-Me-Rooms + the rooms immediately adjacent where the big iron routers were, being the last to fail. It's been a while but IIRC there weren't any specific by-rack or by system protocols.
_dan 3y ago
Similar thing (catastrophic aircon failure due to a flood in a crap colocated DC) happened to us too before we shifted to AWS. Photos from the colo were pretty bizarre - fans balanced on random boxes, makeshift aircon ducting made of cardboard and tape, and some dude flailing an open fire door back and forth all day to get a little bit of fresh air in. Bizarre to see in 2010-ish with multi million dollar customers. We ended up having to strategically shut servers down as well, but the question of what's critical, where is it in the racks, and what's next to it was incredibly difficult to answer. And kinda mind-bending - we'd been thinking of these things as completely virtualised resources for years, suddenly having to consider their physical characteristics as well was a bit of a shock. Just shutting down everything non-critical wasn't enough - there were still now critical non-redundant servers next to each other overheating. All we had to go on was an outdated racktables install, a readout of the case temperature for each, and a map of which machine was connected to which switch port which loosely related to position in the rack - none completely accurate. In the end we got the colo guys to send a photo of the rack front and back and (though not everything was well labelled) we were able to make some decisions and get things stable again. In the end one server that was critical but we couldn't get to run cooler we got lucky with - we were able to pull out the server below and (without shutting it down) have the on site engineer drop it down enough to crack the lid open and get some cool air into it to keep it running (albeit with no redundancy and on the edge of thermal shutdown). We came really close to a major outage that day that would have cost us dearly. I know it sounds like total shambles (and it kinda was) but I miss those days.
- macintux 3y ago
  I find it’s much less stressful to rescue situations where it wasn’t your fault to begin with. Absent the ability to point fingers at a vendor, crises like that are a miserable experience for me.
  - organsnyder 3y ago
    Being the hero always feels better than cleaning up your own messes.
VikingCoder 3y ago
I often think of the SNL sketch with the lines, "We should keep a list of our clients, and how much money they have given us." https://www.youtube.com/watch?v=reUjEFI0Slc https://www.youtube.com/watch?v=reUjEFI0Slc
mgaunard 3y ago
If machines aren't important, why do you have them at all? Because it's an university and we don't care about justifying costs?
- barrucadu 3y ago
  It's possible to have two important things and yet for one to be more important than the other.
- kiitos 3y ago
  Machines are incidental implementation details of the applications which run on them. The applications matter, the machines don't.
syslog 3y ago
Any server that must not fail is just not important enough. (Build your services with HA in mind, so you don‘t have to worry about a situation like this one.)
PaulKeeble 3y ago
One place I worked did a backup power test and when they came back from the diesels to the grid the entire datacentre lost power due to a software bug for about 10 seconds. It caused a massive outage. The problem was a lot of the machines pulled their OS image from central storage servers and there was no where near enough IO to load everything and they had to prioritise what to bring up first to lighten the load and stop everything thrashing. It was a complete nightmare even though the front end to take sales were well isolated from the backend. Working out what was most important across an entire corporation took as long as the problem resolving slowly by just bringing things up randomly. Nowadays you would just run multiple datacentres or cloud HA and we have SSDs but I just can't see such an architecture understanding being possible for any reasonably large company. The cost of keeping it and the dependencies up to date would be huge and it would always be out of date. More documentation isn't the solution, its to have multiple sites.
- jiggawatts 3y ago
  That brings back memories of a similar setup with hundreds of Windows servers booting over the network. We had regular “brownouts” even during the day just because the image streaming servers couldn’t handle the IOPS. Basic maintenance would slow down the servers for ten thousand users and generate support tickets. I jumped up and down and convinced management to buy one of the first enterprise SSDs on the market. It was a PCIe card form factor and cost five digits for a tiny amount of storage. We squeezed in the images using block-level deduplication and clever copy scripts that would run the compaction routine after each file was copied. The difference was staggering. Just two of those cards made hundreds of other servers run like greased lightning. Boot times dropped to single digit seconds instead of minutes. Maintenance changes could be done at any time with zero impact on users. The whole cluster could be rebooted all at once with only a slight slowdown. Fun times.
quickthrower2 3y ago
This is where the cloud kicks ass. Run multiple nodes with geo redundancy (where based on various concerns: cost, contracts, legal). But nodes should cross data centres. Maybe if one city gets nuked (literally or a fire/power outage) you still have uptime. Use Kubernetes maybe.
iwontberude 3y ago
Isn't this the point of decoupling your compute and datastores using CSI with disaggregated storage Kubernetes? So long as you keep your datastores available, whatever compute you can manage to attach it from Kubernetes can run whatever you truly need at capacities that you can handle with that level of hardware. Similarly, you could scale down the workloads on all the machines so they generated less heat without turning anything off at the expense of performance.
jabart 3y ago
Also important servers should also go lower in the rack, but not too low to flood. Learned that from an aircon failure and the top server hit 180F+ and shutdown. Thankfully had temp logs in Grafana to figure that out.
cogman10 3y ago
People like to rag on Kubernetes for it's complexity, but this is the exact sort of scenario where k8s really does shine. The answer to "which machines are important" is "only enough to provide the resources to run everything". You can kill whatever nodes you like just so long as there is enough of them to keep the cluster health and k8s will simply migrate workloads where they need to be. That being said, storage is still an issue. Perhaps NAS is the one place where you might mark "these are the important machines".
- rjbwork 3y ago
  And what if you only have 33% of your nominal cluster capacity available because the AC goes out in your server room? Now which containers should your cluster stop running? You haven't actually solved anything with this, you've just changed the abstraction layer at which you need to make decisions. Probably an improvement, but does not obviate some kind of load shedding heuristic.
  - zetsurin 3y ago
    You have: you've got clearly labelled workloads you can decide which ones are important and not, and easily turn down the ones that aren't important.
  - cogman10 3y ago
    But that's not what this was about. It was about "hey, which of these boxes had the ldap running on it? We need to make sure that gets shifted somewhere else!" K8S let's you say "ok, we don't have enough capacity to run everything so let's shut down the Bitcoin deployment to free up capacity". There's no leg work or bookkeeping to figure out what was running where, instead it's "what do I need to run and what can I shut down or tune down". All from the comfort of the room with AC. And if you're really cleaver, you went ahead and gave system critical pods elevated property. [1] [1] https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/ https://kubernetes.io/docs/concepts/scheduling-eviction/pod-...
    - rjbwork 3y ago

226 points by pabs3 3y ago

tlb 3y ago

Or you might want to have redundant cooling. Cooling system prices seem to scale fairly linearly with the cooling power above a few kW, so instead of one 100 kW system you could buy four 25 kW systems so a single failure won't be a disaster.

nine_k 3y ago
Won't 4 x 25 kW systems mean also 4x the installation cost?
- mdekkers 3y ago
  What the % of install cost over the projected lifetime of the system?
- dist-epoch 3y ago
  Probably less if you install them all at the same time.
- bluGill 3y ago
  Maybe, but costs are not linear and the nonlinear goes different ways for different parts of the install. The costs of the smaller systems installed could be cheaper than just the large system not installed if the smaller systems are standard parts.
throw0101a 3y ago
> Or you might want to have redundant cooling. Can you provide a cost centre or credit card for which they can bill this to? In case you didn't notice the domain, it is UToronto: academic departments aren't generally flush with cash. Further, you have to have physical space to fit the extra cooling equipment and pipes: not always easy or possible to do in old university buildings.
- wongarsu 3y ago
  If you designed the system like this from the start or when replacing it anyways, N+1 redundancy might not me much more expensive than one big cooling unit. The systems can mostly share their ductwork and just have redundancy in the active components, so mostly the chillers. Of course these systems only get replaced every couple decades, if ever, so they are pretty much stuck with the setup they have.
  - throw0101b 3y ago
    > If you designed […] University department IT is not designed, it grows over decades. At some point some benefactor may pay for a new building and the department will move, so that could be a chance to actually design. But modern architecture and architects don't really go well with hosting lots of servers in what is ostensibly office space. I've been involved in the build-out of buildings/office space on three occasions in my career, and trying to get a decent IT space pencilled has always been like pulling teeth.
  - bluGill 3y ago
    > Of course these systems only get replaced every couple decades, if ever This is despite the massive energy savings they could get if they replaced those older systems. Universities often are full of old buildings with terrible insulation heated/cooled by very old/inefficient systems. In 20 years they would be money ahead by tearing down most buildings on campus and rebuilding to modern standards - assuming energy costs don't go up which seems unlikely) But they consider all those old buildings historic and so won't.
    - light_hue_1 3y ago
      > In 20 years they would be money ahead by tearing down most buildings on campus and rebuilding to modern standards - assuming energy costs don't go up which seems unlikely) But they consider all those old buildings historic and so won't. It has nothing to do with considering those building historic. The problem is unless someone wants to donate a $50-100M, new buildings don't happen. And big donors want to donate to massive causes "Build a new building to cure cancer!" not "This building is kind of crappy, let's replace it with a better one". It doesn't matter that over 50 years something could be cheaper if there's no money to fix it now.
marcus0x62 3y ago
With that model, you’d probably want 5 instead of 4 (N+1), but the other thing to consider is if you can duct the cold air to where it needs to go when one or more of the units has failed.
dotancohen 3y ago
This will mean just about 4 times the number of failures, too. And can 75% cooling still cool the server room anyway?
- donkeyd 3y ago
  Maybe not, but some cooling means less servers to shut down.
- bluGill 3y ago
  It means 5 times the number of failures as you intentionally put in an extra unit so that one can be taken offline at any time for maintenance (which itself will keep the whole system more reliable), and if one fails the whole keeps up. The cost is only slightly more to do this when there are 5 smaller units. Those smaller units could be standard off the shelf units as well, so it could be cheaper than a large unit that isn't mad in as large a quantity - this is a consideration that needs to be made case by case) Even if you cheap out and only install 4 units, odds are your failure doesn't happen on the hottest day of the year and so 3 can keep up just fine. It is only when you are unlikely that you need to shut anything down.
  - dotancohen 3y ago
    > Even if you cheap out and only install 4 units, odds are your failure doesn't happen on the hottest day of the year Actually, historically many HVAC failures happen when the system is running at max capacity.
- evilduck 3y ago
  Four service degradations vs. one huge outage event. Pick your poison.

cowsandmilk 3y ago

It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.

throw0101a 3y ago
> Author: proudly declares himself Unix herder, wants to keep track of which systems are important. Because not all environments are webapps which dozens or hundreds of systems configured in a cookie cutter matter. Plenty of IT environments have pets because plenty of IT environments are not Web Scale™. And plenty of IT environments have staff churn and legacy systems where knowledge can become murky (see reference about "archaeology"). An IMAP server is different than a web server is different than an NFS server, and there may also be inter-dependencies between them.
- cookiemonster9 3y ago
  So much this. Not everything fits cattle/chicken/etc models. Even in cases where those models could fit, they are not necessarily the right choice, given staffing, expertise, budgets, and other factors.
- LeonB 3y ago
  I work with one system where the main entity is a “sale” which is processed beginning to end in some fraction of a second. A different system I work with, the main “entity” is, conceptually, more like a murder investigation. Less than 200 of the main entity are created in a year. Many different pieces of information are painstakingly gathered and tracked over a long period of time, with input from many people and oversight from legal experts and strong auditing requirements. Trying to apply the best lessons and principles from one system to the other is rarely a good idea. These kind of characteristics of different systems make a lot of difference to their care and feeding.
karmarepellent 3y ago
I would argue even the "wider industry" still administrates systems that must be treated as pets because they were not designed to be treated as cattle.

karmarepellent 3y ago

It's generally a good idea to have some documentation that states what a machine is used for, the "service", and how important said service is relative to others. At my company we kind of enforce this by not operating machines that are not explicitly assigned to any service. However you have to anticipate that the quality of documentation still varies immensely which might result in you shutting down a service that is actually more important than stated. Fortunately documentation improves after every outage because service owners reiterate on their part of the documentation when their service was shut down as "it appeared unimportant". It's a process.

spydum 3y ago

Asset management is definitely a thing. Tag your environments, tag your apps, and provide your apps criticality ratings based on how important they are to running the business. Then it's a matter of a query to know which servers can be shut, and which absolutely must remain.

ethbr1 3y ago
> provide your apps criticality ratings based on how important they are to running the business In a decentralized, self-service model, you can add "deal with convincing a stakeholder their app is anything less than most-critical." Although it usually works itself out if higher-criticality imposes ongoing time commitments on them as well (aka stick).
- j33zusjuice 3y ago
  That seems like a poorly run company. Idk. Maybe we’ve worked in very different environments, but devs have almost always been aware of the criticality of the app, so convincing people wasn’t hard. In most places, the answer hinges on “is it customer facing?” and/or “does it break a core part of our business?” If the answer is no to both, it’s not critical, and everyone understands that. There’s always some weird outlier, “well, this runs process B to report on process A, and sends a custom report to the CEO …”, but hopefully those exceptions are rare.
  - spydum 3y ago
    agree. ethbr1 is 100% right about this being a problem; if politics is driving your criticality rating, it's probably being done wrong. it should be as simple as your statement, being mindful of some of those downstream systems that aren't always obviously critical (until they are unavailable for $time) edit: whoops, maybe I read the meaning backward, but both issues exist!
  - hinkley 3y ago
    I maintain a couple of apps that are pretty much free to break or to revert back to an older build without much consequence, except for one day a week, when half the team uses them instead of just me. Any other day I can use them to test new base images, new coding or deployment techniques, etc. I just have to put things back by the end of our cycle.

marcus0x62 3y ago

Years ago during a week-long power outage, a telephone central office where we had some equipment suffered a generator failure. The telephone company had a backup plan (they kept one generator on a trailer in the city for such a contingency,) and they had battery capacity[0] for their critical equipment to last until the generator was hooked up. They did have to load shed, though: they just turned off the AC inverters. They figured anything critical in a central office was on DC power, and if you had something on AC, you were just going to have to wait until the backup-backup generator was installed. 0 - at the time, at least, CO battery backup was usually sized for 24 hours of runtime.

Scoundreller 3y ago
> had some equipment suffered a generator failure. The telephone company had a backup plan (they kept one generator on a trailer in the city for such a contingency Well that’s more forward thinking than AT&T’s Nashville CO when it got bombed. They just depended on natgas grid for their generators if grid electric went out. They underestimated the correlation between energy grids. When natgas got cutoff and the UPSs died, they had no ability to hook in roll-up generators and had to hastily install connection points (or hardline them). And no standby contracts for them.
- marcus0x62 3y ago
  To be fair, the CO I described[0] is a local switching center. The Nashville “CO” was the area long lines/tandem/long distance site. Now, I’m not defending the lack of a backup plan, but tandems tend to be far larger than a typical local/class 5 CO. 0 - which like the Nashville long lines building also happens to be a pre-divestiture AT&T site, although at the time it was owned by BellSouth. It is also in Tennessee, although in Memphis, not Nashville.

yevpats 3y ago

Check out CloudQuery - https://github.com/cloudquery/cloudquery https://github.com/cloudquery/cloudquery for an easy cloud asset inventory.

znlfl2 3y ago

[flagged]

Scubabear68 3y ago
I just did a due diligence with a company that only had servers in one data center. They were supremely confident that there was no way a whole DC could be impacted by an event.
- Retr0id 3y ago
  fwiw you're likely replying to someone's GPT bot

radiowave 3y ago

Tracking servers is one thing, but tracking the dependency relationships among them is likely at least as important.

j33zusjuice 3y ago
I’m really glad we realized that before disaster struck. We have a project in-progress to do exactly this. It’d even better if SWE wrote ADRs (or whatever) that document all this stuff up front, but … well, there are only so many battles anyone can fight, right?
h2odragon 3y ago
Writing down and graphing out these relationships is a good way to identify and normalize them. I once had a system with layers of functionality; lvl 0 services were the most critical; lvl 3+ was "user shit" that could be sloughed off at need. Had some stub servers at lvl 0 and 1 that did things like providing a file share of the same name as the lower level services, but not populated; so that inadvertent domain crossing dependencies weren't severe problems. There was a "DB server" stub that only returned "no results." The actual DB server for those queries was on the monster big rack SPARC with the 3dz disks that took 10min to spin up fully. When it came up it took over.

charcircuit 3y ago

Turning off servers seems like the wrong call instead of transitioning servers into a lower powered state which can be exited once the power budget is available again.

bluGill 3y ago
The right answer is turn them all off - anything important is in a redundant data center. But odds are they don't have that. If a redundant data center isn't an option, then you should put more into ensuring the system is resilient - fireproof room (if a server catches on fire it can't spread to the next - there are a lot of considerations here that I don't know about that you need to figure out), plenty of backup power, redundant HVAC, redundant connections to the internet - and you should brainstorm other things that I didn't think of.

giaour 3y ago

Cloud environments have an elegant solution in the form of "spot" or "pre-emptible" instances: if your workload can tolerate interruptions because it's not time sensitive or not terribly important, you can get a pretty steep discount.

jll29 3y ago

I suspect making a list of "I think these ones are not critical." is not sufficient. You may overlook some subtle forms of interdependence. To be sure, you need to test your documentation by actually switching off the "uncritical" assets.

effnorwood 3y ago

[dead]

throwawaaarrgh 3y ago

If your machines are all hypervisors you could migrate important VMs to a couple hosts and turn off the rest. You could also possibly throttle the vcpus, which would slow down the VMs but allow you to run the machines cooler, or more VMs per machine. Finally the ones with long running jobs could just be snapshotted and powered down and restored later, resuming their computation. There's a reason us old fogies were so excited when virtual machines got increasingly robust. We could use them to solve problems quickly that used to be nearly impossible.

oasisbob 3y ago

Good plan! I think this is a relatively common practice within some corners of the telecom world. At university (Western WA in B'ham), I worked for our campus resnet, which had extensive involvement with other networking groups on campus. They ran layers 3 and below on the resnets, we took DNS+DHCP, plus egress, and everything through to layers 8 and 9. The core network gear was co-located in a few musty basements along with the telephone switches. DC and backup power was available, but severely limited under certain failure scenarios. All of the racked networking gear in the primary space was labeled with red and green dots. Green was first to go in any load-shedding scenario. Think: redundant LAN switches, switches carrying local ports, network monitoring servers, other +1 redundant components, etc. I'm not sure if the scheme was ever required in real life, but do know it was based on hard-earned experiences like the author here.

red-iron-pine 3y ago
Used to run data centers for ISPs and such around NoVA. This was built into the building plan by room, with most rooms going down first and Meet-Me-Rooms + the rooms immediately adjacent where the big iron routers were, being the last to fail. It's been a while but IIRC there weren't any specific by-rack or by system protocols.

_dan 3y ago

Similar thing (catastrophic aircon failure due to a flood in a crap colocated DC) happened to us too before we shifted to AWS. Photos from the colo were pretty bizarre - fans balanced on random boxes, makeshift aircon ducting made of cardboard and tape, and some dude flailing an open fire door back and forth all day to get a little bit of fresh air in. Bizarre to see in 2010-ish with multi million dollar customers. We ended up having to strategically shut servers down as well, but the question of what's critical, where is it in the racks, and what's next to it was incredibly difficult to answer. And kinda mind-bending - we'd been thinking of these things as completely virtualised resources for years, suddenly having to consider their physical characteristics as well was a bit of a shock. Just shutting down everything non-critical wasn't enough - there were still now critical non-redundant servers next to each other overheating. All we had to go on was an outdated racktables install, a readout of the case temperature for each, and a map of which machine was connected to which switch port which loosely related to position in the rack - none completely accurate. In the end we got the colo guys to send a photo of the rack front and back and (though not everything was well labelled) we were able to make some decisions and get things stable again. In the end one server that was critical but we couldn't get to run cooler we got lucky with - we were able to pull out the server below and (without shutting it down) have the on site engineer drop it down enough to crack the lid open and get some cool air into it to keep it running (albeit with no redundancy and on the edge of thermal shutdown). We came really close to a major outage that day that would have cost us dearly. I know it sounds like total shambles (and it kinda was) but I miss those days.

macintux 3y ago
I find it’s much less stressful to rescue situations where it wasn’t your fault to begin with. Absent the ability to point fingers at a vendor, crises like that are a miserable experience for me.
- organsnyder 3y ago
  Being the hero always feels better than cleaning up your own messes.

VikingCoder 3y ago

I often think of the SNL sketch with the lines, "We should keep a list of our clients, and how much money they have given us." https://www.youtube.com/watch?v=reUjEFI0Slc https://www.youtube.com/watch?v=reUjEFI0Slc

mgaunard 3y ago

If machines aren't important, why do you have them at all? Because it's an university and we don't care about justifying costs?

barrucadu 3y ago
It's possible to have two important things and yet for one to be more important than the other.
kiitos 3y ago
Machines are incidental implementation details of the applications which run on them. The applications matter, the machines don't.

syslog 3y ago

Any server that must not fail is just not important enough. (Build your services with HA in mind, so you don‘t have to worry about a situation like this one.)

PaulKeeble 3y ago

One place I worked did a backup power test and when they came back from the diesels to the grid the entire datacentre lost power due to a software bug for about 10 seconds. It caused a massive outage. The problem was a lot of the machines pulled their OS image from central storage servers and there was no where near enough IO to load everything and they had to prioritise what to bring up first to lighten the load and stop everything thrashing. It was a complete nightmare even though the front end to take sales were well isolated from the backend. Working out what was most important across an entire corporation took as long as the problem resolving slowly by just bringing things up randomly. Nowadays you would just run multiple datacentres or cloud HA and we have SSDs but I just can't see such an architecture understanding being possible for any reasonably large company. The cost of keeping it and the dependencies up to date would be huge and it would always be out of date. More documentation isn't the solution, its to have multiple sites.

jiggawatts 3y ago
That brings back memories of a similar setup with hundreds of Windows servers booting over the network. We had regular “brownouts” even during the day just because the image streaming servers couldn’t handle the IOPS. Basic maintenance would slow down the servers for ten thousand users and generate support tickets. I jumped up and down and convinced management to buy one of the first enterprise SSDs on the market. It was a PCIe card form factor and cost five digits for a tiny amount of storage. We squeezed in the images using block-level deduplication and clever copy scripts that would run the compaction routine after each file was copied. The difference was staggering. Just two of those cards made hundreds of other servers run like greased lightning. Boot times dropped to single digit seconds instead of minutes. Maintenance changes could be done at any time with zero impact on users. The whole cluster could be rebooted all at once with only a slight slowdown. Fun times.

quickthrower2 3y ago

This is where the cloud kicks ass. Run multiple nodes with geo redundancy (where based on various concerns: cost, contracts, legal). But nodes should cross data centres. Maybe if one city gets nuked (literally or a fire/power outage) you still have uptime. Use Kubernetes maybe.

iwontberude 3y ago

Isn't this the point of decoupling your compute and datastores using CSI with disaggregated storage Kubernetes? So long as you keep your datastores available, whatever compute you can manage to attach it from Kubernetes can run whatever you truly need at capacities that you can handle with that level of hardware. Similarly, you could scale down the workloads on all the machines so they generated less heat without turning anything off at the expense of performance.

jabart 3y ago

Also important servers should also go lower in the rack, but not too low to flood. Learned that from an aircon failure and the top server hit 180F+ and shutdown. Thankfully had temp logs in Grafana to figure that out.

cogman10 3y ago

People like to rag on Kubernetes for it's complexity, but this is the exact sort of scenario where k8s really does shine. The answer to "which machines are important" is "only enough to provide the resources to run everything". You can kill whatever nodes you like just so long as there is enough of them to keep the cluster health and k8s will simply migrate workloads where they need to be. That being said, storage is still an issue. Perhaps NAS is the one place where you might mark "these are the important machines".

rjbwork 3y ago
And what if you only have 33% of your nominal cluster capacity available because the AC goes out in your server room? Now which containers should your cluster stop running? You haven't actually solved anything with this, you've just changed the abstraction layer at which you need to make decisions. Probably an improvement, but does not obviate some kind of load shedding heuristic.
- zetsurin 3y ago
  You have: you've got clearly labelled workloads you can decide which ones are important and not, and easily turn down the ones that aren't important.
- cogman10 3y ago
  But that's not what this was about. It was about "hey, which of these boxes had the ldap running on it? We need to make sure that gets shifted somewhere else!" K8S let's you say "ok, we don't have enough capacity to run everything so let's shut down the Bitcoin deployment to free up capacity". There's no leg work or bookkeeping to figure out what was running where, instead it's "what do I need to run and what can I shut down or tune down". All from the comfort of the room with AC. And if you're really cleaver, you went ahead and gave system critical pods elevated property. [1] [1] https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/ https://kubernetes.io/docs/concepts/scheduling-eviction/pod-...
  - rjbwork 3y ago