6 ms·

It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herd

It is interesting to contrast with where the wider industry has gone. Industry: don’t treat your systems like pets. Author: proudly declares himself Unix herder, wants to keep track of which systems are important.

throw0101a 3y ago
> Author: proudly declares himself Unix herder, wants to keep track of which systems are important. Because not all environments are webapps which dozens or hundreds of systems configured in a cookie cutter matter. Plenty of IT environments have pets because plenty of IT environments are not Web Scale™. And plenty of IT environments have staff churn and legacy systems where knowledge can become murky (see reference about "archaeology"). An IMAP server is different than a web server is different than an NFS server, and there may also be inter-dependencies between them.
- cookiemonster9 3y ago
  So much this. Not everything fits cattle/chicken/etc models. Even in cases where those models could fit, they are not necessarily the right choice, given staffing, expertise, budgets, and other factors.
- LeonB 3y ago
  I work with one system where the main entity is a “sale” which is processed beginning to end in some fraction of a second. A different system I work with, the main “entity” is, conceptually, more like a murder investigation. Less than 200 of the main entity are created in a year. Many different pieces of information are painstakingly gathered and tracked over a long period of time, with input from many people and oversight from legal experts and strong auditing requirements. Trying to apply the best lessons and principles from one system to the other is rarely a good idea. These kind of characteristics of different systems make a lot of difference to their care and feeding.
karmarepellent 3y ago
I would argue even the "wider industry" still administrates systems that must be treated as pets because they were not designed to be treated as cattle.
luma 3y ago
When you are responsible for the full infrastructure, sequencing power down and power on in coordination with your UPS is a common solution. Network gear needs a few minutes to light up ports, core services like DNS and identity services might need to light up next, then storage, then hypervisors and container hosts, then you can actually start working on app dependencies. This sort of sequencing leads itself naturally to having a plan for limited capacity “keep the lights on” workload shedding when facing a situation like the OP. Not everyone has elected to pay Bezos double the price for things they can handle themselves, and this is part of handling it.
- b112 3y ago
  Double? Try 100x!!
  - marcosdumay 3y ago
    Exactly. If it was double, it would be a non-brainier.
  - philsnow 3y ago
    If you’re running a couple ec2 instances in one AZ then yeah it’s closer to 100x, but if you wanted to replicate the durability of S3, it would cost you a lot in terms of redundancy (usually “invisible” to the customer) and ongoing R&D and support headcount. Yes, even when you add it all up, Amazon still charges a premium even over that all-in cost. That’s sweat equity.
p_l 3y ago
Even if you're running "cattle", you still need to keep track of which systems are important, because to surprise of many, the full infrastructure is more like the ranch, and cattle is just part of it. (and here I remind myself again to write the screed against "cattle" metaphor...)
bayindirh 3y ago
HPC admin here (and possibly managing a similar system topology with their room). In heterogeneous system rooms, you can't stuff everything into a virtualization cluster with a shared storage and migrate things on the fly, thinking that every server (in hardware) is a cattle and you can just herd your VMs from host to host. A SLURM cluster is easy. Shutdown all the nodes, controller will say "welp, no servers to run the workloads, will wait until servers come back", but storage systems are not that easy (ordering, controller dependencies, volume dependencies, service dependencies, etc.). Also there are servers which can't be virtualized because they're hardware dependent, latency dependent, or just filling the server they are in, resource wise. We also have some pet servers, and some cattle. We "pfft" to some servers and scramble for others due to various reasons. We know what server runs which service by the hostname, and never install pet servers without team's knowledge. So if something important goes down everyone at least can attend the OS or the hardware it's running on. Even in a cloud environment, you can't move a VSwitch VM as you want, because you can't have the root of a fat SDN tree on every node. Even the most flexible infrastructure has firm parts to support that flexibility. It's impossible otherwise. Lastly, not knowing which servers are important is a big no-no. We had "glycol everywhere" incidents and serious heatwaves, and all we say is, "we can't cool room down, scale down". Everybody shuts the servers they know they can, even if somebody from the team is on vacation. Being a sysadmin is a team game.
readscore 3y ago
At a FAANG, our services are cattle, but we still plan which services to keep running when we need to drain 50% of a DC. Latency is important. Money makers > Latency sensitive > Optional requests > Background requests > Batch traffic. Bootstrapping is important. If A depends on B, you might to drain A first, or A and B together.
- _heimdall 3y ago
  The cattle metaphors really is a bad one. Anyone raising cattle should do the same thing, knowing which animals are the priority in case of draught, disease, etc. Hopefully one never has to face that scenario, but its much easier to pick up the pieces when you know where the priorities are whether you're having to power down servers or thin a herd.
  - bluGill 3y ago
    Cattle are often interchangeable. You cull any that catch a disease (in some cases the USDA will cull the entire herd if just one catches something - bio security is a big deal) In the case of drought you pick a bunch to get rid of - based on market prices (If everyone else is you will try to keep yours because the market is collapsing - but this means managing feed and thus may mean culling more of the herd latter.) Some cattle we can measure. Milk cows are carefully managed as to output - the farmer knows how much the milk each one is worth and so they can cull the low producers. However milk is so much more valuable than meat that they never cull based on drought - milk can always outbid meat for feed. If milk demand goes down the farmer might cull come - but often the farmer is under contract for X amount of milk and so they cannot manage prices.
    - p_l 3y ago
      Honestly, big issue with the cattle metaphor is that the individual services you run on servers are very much often not interchangeable. A DNS service is not NTP is not mail gateway is not application load balancer is not database etc etc etc At best, multiple replicas of those are cattle. And while you can treat the servers underlying them as interchangeable, that doesn't change the fact the services you run on them are not.
pch00 3y ago
> Industry: don’t treat your systems like pets. The industry has this narrative because it suits their desire to sell higher-margined cloud services. However in the real world, especially in academia as cks is, the reality is that many workloads are still not suitable for the cloud.
tonyarkles 3y ago
I’m absolutely loving the term Unix herder and will probably adopt it :) I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here: - budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the hardware they’ve bought you. They won’t be as happy to dramatically increase your opex. - storage is the ultimate pet. In a university IT department you’re going to have people who need access to tons and tons of speedy storage both short-term and long-term. I’m smiling a little bit thinking about a job 10 years ago who adopted the cattle-not-pets mentality. The IT department decided they were done with their pets, moved everything to a big vSphere cluster, and backed it by a giant RAID-5 array. There was a disk failure, but that’s ok, RAID-5 can handle that. And then the next day there was a second disk failure. Boom. Every single VM in the engineering department is gone including all of the data. It was all backed up to tape and slowly got restored but the blast radius was enormous.
- throw0101b 3y ago
  > You have a capex budget […] As someone who has worked IT in academia: no, you do not. :)
  - tonyarkles 3y ago
    I'm laughing (and crying) with you, not at you. From my past life in academia, you're totally right. But that kind of reinforces the point... you do occasionally get some budget for servers and then have to make them last as long as possible :). Those one-time expenses are generally more palatable than recurring cloud storage costs though.
- jodrellblank 3y ago
  At the risk of no true Scotsman, that doesn’t sound like “cattle not pets“; when the cattle are sent to the slaughterhouse there isn’t any blast radius, there’s just more cattle taking over. You explicitly don’t have to replace them with exact clones of the original cattle from tape very slowly, you spin up a herd of more cattle in moments.
viraptor 3y ago
I think you're missing some aspects of cattle. You're still supposed to keep track of what happens and where. You still want to understand why and how each of the servers in the autoscaling group (or similar) behaves. The cattle part just means they're unified and quickly replaceable. Buy they still need to be well tagged, accounted for in planning, removed when they don't fulfil the purpose anymore, identified for billing, etc. And also importantly: you want to make sure you have a good enough description for them that you can say "terraform/cloudformation/ansible: make sure those are running" - without having to find them on the list and do it manually.
falcor84 3y ago
Where's the contrast? Herding is something you do with cattle rather than pets.
_heimdall 3y ago
I'm pretty sure anyone in the industry that draws this distinction between cattle and pets has never worked with cattle and only knows of general ideas about the industrial cattle business.
- bluGill 3y ago
  Ranchers do eat their pets. They generally do love the cattle, but they also know at the end of a few years they get replaced - it is the cycle of life.
- antod 3y ago
  Likewise, anyone talking about civil engineering and bridges. At least back in the day Slashdot was fully aware of how broken their ubiquitous car analogies were and played it up.
- jodrellblank 3y ago
  https://largest.org/geography/largest-cattle-ranches-in-the-us/ https://largest.org/geography/largest-cattle-ranches-in-the-... says Deseret Ranches has 44,000 cattle. Google tells me beef cattle are slaughtered after 2 years. Split 2 years among 44,000 cattle and you get to spend at most 24 minutes with each one, if you dedicate 2 years of your life to nothing else but that, not even sleep, travel, eating. If you let them live their natural life expectancy of 20 years, you get 240 minutes with each cow - two hours in its 20 year life. "I care about my cattle", yes, I don't think "cattle" is supposed to mean "stop caring about the things you work on". "I know each and every one as well as the family dog I've had for ten years", no. That's not possible. You raise them industrially and kill them for profit/food, that's a different dynamic than with Spot.
  - _heimdall 3y ago
    I believe this is just a great example of why cattle shouldn't be raised in such high volume industrial processes. Have you ever been around cattle? Or helped them calve? Or slaughtered one for meat? I know every one of my animals and understand the herd dynamics, from who the lead cow is to who is the asshole that is the one often starting fights and annoying the others. We shouldn't be throwing so many animals into such a controlled and confined system that they are reduced to numbers on a spreadsheet. We shouldn't raise an animal for slaughter after dedicating at most 24 minutes to them.
krisoft 3y ago
I don't see the contrast here. > proudly declares himself Unix herder You know what herder's herd? Cattle. Not pets. > wants to keep track of which systems are important. I mean obviously? Industry does the same. Probably with automation, tools and tagging during provisioning. The pet mentality is when you create a beautiful handcrafted spreadsheet showing which services run on the server named "Mjolnir" and which services run on the server named "Valinor". The cattle mentality is when you have the same in a distributed key-value database with UUIDs instead of fancyful server names. Or the pet mentality is when you prefer to not shut down "Mjolnir" because it has more than 2 years of uptime or some other silly reason like that. (as opposed to not shutting it down because you know that you would loose more money that way than by risking it overheating and having to buy a new one.)
johann8384 3y ago
Pets make sense sometimes. I also think there are still plenty of companies, large ones, with important services and data, that just don't operate in a way that allows the data center teams to do this either. I have some experience with both health insurance and life insurance companies for example where "this critical thing #8 that we would go out of business without" stillnloves solely on "this server right here". In university settings you have systems that are owned by a wide array of teams. These organizations aren't ready or even looking to implement a platform model where the underlying hardware can be generic.
mrighele 3y ago
The issue here is not much the hardware, but the services that run on top of them. I guess that many companies that use "current practices" have plenty of services that they don't even know about running on their clusters. The main difference is that instead of the kind of issues that the link talks about, you have those services running year after year, using resources, for the joy of the cloud companies. This happens even at Google [1]: "There are several remarkable aspects to this story. One is that running a Bigtable was so inconsequential to Google’s scale that it took 2 years before anyone even noticed it, and even then, only because the version was old. " [1] https://steve-yegge.medium.com/dear-google-cloud-your-deprecation-policy-is-killing-you-ee7525dc05dc https://steve-yegge.medium.com/dear-google-cloud-your-deprec...
NovemberWhiskey 3y ago
Not sure if the goal was just to make an amusing comparison, but these are actually two completely different concerns. Building your systems so that they don't depend on permanent infrastructure and snowflake configurations is an orthogonal concern from understanding how to shed load in a business-continuity crisis.

throw0101a 3y ago

> Author: proudly declares himself Unix herder, wants to keep track of which systems are important. Because not all environments are webapps which dozens or hundreds of systems configured in a cookie cutter matter. Plenty of IT environments have pets because plenty of IT environments are not Web Scale™. And plenty of IT environments have staff churn and legacy systems where knowledge can become murky (see reference about "archaeology"). An IMAP server is different than a web server is different than an NFS server, and there may also be inter-dependencies between them.

cookiemonster9 3y ago
So much this. Not everything fits cattle/chicken/etc models. Even in cases where those models could fit, they are not necessarily the right choice, given staffing, expertise, budgets, and other factors.
LeonB 3y ago
I work with one system where the main entity is a “sale” which is processed beginning to end in some fraction of a second. A different system I work with, the main “entity” is, conceptually, more like a murder investigation. Less than 200 of the main entity are created in a year. Many different pieces of information are painstakingly gathered and tracked over a long period of time, with input from many people and oversight from legal experts and strong auditing requirements. Trying to apply the best lessons and principles from one system to the other is rarely a good idea. These kind of characteristics of different systems make a lot of difference to their care and feeding.

karmarepellent 3y ago

I would argue even the "wider industry" still administrates systems that must be treated as pets because they were not designed to be treated as cattle.

luma 3y ago

When you are responsible for the full infrastructure, sequencing power down and power on in coordination with your UPS is a common solution. Network gear needs a few minutes to light up ports, core services like DNS and identity services might need to light up next, then storage, then hypervisors and container hosts, then you can actually start working on app dependencies. This sort of sequencing leads itself naturally to having a plan for limited capacity “keep the lights on” workload shedding when facing a situation like the OP. Not everyone has elected to pay Bezos double the price for things they can handle themselves, and this is part of handling it.

b112 3y ago
Double? Try 100x!!
- marcosdumay 3y ago
  Exactly. If it was double, it would be a non-brainier.
- philsnow 3y ago
  If you’re running a couple ec2 instances in one AZ then yeah it’s closer to 100x, but if you wanted to replicate the durability of S3, it would cost you a lot in terms of redundancy (usually “invisible” to the customer) and ongoing R&D and support headcount. Yes, even when you add it all up, Amazon still charges a premium even over that all-in cost. That’s sweat equity.

p_l 3y ago

Even if you're running "cattle", you still need to keep track of which systems are important, because to surprise of many, the full infrastructure is more like the ranch, and cattle is just part of it. (and here I remind myself again to write the screed against "cattle" metaphor...)

bayindirh 3y ago

HPC admin here (and possibly managing a similar system topology with their room). In heterogeneous system rooms, you can't stuff everything into a virtualization cluster with a shared storage and migrate things on the fly, thinking that every server (in hardware) is a cattle and you can just herd your VMs from host to host. A SLURM cluster is easy. Shutdown all the nodes, controller will say "welp, no servers to run the workloads, will wait until servers come back", but storage systems are not that easy (ordering, controller dependencies, volume dependencies, service dependencies, etc.). Also there are servers which can't be virtualized because they're hardware dependent, latency dependent, or just filling the server they are in, resource wise. We also have some pet servers, and some cattle. We "pfft" to some servers and scramble for others due to various reasons. We know what server runs which service by the hostname, and never install pet servers without team's knowledge. So if something important goes down everyone at least can attend the OS or the hardware it's running on. Even in a cloud environment, you can't move a VSwitch VM as you want, because you can't have the root of a fat SDN tree on every node. Even the most flexible infrastructure has firm parts to support that flexibility. It's impossible otherwise. Lastly, not knowing which servers are important is a big no-no. We had "glycol everywhere" incidents and serious heatwaves, and all we say is, "we can't cool room down, scale down". Everybody shuts the servers they know they can, even if somebody from the team is on vacation. Being a sysadmin is a team game.

readscore 3y ago

At a FAANG, our services are cattle, but we still plan which services to keep running when we need to drain 50% of a DC. Latency is important. Money makers > Latency sensitive > Optional requests > Background requests > Batch traffic. Bootstrapping is important. If A depends on B, you might to drain A first, or A and B together.

_heimdall 3y ago
The cattle metaphors really is a bad one. Anyone raising cattle should do the same thing, knowing which animals are the priority in case of draught, disease, etc. Hopefully one never has to face that scenario, but its much easier to pick up the pieces when you know where the priorities are whether you're having to power down servers or thin a herd.
- bluGill 3y ago
  Cattle are often interchangeable. You cull any that catch a disease (in some cases the USDA will cull the entire herd if just one catches something - bio security is a big deal) In the case of drought you pick a bunch to get rid of - based on market prices (If everyone else is you will try to keep yours because the market is collapsing - but this means managing feed and thus may mean culling more of the herd latter.) Some cattle we can measure. Milk cows are carefully managed as to output - the farmer knows how much the milk each one is worth and so they can cull the low producers. However milk is so much more valuable than meat that they never cull based on drought - milk can always outbid meat for feed. If milk demand goes down the farmer might cull come - but often the farmer is under contract for X amount of milk and so they cannot manage prices.
  - p_l 3y ago
    Honestly, big issue with the cattle metaphor is that the individual services you run on servers are very much often not interchangeable. A DNS service is not NTP is not mail gateway is not application load balancer is not database etc etc etc At best, multiple replicas of those are cattle. And while you can treat the servers underlying them as interchangeable, that doesn't change the fact the services you run on them are not.

pch00 3y ago

> Industry: don’t treat your systems like pets. The industry has this narrative because it suits their desire to sell higher-margined cloud services. However in the real world, especially in academia as cks is, the reality is that many workloads are still not suitable for the cloud.

tonyarkles 3y ago

I’m absolutely loving the term Unix herder and will probably adopt it :) I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here: - budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the hardware they’ve bought you. They won’t be as happy to dramatically increase your opex. - storage is the ultimate pet. In a university IT department you’re going to have people who need access to tons and tons of speedy storage both short-term and long-term. I’m smiling a little bit thinking about a job 10 years ago who adopted the cattle-not-pets mentality. The IT department decided they were done with their pets, moved everything to a big vSphere cluster, and backed it by a giant RAID-5 array. There was a disk failure, but that’s ok, RAID-5 can handle that. And then the next day there was a second disk failure. Boom. Every single VM in the engineering department is gone including all of the data. It was all backed up to tape and slowly got restored but the blast radius was enormous.

throw0101b 3y ago
> You have a capex budget […] As someone who has worked IT in academia: no, you do not. :)
- tonyarkles 3y ago
  I'm laughing (and crying) with you, not at you. From my past life in academia, you're totally right. But that kind of reinforces the point... you do occasionally get some budget for servers and then have to make them last as long as possible :). Those one-time expenses are generally more palatable than recurring cloud storage costs though.
jodrellblank 3y ago
At the risk of no true Scotsman, that doesn’t sound like “cattle not pets“; when the cattle are sent to the slaughterhouse there isn’t any blast radius, there’s just more cattle taking over. You explicitly don’t have to replace them with exact clones of the original cattle from tape very slowly, you spin up a herd of more cattle in moments.

viraptor 3y ago

I think you're missing some aspects of cattle. You're still supposed to keep track of what happens and where. You still want to understand why and how each of the servers in the autoscaling group (or similar) behaves. The cattle part just means they're unified and quickly replaceable. Buy they still need to be well tagged, accounted for in planning, removed when they don't fulfil the purpose anymore, identified for billing, etc. And also importantly: you want to make sure you have a good enough description for them that you can say "terraform/cloudformation/ansible: make sure those are running" - without having to find them on the list and do it manually.

falcor84 3y ago

Where's the contrast? Herding is something you do with cattle rather than pets.

_heimdall 3y ago

I'm pretty sure anyone in the industry that draws this distinction between cattle and pets has never worked with cattle and only knows of general ideas about the industrial cattle business.

bluGill 3y ago
Ranchers do eat their pets. They generally do love the cattle, but they also know at the end of a few years they get replaced - it is the cycle of life.
antod 3y ago
Likewise, anyone talking about civil engineering and bridges. At least back in the day Slashdot was fully aware of how broken their ubiquitous car analogies were and played it up.
jodrellblank 3y ago
https://largest.org/geography/largest-cattle-ranches-in-the-us/ https://largest.org/geography/largest-cattle-ranches-in-the-... says Deseret Ranches has 44,000 cattle. Google tells me beef cattle are slaughtered after 2 years. Split 2 years among 44,000 cattle and you get to spend at most 24 minutes with each one, if you dedicate 2 years of your life to nothing else but that, not even sleep, travel, eating. If you let them live their natural life expectancy of 20 years, you get 240 minutes with each cow - two hours in its 20 year life. "I care about my cattle", yes, I don't think "cattle" is supposed to mean "stop caring about the things you work on". "I know each and every one as well as the family dog I've had for ten years", no. That's not possible. You raise them industrially and kill them for profit/food, that's a different dynamic than with Spot.
- _heimdall 3y ago
  I believe this is just a great example of why cattle shouldn't be raised in such high volume industrial processes. Have you ever been around cattle? Or helped them calve? Or slaughtered one for meat? I know every one of my animals and understand the herd dynamics, from who the lead cow is to who is the asshole that is the one often starting fights and annoying the others. We shouldn't be throwing so many animals into such a controlled and confined system that they are reduced to numbers on a spreadsheet. We shouldn't raise an animal for slaughter after dedicating at most 24 minutes to them.

krisoft 3y ago

I don't see the contrast here. > proudly declares himself Unix herder You know what herder's herd? Cattle. Not pets. > wants to keep track of which systems are important. I mean obviously? Industry does the same. Probably with automation, tools and tagging during provisioning. The pet mentality is when you create a beautiful handcrafted spreadsheet showing which services run on the server named "Mjolnir" and which services run on the server named "Valinor". The cattle mentality is when you have the same in a distributed key-value database with UUIDs instead of fancyful server names. Or the pet mentality is when you prefer to not shut down "Mjolnir" because it has more than 2 years of uptime or some other silly reason like that. (as opposed to not shutting it down because you know that you would loose more money that way than by risking it overheating and having to buy a new one.)

johann8384 3y ago

Pets make sense sometimes. I also think there are still plenty of companies, large ones, with important services and data, that just don't operate in a way that allows the data center teams to do this either. I have some experience with both health insurance and life insurance companies for example where "this critical thing #8 that we would go out of business without" stillnloves solely on "this server right here". In university settings you have systems that are owned by a wide array of teams. These organizations aren't ready or even looking to implement a platform model where the underlying hardware can be generic.

mrighele 3y ago

The issue here is not much the hardware, but the services that run on top of them. I guess that many companies that use "current practices" have plenty of services that they don't even know about running on their clusters. The main difference is that instead of the kind of issues that the link talks about, you have those services running year after year, using resources, for the joy of the cloud companies. This happens even at Google [1]: "There are several remarkable aspects to this story. One is that running a Bigtable was so inconsequential to Google’s scale that it took 2 years before anyone even noticed it, and even then, only because the version was old. " [1] https://steve-yegge.medium.com/dear-google-cloud-your-deprecation-policy-is-killing-you-ee7525dc05dc https://steve-yegge.medium.com/dear-google-cloud-your-deprec...

NovemberWhiskey 3y ago

Not sure if the goal was just to make an amusing comparison, but these are actually two completely different concerns. Building your systems so that they don't depend on permanent infrastructure and snowflake configurations is an orthogonal concern from understanding how to shed load in a business-continuity crisis.