4 ms·

As an ex-FAANG engineer, this is FAANG advice. Pets are just fine. Most companies arent FAANG and don't need that class of solution. An R620 plugged into a swi

As an ex-FAANG engineer, this is FAANG advice. Pets are just fine. Most companies arent FAANG and don't need that class of solution. An R620 plugged into a switch in a colo, a bash script via cron, or a cloudflare worker are just fine for a lot of use cases. The only time it stops being fine is when you can't afford to do your pet -> cattle migration as you scale up. But I don't think this is a common death for companies. If you call "cattle" a cloudflare worker or lambda function - fine. But when we are talking about multiple redundant servers with load balancing across them, you really need to justify the cost of that vs the value you squeeze out. Sometimes you're squeezing the juice out of the rind.

mr_toad 4y ago
Treating servers as disposable is about more than just scale. It helps avoid creating snowflake servers, makes DR more predictable, and makes creating dev environments much easier.
- viraptor 4y ago
  Also, just knowing what you're running at the time is great. Changing things manually means you'll forget some of them when you need to do a bigger change. Or when you're seeing up server no.2. We've got great tooling these days, so whether you're going with nixos, docker, puppet, shell scripts for installation, or something else, the cattle approach gives you benefits.
- fragmede 4y ago
  I'd say that until you're FANG-level or at least hockey-sticking, it's actually totally okay for some of your things to be pets and that cattleizing literally everything is actually a premature optimization. API boxes which you have 100 of? Absolutely cattleize. That one bespoke server that runs that one service that is totally weird? Just let it be weird for a while. Work on more important things.
  - jon-wood 4y ago
    I think there’s degrees of cattleisation, I have in the past deployed crappy vendor software which needs a unique license key for each running instance. We discussed some sort of license service which would allow keys to be checked out when a container is started, but in the end settled on an auto scaling group with 1 instance for each running container with the key in an environment variable. That got us the comfort of knowing if the host machine died in the night the task would be rescheduled without a bunch of extra engineering to be able to scale arbitrarily when we only needed a couple of instances running.
    - fragmede 4y ago
      Is this on AWS where instances get reaped and can't be transparently migrated to a different physical machine like that can on GCP, or is my info out of date? (Just haven't been on AWS for my latest projects.) Because yeah, that would also keep me up at night if random instances will randomly get deleted. My proposal that pets are actually totally okay relies on that not being the case. My real, pointed question is: what was the opportunity cost of cattelizing that vs any of the other work you could have done in that time? Because given a fresh instance but I'm allowed to use the rest of my infra, setting that machine back up in the morning should just be to instantiating the right docker container on the host and copying the license file in. Depending on what the vendor software does, I probably still wouldn't ASG that until it had actually happened more often than once a month - there are so many more things to stress about! Even if this were the splunk license checker, losing some log files would be quite disappointing but not the absolute end of the world - if you're losing api calls to stripe or the equivalent that's a different matter!
nuker 4y ago
> Pets are just fine. They are not, if they are being configured by hand, using mouse or cli
- grosswait 4y ago
  I still say it depends on the scale of the business but if config as code is not in used, all the clicks and commands darn well better be documented
  - dijksterhuis 4y ago
    $PREVIOUS_COMPANY used to have pages and pages and pages of HOWTOs in Confluence re what to click on in the AWS console for the stuff that had to be done manually. Doubled up as amazing training / onboarding material for folks that maybe hadn't worked intimately with AWS before (including myself).
- r3trohack3r 4y ago
  There is a lot to unpack in this statement. First, this sentiment is already covered by the article under: > Git should be your only source of truth. Discard any local files or changes, what's not pushed into the repository, does not exist. Second, don't fool yourself into thinking, just because you wrote a script or configured some provisioning service, that means you've pulled yourself out of the valley of nondeterministic unreproducible production environments. Pets can and do live in git. A majority of the systems I've worked on in both FAANG and non-FAANG in my career couldn't be reproduced from the source repo in the case of a catastrophic disaster (I.e. us-east-1 disappears) or a year of being stable enough to not deploy even though their "source of truth" lives in git. The world moves out from under your repo and many of the steps you take for granted when writing your automation aren't understood by the next engineer running the automation. Then one day you have to go upgrade this one repo and realize it's IaC is no longer compatible with the infra in the real world and you burn a day trying to figure out what's wrong with a 1000LoC YAML file.
  - nuker 4y ago
    Not sure I got your point ... > Then one day you have to go upgrade this one repo and realize it's IaC is no longer compatible with the infra in the real world Thats yet another reason to avoid pets. When I say pets I mean relatively big pieces of SW, usually 3rd party, that need non-trivial configuration. Like Jenkins or Windows or some security gizmos.
dijksterhuis 4y ago
> But when we are talking about multiple redundant servers with load balancing across them, you really need to justify the cost of that vs the value you squeeze out. Sometimes you can justify using a thing for the wrong reasons. I recently attached 1x NLB to each of our Swarm clusters to migrate to automatically managed certificates directly attached to the NLB (Digital Ocean). $COMPANY has maybe ~3 users accessing each production application at a time. So the NLB itself is utterly pointless. But Engineering no longer have to fix the certificates each quarter after users see an insecure browser warning and email us about it. 100% worth it for $12 pcm (per swarm cluster).
throwawaaarrgh 4y ago
Pets are fine in the sense of "there's no way our servers would just disappear, and Larry the DevOps Guy who knows everything will never leave us..." Cattle is the best approach. Practice it and make it your default.
- astrange 4y ago
  Unfortunately most base OSes are designed around being pets - the FreeBSD manual is all about a world where you get a single machine, give it a cute name, install a web server, install a C compiler and the OS source, build a custom kernel for some reason… This may make sense if you can only afford a single machine you want to hyperoptimize but it’s not a good workflow for anyone. Not even BSD kernel developers.
  - toast0 4y ago
    All that is scriptable. I ran thousands of FreeBSD machines. When you get a new one, you run the script that sets up users, pulls the source, builds the custom kernel, installs the packages, etc. A little bit of automation goes a long way. (make world is a reasonable, although not particularly comprehensive, burn-in test too, but if you were in a hurry, you could build in one place and distribute)
    - astrange 4y ago
      You shouldn’t have a compiler on your webserver at all, and building a kernel should happen on a different machine than installing it. (And why isn’t the official kernel good enough?) Beyond the security surface that’s just asking to brick the machine; if you never make any mistakes it’s because you never tried anything very interesting with it in the first place. Furthermore, I claim that “you can write a script to edit /etc” is not how an OS should support configuration. Instead of being possible to keep a copy of the config in a script somewhere else, it should be impossible to do anything but that. So basically /etc and UNIX are bad ‘cause they’re not declarative.
      - 4y ago
oofnik 4y ago
No, pets are not fine. Just the other day I had to perform some maintenance on a long-running VM hosting some monitoring software. A backup VM is supposedly always running and ready to handle the workload in case of downtime. The switchover seemed to go fine, at first. Turns out, someone long ago had manually added a cron job to the primary server without adding it to the backup server, without documenting what it does, what permissions it needs, how it works, or why it's needed. This was only discovered after some manager in a different department complained that he stopped receiving the daily report to his e-mail inbox. If whoever deployed the report generation script took an extra hour to document what the script did, or even better, added it to VCS as part of the provisioning process for the server and re-deployed the server to ensure that the process works as expected, a day's worth of headache could have been averted.

mr_toad 4y ago

Treating servers as disposable is about more than just scale. It helps avoid creating snowflake servers, makes DR more predictable, and makes creating dev environments much easier.

viraptor 4y ago
Also, just knowing what you're running at the time is great. Changing things manually means you'll forget some of them when you need to do a bigger change. Or when you're seeing up server no.2. We've got great tooling these days, so whether you're going with nixos, docker, puppet, shell scripts for installation, or something else, the cattle approach gives you benefits.
fragmede 4y ago
I'd say that until you're FANG-level or at least hockey-sticking, it's actually totally okay for some of your things to be pets and that cattleizing literally everything is actually a premature optimization. API boxes which you have 100 of? Absolutely cattleize. That one bespoke server that runs that one service that is totally weird? Just let it be weird for a while. Work on more important things.
- jon-wood 4y ago
  I think there’s degrees of cattleisation, I have in the past deployed crappy vendor software which needs a unique license key for each running instance. We discussed some sort of license service which would allow keys to be checked out when a container is started, but in the end settled on an auto scaling group with 1 instance for each running container with the key in an environment variable. That got us the comfort of knowing if the host machine died in the night the task would be rescheduled without a bunch of extra engineering to be able to scale arbitrarily when we only needed a couple of instances running.
  - fragmede 4y ago
    Is this on AWS where instances get reaped and can't be transparently migrated to a different physical machine like that can on GCP, or is my info out of date? (Just haven't been on AWS for my latest projects.) Because yeah, that would also keep me up at night if random instances will randomly get deleted. My proposal that pets are actually totally okay relies on that not being the case. My real, pointed question is: what was the opportunity cost of cattelizing that vs any of the other work you could have done in that time? Because given a fresh instance but I'm allowed to use the rest of my infra, setting that machine back up in the morning should just be to instantiating the right docker container on the host and copying the license file in. Depending on what the vendor software does, I probably still wouldn't ASG that until it had actually happened more often than once a month - there are so many more things to stress about! Even if this were the splunk license checker, losing some log files would be quite disappointing but not the absolute end of the world - if you're losing api calls to stripe or the equivalent that's a different matter!

nuker 4y ago

> Pets are just fine. They are not, if they are being configured by hand, using mouse or cli

grosswait 4y ago
I still say it depends on the scale of the business but if config as code is not in used, all the clicks and commands darn well better be documented
- dijksterhuis 4y ago
  $PREVIOUS_COMPANY used to have pages and pages and pages of HOWTOs in Confluence re what to click on in the AWS console for the stuff that had to be done manually. Doubled up as amazing training / onboarding material for folks that maybe hadn't worked intimately with AWS before (including myself).
r3trohack3r 4y ago
There is a lot to unpack in this statement. First, this sentiment is already covered by the article under: > Git should be your only source of truth. Discard any local files or changes, what's not pushed into the repository, does not exist. Second, don't fool yourself into thinking, just because you wrote a script or configured some provisioning service, that means you've pulled yourself out of the valley of nondeterministic unreproducible production environments. Pets can and do live in git. A majority of the systems I've worked on in both FAANG and non-FAANG in my career couldn't be reproduced from the source repo in the case of a catastrophic disaster (I.e. us-east-1 disappears) or a year of being stable enough to not deploy even though their "source of truth" lives in git. The world moves out from under your repo and many of the steps you take for granted when writing your automation aren't understood by the next engineer running the automation. Then one day you have to go upgrade this one repo and realize it's IaC is no longer compatible with the infra in the real world and you burn a day trying to figure out what's wrong with a 1000LoC YAML file.
- nuker 4y ago
  Not sure I got your point ... > Then one day you have to go upgrade this one repo and realize it's IaC is no longer compatible with the infra in the real world Thats yet another reason to avoid pets. When I say pets I mean relatively big pieces of SW, usually 3rd party, that need non-trivial configuration. Like Jenkins or Windows or some security gizmos.

dijksterhuis 4y ago

> But when we are talking about multiple redundant servers with load balancing across them, you really need to justify the cost of that vs the value you squeeze out. Sometimes you can justify using a thing for the wrong reasons. I recently attached 1x NLB to each of our Swarm clusters to migrate to automatically managed certificates directly attached to the NLB (Digital Ocean). $COMPANY has maybe ~3 users accessing each production application at a time. So the NLB itself is utterly pointless. But Engineering no longer have to fix the certificates each quarter after users see an insecure browser warning and email us about it. 100% worth it for $12 pcm (per swarm cluster).

throwawaaarrgh 4y ago

Pets are fine in the sense of "there's no way our servers would just disappear, and Larry the DevOps Guy who knows everything will never leave us..." Cattle is the best approach. Practice it and make it your default.

astrange 4y ago
Unfortunately most base OSes are designed around being pets - the FreeBSD manual is all about a world where you get a single machine, give it a cute name, install a web server, install a C compiler and the OS source, build a custom kernel for some reason… This may make sense if you can only afford a single machine you want to hyperoptimize but it’s not a good workflow for anyone. Not even BSD kernel developers.
- toast0 4y ago
  All that is scriptable. I ran thousands of FreeBSD machines. When you get a new one, you run the script that sets up users, pulls the source, builds the custom kernel, installs the packages, etc. A little bit of automation goes a long way. (make world is a reasonable, although not particularly comprehensive, burn-in test too, but if you were in a hurry, you could build in one place and distribute)
  - astrange 4y ago
    You shouldn’t have a compiler on your webserver at all, and building a kernel should happen on a different machine than installing it. (And why isn’t the official kernel good enough?) Beyond the security surface that’s just asking to brick the machine; if you never make any mistakes it’s because you never tried anything very interesting with it in the first place. Furthermore, I claim that “you can write a script to edit /etc” is not how an OS should support configuration. Instead of being possible to keep a copy of the config in a script somewhere else, it should be impossible to do anything but that. So basically /etc and UNIX are bad ‘cause they’re not declarative.
    - 4y ago

oofnik 4y ago

No, pets are not fine. Just the other day I had to perform some maintenance on a long-running VM hosting some monitoring software. A backup VM is supposedly always running and ready to handle the workload in case of downtime. The switchover seemed to go fine, at first. Turns out, someone long ago had manually added a cron job to the primary server without adding it to the backup server, without documenting what it does, what permissions it needs, how it works, or why it's needed. This was only discovered after some manager in a different department complained that he stopped receiving the daily report to his e-mail inbox. If whoever deployed the report generation script took an extra hour to document what the script did, or even better, added it to VCS as part of the provisioning process for the server and re-deployed the server to ensure that the process works as expected, a day's worth of headache could have been averted.