5 ms·

I’m absolutely loving the term Unix herder and will probably adopt it :) I’m generally with you and the wider industry on the cattle-not-pets thing but there a

I’m absolutely loving the term Unix herder and will probably adopt it :) I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here: - budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the hardware they’ve bought you. They won’t be as happy to dramatically increase your opex. - storage is the ultimate pet. In a university IT department you’re going to have people who need access to tons and tons of speedy storage both short-term and long-term. I’m smiling a little bit thinking about a job 10 years ago who adopted the cattle-not-pets mentality. The IT department decided they were done with their pets, moved everything to a big vSphere cluster, and backed it by a giant RAID-5 array. There was a disk failure, but that’s ok, RAID-5 can handle that. And then the next day there was a second disk failure. Boom. Every single VM in the engineering department is gone including all of the data. It was all backed up to tape and slowly got restored but the blast radius was enormous.

throw0101b 3y ago
> You have a capex budget […] As someone who has worked IT in academia: no, you do not. :)
- tonyarkles 3y ago
  I'm laughing (and crying) with you, not at you. From my past life in academia, you're totally right. But that kind of reinforces the point... you do occasionally get some budget for servers and then have to make them last as long as possible :). Those one-time expenses are generally more palatable than recurring cloud storage costs though.
jodrellblank 3y ago
At the risk of no true Scotsman, that doesn’t sound like “cattle not pets“; when the cattle are sent to the slaughterhouse there isn’t any blast radius, there’s just more cattle taking over. You explicitly don’t have to replace them with exact clones of the original cattle from tape very slowly, you spin up a herd of more cattle in moments.
- logifail 3y ago
  > when the cattle are sent to the slaughterhouse Data isn't "sent to the slaughterhouse". Ever. Data can be annoying that way.
  - datadrivenangel 3y ago
    The true problem with kubernetes and modern cloud. Not insurmountable, but painful when your data is large compared to your processing needs.
  - j33zusjuice 3y ago
    I think the point is that systems that aren’t replaced easily shouldn’t be managed like they are, not … whatever it is you’re getting at here.
    - carbotaniuman 3y ago
      Sadly you can't just spool down data either - data isn't fungible!
- chris_wot 3y ago
  Analogies always break down under scrutiny. Any cattle farmer would find “spinning up a herd of cattle” to be hilarious.
- tonyarkles 3y ago
  > you spin up a herd of more cattle in moments Where does the decade of data they've been collecting come from when you "spin up a new herd"?
  - ElevenLathe 3y ago
    It's stored in another system that is probably treated as a pet, hopefully by somebody way better at it with a huge staff (like AWS). Even if it's a local NetApp cluster or something, you can leave the state to the storage admins rather than random engineers that may or may not even be with the company any more.
jsjohnst 3y ago
> moved everything to a big vSphere cluster, and backed it by a giant RAID-5 array I’m with sibling commenter, if said IT department genuinely thought that the core point in “cattle-not-pets” was met by their single SuperMegaCow, then they missed the point entirely.
verticalscaler 3y ago
The devops meme referenced, "cattle not pets", probably popularized by a book called "The Phoenix Project". The real point is that pets are job security for the Unix herder. If you end up with one neckbeard running the joint that's even worse as a single point of failure.
- datadrivenangel 3y ago
  If you end up with one cloud guru running your cloud, that's maybe worse as a single point of failure. AWS system admins may be more fungible these days than unix sysadmins. Maybe.
  - verticalscaler 3y ago
    Correct.
fishtacos 3y ago
>>The IT department decided they were done with their pets, moved everything to a big vSphere cluster, and backed it by a giant RAID-5 array. There was a disk failure, but that’s ok, RAID-5 can handle that. Precisely why, when I was charged with setting up a 100 TB array for a law firm client at previous job, I went for RAID-6, even though it came with a tremendous write speed hit. It was mostly archived data that needed retention for a long period of time, so it wasn't bad for daily usage, and read speeds were great. Had the budget been greater, RAID 10 would've been my choice. (requisite reminder: RAID is not backup) Not related, but they were hit with a million dollar ransomware attack (as in: the hacker group requested a million dollar payment), so that write speed limitation was not the bottleneck considering internet speed when restoring. Ahhh.... what a shitshow, the FBI got involved, and never worked for them again. I did warn them though: zero updates (disabled) and also disabled firewall on the host data server (windows) was a recipe for disaster. Within 3 days they got hit, and the boss had the temerity to imply I had something to do with it. Glad I'm not there anymore, but what a screwy opsec situation I thankfully no longer have to support.
- justsomehnguy 3y ago
  > even though it came with a tremendous write speed hit Only on a writes < stripe. If your writes are bigger then you can have way more speed than RAID10 on the same set, limited only by the RAID controller CPU.
  - fishtacos 3y ago
    Due to network limitations and contract budgeting, I never got the chance to upgrade them to 10 Gb, but can confirm I could hit 1000 Mbps (100+ MB/s) on certain files on RAID-6. It sadly averaged out to about 55-60 MB/s writes (HDD array, Buffalo), which again, for this use case was acceptable, but below expectations. I didn't buy the unit, I didn't design the architecture it was going into, merely a cog in the support machinery.
- rjbwork 3y ago
  > the boss had the temerity to imply I had something to do with it. What was your response? I feel like mine would be "you are now accusing me of a severe crime, all further correspondence will be through my lawyer, good luck".
justsomehnguy 3y ago
We bought a 108-drive DAS some years ago, for a backup storage. I needed to actively object the idea of a 108-wide RAID5 on that. File systems get borked, RAID arrays can have multiple failures, admins can issue rm -rf in the wrong place.
jrumbut 3y ago
> budgets often work differently. Very differently. Instead of a system you continually iterate on for its entire lifetime, if you're in a more regulated research area you might build it once, get it approved, and then it's only critical updates for the next five (or more!) years while data is collected. Not many of the IT principles developed for web app startups apply in the research domain. They're less like cattle or pets and more like satellites which have very limited ability to be changed after launch.

throw0101b 3y ago

> You have a capex budget […] As someone who has worked IT in academia: no, you do not. :)

tonyarkles 3y ago
I'm laughing (and crying) with you, not at you. From my past life in academia, you're totally right. But that kind of reinforces the point... you do occasionally get some budget for servers and then have to make them last as long as possible :). Those one-time expenses are generally more palatable than recurring cloud storage costs though.

jodrellblank 3y ago

At the risk of no true Scotsman, that doesn’t sound like “cattle not pets“; when the cattle are sent to the slaughterhouse there isn’t any blast radius, there’s just more cattle taking over. You explicitly don’t have to replace them with exact clones of the original cattle from tape very slowly, you spin up a herd of more cattle in moments.

logifail 3y ago
> when the cattle are sent to the slaughterhouse Data isn't "sent to the slaughterhouse". Ever. Data can be annoying that way.
- datadrivenangel 3y ago
  The true problem with kubernetes and modern cloud. Not insurmountable, but painful when your data is large compared to your processing needs.
- j33zusjuice 3y ago
  I think the point is that systems that aren’t replaced easily shouldn’t be managed like they are, not … whatever it is you’re getting at here.
  - carbotaniuman 3y ago
    Sadly you can't just spool down data either - data isn't fungible!
chris_wot 3y ago
Analogies always break down under scrutiny. Any cattle farmer would find “spinning up a herd of cattle” to be hilarious.
tonyarkles 3y ago
> you spin up a herd of more cattle in moments Where does the decade of data they've been collecting come from when you "spin up a new herd"?
- ElevenLathe 3y ago
  It's stored in another system that is probably treated as a pet, hopefully by somebody way better at it with a huge staff (like AWS). Even if it's a local NetApp cluster or something, you can leave the state to the storage admins rather than random engineers that may or may not even be with the company any more.

jsjohnst 3y ago

> moved everything to a big vSphere cluster, and backed it by a giant RAID-5 array I’m with sibling commenter, if said IT department genuinely thought that the core point in “cattle-not-pets” was met by their single SuperMegaCow, then they missed the point entirely.

verticalscaler 3y ago

The devops meme referenced, "cattle not pets", probably popularized by a book called "The Phoenix Project". The real point is that pets are job security for the Unix herder. If you end up with one neckbeard running the joint that's even worse as a single point of failure.

datadrivenangel 3y ago
If you end up with one cloud guru running your cloud, that's maybe worse as a single point of failure. AWS system admins may be more fungible these days than unix sysadmins. Maybe.
- verticalscaler 3y ago
  Correct.

fishtacos 3y ago

>>The IT department decided they were done with their pets, moved everything to a big vSphere cluster, and backed it by a giant RAID-5 array. There was a disk failure, but that’s ok, RAID-5 can handle that. Precisely why, when I was charged with setting up a 100 TB array for a law firm client at previous job, I went for RAID-6, even though it came with a tremendous write speed hit. It was mostly archived data that needed retention for a long period of time, so it wasn't bad for daily usage, and read speeds were great. Had the budget been greater, RAID 10 would've been my choice. (requisite reminder: RAID is not backup) Not related, but they were hit with a million dollar ransomware attack (as in: the hacker group requested a million dollar payment), so that write speed limitation was not the bottleneck considering internet speed when restoring. Ahhh.... what a shitshow, the FBI got involved, and never worked for them again. I did warn them though: zero updates (disabled) and also disabled firewall on the host data server (windows) was a recipe for disaster. Within 3 days they got hit, and the boss had the temerity to imply I had something to do with it. Glad I'm not there anymore, but what a screwy opsec situation I thankfully no longer have to support.

justsomehnguy 3y ago
> even though it came with a tremendous write speed hit Only on a writes < stripe. If your writes are bigger then you can have way more speed than RAID10 on the same set, limited only by the RAID controller CPU.
- fishtacos 3y ago
  Due to network limitations and contract budgeting, I never got the chance to upgrade them to 10 Gb, but can confirm I could hit 1000 Mbps (100+ MB/s) on certain files on RAID-6. It sadly averaged out to about 55-60 MB/s writes (HDD array, Buffalo), which again, for this use case was acceptable, but below expectations. I didn't buy the unit, I didn't design the architecture it was going into, merely a cog in the support machinery.
rjbwork 3y ago
> the boss had the temerity to imply I had something to do with it. What was your response? I feel like mine would be "you are now accusing me of a severe crime, all further correspondence will be through my lawyer, good luck".

justsomehnguy 3y ago

We bought a 108-drive DAS some years ago, for a backup storage. I needed to actively object the idea of a 108-wide RAID5 on that. File systems get borked, RAID arrays can have multiple failures, admins can issue rm -rf in the wrong place.

jrumbut 3y ago

> budgets often work differently. Very differently. Instead of a system you continually iterate on for its entire lifetime, if you're in a more regulated research area you might build it once, get it approved, and then it's only critical updates for the next five (or more!) years while data is collected. Not many of the IT principles developed for web app startups apply in the research domain. They're less like cattle or pets and more like satellites which have very limited ability to be changed after launch.