11 ms·
It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracke
by jeffbee 7y ago
It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones at the tops of the racks, or the ends of the rows, are hotter (or colder); there's some difference between the same model of hard disk made in Hungary compared to the ones made in Mexico; at some date the BIOS vendor made an undocumented firmware revision that changes an obscure energy/performance register in your CPU; you have a machine with a dead CMOS battery that worked normally until it was rebooted.
Cattle is a good philosophy but it takes a huge amount of work to approach perfection.
- sitkack 7y agoThere needs to be some level of conformity between instances, they stop being a heard and more of a zoo if the skew is too large. The workloads running on the instances shouldn't be able to tell which instance type they are running on, or your workloads should be written such that it doesn't matter (but at some point it will). Things that grow and move together, wear together, so you will end up with a system that is designed against the empirical contract, not the stated one.
- jeffbee 7y agoMy approach has always been that somewhere in my fleet there is a heat sink that fell off and a CPU running at 400MHz. My last two jobs have started with me sitting down at my desk on day 1 and demonstrating this fact. After concluding that the zoo is unavoidable, the only thing left to do is write the software accordingly.
- saagarjha 7y ago> After concluding that the zoo is unavoidable, the only thing left to do is write the software accordingly. What do you do to write software accordingly? Make it detect when it's running on a dud? Have it run as best as it can anyways?
- jeffbee 7y agoSuicide is a good solution, if some higher-level thing will notice and move the task to another machine. Batch frameworks can kill slow shards, or re-assign their work to faster shards. Clients of online services can direct more traffic to working shards and less or none to slow or broken ones.
- dekhn 7y agoOnce, the head of IT of a company I used to work for was touring the datacenter, passing some new racks filled with blade servers. He stopped, said "why are all the fans running full blast on this rack?" and the admins checked and they were running some test workload at scale somebody had forgotten about a few weeks before. Everybody was embarassed because no monitoring caught it, but the VP of IT did by walking past the cage.
- gerdesj 7y agoI walk the server room daily, every morning. I've tended our monitoring system for 15 years now and I don't trust myself to be infallible. I'm also the MD ...
- deleted 7y ago[deleted]
- jiveturkey 7y agoMD = VP for those not in finance
- bennyelv 7y agoIn the UK MD = CEO (when not finance), so without more context you couldn’t really say. He’s either an MD in finance, which is the equivalent of a VP (roughly) in other kinds of companies, or he’s the CEO.
- killjoywashere 7y agoAdmiral Rickover was known for walking into the engineering spaces of nuclear ships and just throwing a valve handle that would force a reactor scram. Not infrequently on a submerged submarine. Just to make sure the team was on their toes.
- MaxBarraclough 7y agoI'm not a sailor or a nuclear engineer, but that doesn't sound like a great idea. Should the Chaos Monkey approach really be used on nuclear systems? Aside: was it even legal for him to do that?
- walshemj 7y agoback when we where buying hardware for big (at the time) intranet. All the servers where brought of the same batch of suns production line, I recall our sysadmin saying he rely wanted to do the same for the disks ie case of identical drives
- jeffbee 7y agoA super-bad idea because drives made on the same week in the same facility will all fail at the same moment.
- walshemj 7y agoUnlikely that's not how statistics works in production engineering
- jeffbee 7y agoWith respect, I just went through a "code red" at a large, well-known cloud storage company caused by synchronized late-life death of hard disks all manufactured in the same batch. That's the second time in my career that I've been through the same phenomenon. Hard disks that are made together wear out together.
- solarengineer 7y agoI can confirm this. I learned the hard way to buy hard disk drives of the same model but from different batches.
- exikyut 7y agoI'm curious how wide the failure window was (timespan, ramp-up/down, etc), relative to how many devices were involved. And I wonder how well the signal in that ratio might scale down to hundreds or tens of disks.
- walshemj 7y agoShockingly bad production engineering then.
- magicalhippo 7y ago> It is indeed folly to assume that cattle have no identity. My dad had a book over almost all the cow names in Norway[1], per 1988. As a kid I found it rather fun to just flip through it and read some names, often wondering how they came up with them. However since then it seems the tradition of naming cattle has dropped[2] to less than 30%. [1]: "Gullhorn og dei andre : kunamn i Noreg" https://urn.nb.no/URN:NBN:no-nb_digibok_2010111708049 https://urn.nb.no/URN:NBN:no-nb_digibok_2010111708049 [2]: https://www.nrk.no/nordland/kyrne-far-ikke-lenger-navn-1.8357475 https://www.nrk.no/nordland/kyrne-far-ikke-lenger-navn-1.835...
- simook 7y agoI only know of small farmers who name their livestock, but that was back in the States.
- jfk13 7y agoCompared to US-scale farming, I'd guess most Norwegian farmers are "small".
- magicalhippo 7y agoIndeed. When the 1988 study was done, which resulted in the book of cow names among other things, there were about 360k cows in Norway total, and the average number of cows per farm was ~5.6. These days the number has risen, IIRC around 35, though that's still quite a low number compared to larger countries I imagine. I'm pretty sure the variance is quite high however, with a fair number of farms with just a few cows dragging down the mean.
- obmelvin 7y ago> particular well-known machine was unracked are you able to comment a bit further on why this machine was well known?
- jeffbee 7y agoBecause of the way engineers habitually run batch jobs with more replicas than there are machines, this one broken computer had crapped up every map/reduce job in that facility for a long time, and it had been sent to repairs many times without benefit. Many people knew instinctively that if their job was stuck it was probably because of the shard on xyz42 (or whatever the node name was).
- the-rc 7y agoI still remember the machine name. It starts with an l and ends with a 6. Over the course of a couple of years, pretty much all of its components (CPUs, RAM, drives) were replaced at least once. You could look up its maintenance history and it went on and on. I'm not sure if it was well known across all of engineering; from what I recall, it was in a cluster in Oregon reserved for a specific team. Because it was company property, no matter how doomed, they had to get signoff from upper management, close to Eric Schmidt's level, before they could destroy it.
- gwillen 7y agoMy recollection, assuming it's the same machine I'm thinking of, is that it wasn't reserved for our team; rather, we left a do-nothing job permanently allocated to it, in order to prevent some poor other sucker from getting their job scheduled on it. (Because we, through painful experience, were well aware the machine had hardware problems; but we had long since given up on convincing the responsible parties to take it out of the pool, since it passed all their internal tests every time we complained. I don't remember how long this situation existed before someone finally took it out back and shot it.) Could be a different incident and a different machine, though. I'm sure this story happened more than once.
- fizwhiz 7y ago> There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Eyy what would I search on moma to find this video?
- dmoy 7y agoHuh, I can't find it. I can confirm I've read the same thing though, years back.
- the-rc 7y agoThere used to be a go link for it (same as the machine name), but knowing Google, it might be stale. There's a good chance you can find more on the internal folklore site. If that one is still around, too.
- zomglings 7y agoms/die+mothafuckaz
- mjevans 7y agoSounds like they were re-enacting Office Space. If you haven't seen that movie... It's culturally relevant even today. Has a bit of profanity and such though.
- 0xffff2 7y ago> There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field I really hope the video is edited to strongly resemble this scene: https://www.youtube.com/watch?v=N9wsjroVlu8 https://www.youtube.com/watch?v=N9wsjroVlu8
- the-rc 7y agoIt was made to reference exactly that, down to the attire.
- layoutIfNeeded 7y agoI don’t think they make hard disks in Hungary.
- the-rc 7y agoThe infamous IBM Deskstars aka Deathstars were made there.
- lostmyuser0678 7y agoAnymore.
- grawlinson 7y agoOne of my computers has been aptly named 'THESEUS' due to what was replaced on it. By the time it was repaired to an acceptable level, the only original component remaining was the chassis.
- jiggawatts 7y agoHeh... that reminds me of my favourite troubleshooting story! We had a customer with regularly failing tape backups. CRC errors, verify pass failures, even failed writes, and so forth. We replaced the tapes with new ones. Same issues. We replaced the tape drive with a new one. Still the same problems. We replaced the internal ribbon cable and the SCSI controller. No luck. Firmware flashed everything. Didn't help. New server chassis, wiped the OS and reinstalled everything from scratch. Changed the backup software just in case. The backups still failed! Literally no part was the same. I went on site to start looking into things like the power cables, the UPS, or vibration issues. Basically were getting desperate and grasping at straws. I was sitting down in an office, casually chatting with the IT guy while we were waiting for 5pm so we could reboot the server. He's leaning back in is office chair, and he casually picks up one of the tape cartridges and throws it up in the air and then catches it before it hits the ground. Just playing. Over and over. I asked him if he does that a lot. "Yes, it's fun!" he answered. ಠ_ಠ
- jagannathtech 7y agoMany would have hit the ground too! I'm twitching...
- asmosoinio 7y agoThat is quite the story! Sounds like a very large amount of resources spent on that case. What was your companys role? Backup services/devices?
- jiggawatts 7y agoThis was general IT consulting back in the early 2000s. The customer was small, they only had three tower servers and only one had a tape drive.
- karmakaze 7y agoThere was a time when I managed a few clusters of MySQL servers provisioned on OVH bare-metal. [Yes, we know "they're the worst" at least used to be.] Everything was running well until we upgraded to their largest offerings. Turned out they had NUMA memory so had to retune the MySQL parameters. After sorting that out they all worked fine except one machine. Note that we already had a pretty good relationship so I'd already requested the exact same motherboards and BIOS on these machines. Somehow one machine always came up different. I couldn't trust OVH to set the CMOS the same so I carefully checked against one of the 'normal' machines. The cores reported differently and had different affinity characteristics. Never did find out why that was, perhaps a CPU microcode/stepping that I never investigated. Anyway I just worked around the issue (with a different tuning for this one machine) because asking for a replacement resulted in a similar situation.
- masonic 7y ago"There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs..." Are you perhaps thinking of the printer execution scene from "Office Space"? https://www.youtube.com/watch?v=N9wsjroVlu8 https://www.youtube.com/watch?v=N9wsjroVlu8