14 ms·

I swear this guy exists at every company I've ever worked. He's the guy you go to when you find some legacy code which you have no idea how the hell it works a

• throwawaaarrgh 3y ago
  "... And the AIX machines were a real beast. We had 20 of those things, and each had its own 20 amp circuit. Had a tendency for the grounding to the steel case to fail. If you wanted to upgrade it you'd first need to grab a pair of thick leather gloves...."
  • foobarian 3y ago
    I for one do not mind that part of computing getting abstracted away by the cloud, whatever its other failings. :-)
    • bayindirh 3y ago
      Working on the hardware directly is more than half of the fun.
      • foobarian 3y ago
        An old IBM tech told me that hard drives failed when power cycling equipment could be temporarily resuscitated by placing the drive on the floor, twisting it hard by hand, then as quickly as possible connecting it back into the machine and powering on. Apparently getting the platters to spin helped them get around a worn out motor that couldn't make the initial spin-up.
        • bayindirh 3y ago
          I would believe them. I have heard a similar story. A data center had to go through a planned power cycle event. All systems were working when they shut down, but ~40% of the disks failed to spin up due to worn down motors. That's a lot of disks.
          • vbezhenar 3y ago
            What’s a procedure to avoid it? Reboot machine once a week so disks fail not all at once?
        • wrycoder 3y ago
          Gotta know which way to twist it.
        • tristor 3y ago
          Yes, stiction is a real issue in old hard drives. In fact something that used to work well if you were cold booting an old server even in the mid 90s, with the expectations of an immediate migration, was giving the drives a good sharp slap to help break stiction so they could spin up. The only issue is it could cause a head crash. Another fun trick from back in the day was sticking hard drives in the freezer to cause the platters to shrink enough to dislodge the head and restore the air gap if you had a head crash so you could try to recover the data. As much as I love hardware, I much prefer our current solid state wonderland.
          • justsomehnguy 3y ago
            > to shrink enough to dislodge the head and restore the air gap Huh? What I heard is what a cold drive would be a bit more magnetically stable and IMMSMV that surely worked, because I never had a drive with a stuck heads but I did had drives what would just abort the read or return gibberish, but after an hour in the freezer they would read just fine, till they heat up again. That's for the drives mfg after 2000, if that matters.
            • tristor 3y ago
              My understanding may be incorrect, after all the freezing the drives trick is mostly something that was shared among sysadmins like an old wive's tale. Nonetheless, it does work (or did). The way I had always understood it was that if you had a head crash (which was caused by the head physically contacting the platter, overcoming the resistance of the air gap between the head and platter, usually due to physical impact) that the magnetism of the head would prevent it from lifting back up on its own, and that freezing the drive would cause the metal in the platter to contract away from the head, which was restricted in movement by its armature, thereby restoring the air gap and lifting the head away. If you started the drive spinning before it heated, the head would stay out of contact and you could successfully read data (some of it, for awhile). That said, my understanding may be incorrect.
        • KMag 3y ago
          I remember upgrading from a 4.3 GB HD to a 20 GB HD around 2001. I had the old hard drive sitting on my desk, and the new hard drive sitting in the case, both attached to the motherboard via IDE ribbon cables. Most of the way through backing up the 4.3 GB HD to the 20 GB HD, I heard a screech and the old hard drive twisted slightly on the desk... conservation of linear momentum. The drive was visible to the BIOS, but refused to spin up after that. I put the hard drive in a ziplock bag, put it in the freezer to give the parts slightly more clearance from thermal contraction, and slammed the hard drive nice and hard on the desktop to free the stiction. That revived the drive long enough to finish my copy. I did have backups of all of the really important information, but restoring from a stack of floppies is more tedious, slower, and less fun than percussive maintenance on a hard drive. Back when I had an internship working on MEMS gyros for GPS guided mortar rounds, we had to sometimes perform similar percussive maintenance if static electricity had caused the moving parts to contact the substrate. We took the gyro, and smacked it hard on the desk in an "eyeballs out" orientation to give it something like 10 to 100 Gs of acceleration in an attempt to un-stick the MEMS gyro.
          • SomeRndName11 3y ago
            Yeah, did same in 1999 or 2000, but we _did_ _not_ take the HDD from the freezer, just pulled the ribbon and power cable through the narrow slit between the door's rubber gasket and the fridge's body.
          • KMag 3y ago
            Clearly, I meant conservation of rotational momentum, not conservation of linear momentum.
      • throw0101b 3y ago
        > Working on the hardware directly is more than half of the fun. It never really worked until you gave blood to the computing gods on a sharp edge of a metal case.
        • bayindirh 3y ago
          Looks to hands and fingers intently Hmm, yes, true.
        • RajT88 3y ago
          I have 6 stitches in my left hand from such an offering.
          • KMag 3y ago
            Unless it was a recent incident, you should probably see a doctor about getting those stitches removed.
        • awnion 3y ago
          "Hey, look, I have all five fingers! Three on one hand and two on the other!"
        • Arrath 3y ago
          I feel like the general improvement in fit & finish of cases and chassis is overlooked. I, for one, haven't flayed open a knuckle on a random razor edge or burr on the inside of a case in like 15 years.
    • philk10 3y ago
      Every Friday I had to take a bunch of magnetic tapes to the bank so they could look after our backups ( which was a good excuse to pop into the pub next door to the bank...) Oh, and we never ever tested that the backups actually worked
      • KMag 3y ago
        My first job out of school was engineering consulting, writing software models of pre-production military radio hardware. The client was a military contractor, and they paid a crazy sum in order to have 3 DAT tapes: one tape always in the drive used to make the daily backups of the shared working directory, one tape always stored in a bomb-proof bunker, and one tape potentially in-transit via courier. I think it was once a month that the client's sysadmins rotated in the "unused" tape, sent the latest tape to the bunker, and waited for the old tape from the bunker to return as the "unused" tape. Due to absurd government requirements, we were not allowed to use any version control software other than PVCS, and PVCS could only be used for storing official releases of the models. So, the last person in the office on Friday needed to make a dated zipfile of the shared project working directory. So, one day I convinced the project manager to just let me rename our working directory, create a new empty one in its place, and tell our sysadmins that I accidentally deleted the contents of our working directory. It turns out that the client's sysadmins hadn't set up anything to expunge the oldest backup and had dutifully ignored the alarms that the tape was full. So, the latest available backup was from about a week after they last changed tapes! The senior sysadmin prominently had a sign on his cubicle reading "Programmers are the ditch diggers of the 21st century." Those same sysadmins were supposed to keep the server room door open while I reinstalled Solaris on a Sunfire V1280 via serial connection from my laptop, because there were cables carrying TSSCI (Top Secret - Secret Compartmentalized Information) traffic. However, the V1280 consumed so much power that the server-room was under-cooled, so the sysadmins illegally shut the door to keep their cubicles cooler. They had top-secret clearances. They knew their obligations to keep me away from those TSSCI cables. Those lazy lazy arrogant sysadmins. Anyway, unless you have recently passed a restore-from-backup drill, you don't really have backups. Even if you pay absurd amounts of money for sysadmins with top-secret clearances and absurd amounts of money to store your backups in a bomb-proof bunker.
  • deleted 3y ago
    [deleted]
  • baz00 3y ago
    You kids sitting there. All you do is change your EC2 instance size and restart it. Back in my day I had to go and install the RAM by hand. And you know 128Mb of RAM then took up a whole suitcase and weighed more than your laptop The I fall asleep at my desk, periodically waking up shouting "STOP-A!" and hallucinating about being bitten by thick ethernet vampires. Going to make myself a coffin...
    • arethuza 3y ago
      "I had to go and install the RAM by hand" When the Sun 4/330 I used to have needed a RAM upgrade someone flew up from London to Edinburgh with the RAM and installed it. Fortunately we were in the Grassmarket about 10 minutes from the Castle so he had plenty of things for him to do to keep him amused until his return flight. Edit: I checked and the max the 4/330 could take was 96MB so I suspect the upgrade was an awesome 64MB. That machine cost more than most cars I have owned - even without allowing for inflation.
      • baz00 3y ago
        Yeah crazy money. My father used to import RAM when it was really really expensive in the 1980s. He used to take it on planes from Taiwan as hand luggage handcuffed to himself because it was that valuable.
        • smcameron 3y ago
          I worked in a shop in the early 90s with a bunch of Suns, and we had hired this new sysadmin. One day we noticed that half the RAM was missing from a workstation, and few minutes later noticed half the RAM was missing from all the workstations. Turned out the new sysadmin was a cokehead and had stolen it.
          • baz00 3y ago
            Oh man that's funny. Only because something similar happened to us. Hired new cleaners circa 2001. Got into the office one morning and all the PC cases were open. All the RAM and CPUs had been stolen.
  • sylario 3y ago
    I did a real internship (as in it was for my diploma and lasted only a few months) in a medical facility operating Big Machines. I remember a guy soldering a new battery to some old DEC/Digital hardware.
  • adolph 3y ago
    "They were afraid to turn off or reboot the DEC Vax for fear that if the disks stopped spinning they wouldn't start again. I saw it once at the retirement ceremony and it didn't look like a server but more like a set of industrial washer and driers. The users transferred files using the Kermit protocol and printed locally by a telnet client that could understand the special characters that redirected character output from the terminal to a lineprinter."
• Waterluvian 3y ago
  Maybe we're thinking of two different archetypes, but I wonder if their "hard not to love" nature is part of the survivorship bias of why they're still there.
  • itronitron 3y ago
    A former coworker of mine claims that he lasted over forty years in the company out of sheer stubbornness. He might still be there.
• scns 3y ago
  Easy to love?
  • CrazyStat 3y ago
    "Hard not to X" is not the same as "Easy to X". It may be easy to do something but also easy not to do it. It would be easy for me to yell "Elephant!" right now, for example, but also easy not to do it.
• KnobbleMcKnees 3y ago
  They are hard not to love. They also hate each and every one of you.
  • daddylongstroke 3y ago
    Not necessarily hate...there's always the 1-2% of new hires that are highly impressive and gives us hope for the future. They tend not to last too long. The rest seems hell bent on never even cracking open documentation, and keep asking "where are the architecture videos?" (at least where I work).
  • the_sleaze9 3y ago
    This has not been my experience at all
    • KnobbleMcKnees 3y ago
      They've hidden it well
  • danesparza 3y ago
    Or at the very least: incredibly suspicious of every one of you, and very unimpressed with the latest fad in software development.
• theteapot 3y ago
  I think they're called senior devs who've been there longer than you.
• fnordpiglet 3y ago
  They also tend to be brilliant engineers. It’s a heck of a lot easier to always be writing 1.0 of something. It’s a hellaciously difficult engineering task to improve a system that’s been improved upon for decades. These folks should be admired and respected, they’re by far the most crucial people at any company older than 30 years old. Instead we get condescending garbage like the linked article.
  • tomatotomato37 3y ago
    I just wish hiring managers saw it the same way. It's hard to explain how your last major project was a web enhancement that would have taken a couple days on a virgin codebase, because for you it involved a month of excavating through multiple strata of code dating nearly two decades old, spread across three codebases, two backend languages, a code generator which no one working there understands anymore, and a library who's documentation now only exists on the wayback machine.
    • fnordpiglet 3y ago
      A secret I’ve learned is you only need one job, so you don’t need to take just any job or be hireable at every job. It might take longer, but finding a manager that “gets it” is very important. The most important thing is to find someone who recognizes further that a team is more than the sum of its parts, that each individual is unique and not a cog to be evaluated on a metric rubric and forced into compliance with some idealized engineer. They do exist - I was one for many many years. I’ve recently given up in disgust though and switched back to IC, but senior enough I can shape how managers manage engineers. Regardless, my advice is, if it’s helpful, find the right job with the right team and the right company. It’ll take longer, may pay less at first, but finding the right place that respects work at its value not at its marketability is key to any successful career.
    • urtie 3y ago
      Or a library who's code now only exists on the wayback machine... and of the modified version that you actually use only a single copy exists on an old machine that one of the devs in the company never bothered to hand in for recycling... Never mind that it's part of a decade old contract stipulating a quarter century support... Greenfield is easy. :)
• dkarl 3y ago
  It's funny, startups turn over so fast that I've managed to become that guy while staying at a company less than two years.
  • renegade-otter 3y ago
    Hard to be anything else if the entire job is spinning up a yet another microservice and then spending most of your time debugging in production because shit keeps breaking all the time.
    • norgie 3y ago
      Oh good, we’re not the only ones. The rest of my time is spent saying I can’t help people fix problems with some old microservice my team “supports”
  • jollyllama 3y ago
    You'll have that. I've always said those places should keep an anthropologist on staff.
• onlypositive 3y ago
  Damnit that's me at the moment. I need a new job.
• blantonl 3y ago
  ...these OS/2 machines were once interconnected by a networking technology called 'token ring'.. no joke. It was basically an entire network where if one single node on the network misbehaved it would bring down the entire network - they didn't self recover either! you had to reboot them all. I remember once when I misconfigured a token ring card on a PC and it brought down the mainframe! It shut down the bank for 30 minutes"
  • actionfromafar 3y ago
    To be fair, in those days you could sink Ethernet (broadcast hubs, no switches!) with a faulty or malicious PC spamming the network, too. Didn’t have to reboot the others, though.
    • arbitrage 3y ago
      > in those days you could sink Ethernet with a faulty PC That is still possible in the modern era.
      • 13of40 3y ago
        I used to work in a place where every office on the floor was connected to the same ethernet hub. We also tested a server product that became a DHCP server if you checked the wrong box, so periodically everything would stop working until the team lead came out of his den and went to yell at the new guy.
      • Doxin 3y ago
        step one: disable spanning tree, because the LAN is so big it takes half an hour for a port to come up after plugging something in step two: plug a very cheap switch into the network step three: have someone move their desk around, and plug a spare cable into the switch at both ends step four: spend a day doing a binary search by unplugging banks of switches to figure out where the heck all these packets are coming from.
    • tannhaeuser 3y ago
      What do you mean hubs? Back in the day we manually pierced thick Ethernet coax to get access; of course you could ruin the whole segment running tens or hundreds of meters if you didn't take care.
• coffeebeqn 3y ago
  I’ve seen a few of those. Some are worth their weight in gold. Most are set in their ways from the trauma of trying to make everything not fall apart for so long. Almost all have stagnated and are incapable of bringing in new ideas. Almost all are husks of human beings from putting up and personalizing so much of company dysfunction
  • diegoperini 3y ago
    > incapable of bringing in new ideas But they are REALLY capable of detecting bad ideas.
    • giraffe_lady 3y ago
      They always say no, and are always correct.
      • coffeebeqn 3y ago
        I mean I’ve literally had people like this fight adding in a version control system.
        • bqmjjx0kac 3y ago
          How can we be sure we can trust this "git" thing? Besides, CVS already does everything we need.
          • SomeRndName11 3y ago
            The CVS receipts are too long though.
            • ycombobreaker 3y ago
              A real life application of Odysseus's Oar. I have mentored junior devs who have "heard" of SVN and also think CVS is just a pharmacy. It was a real gut-punch the first time.
• sanderjd 3y ago
  This is actually something I've identified as a major con of working at startups (there are a lot of pros too, and I'm at a startup right now that I love): It is impossible to have these people, because the company is just too young. And also I think this kind of person doesn't usually join a startup for its first, like, decade.
• WaitWaitWha 3y ago
  I recognize the parody about me. ;) Do you know why I am giving you the two hour history? First, I recognize that I need human interaction, but do not really like people. Second, I am hoping that you will learn & understand instead of just memorize & regurgitate so in the future you can resolve it yourself instead of bothering me. Now, go away; I had my week's fill of socializing. /joking - or am I?
  • Aeolun 3y ago
    > Second, I am hoping that you will learn & understand instead of just memorize & regurgitate so in the future you can resolve it yourself instead of bothering me. Where does all this hope come from? I’m barely 15 years into my career and I’ve nearly given up…
    • loa_in_ 3y ago
      Hope is dangerous, for it's itself very hard to resist.
      • salawat 3y ago
        I've started to attempt to gently grind that into juniors. After a certain point, hope is a vice you cannot afford. It is simply something you outgrow, if only because of the unceasing retirement from the mortal coil of those that can bail you out. Hope is for the young and inexperienced. Everytime I come to your rescue, you'd best pay attention. I have an expiration date.
        • loa_in_ 3y ago
          Give them a break! It's an advanced lesson.
  • InvertedRhodium 3y ago
    For me, it's usually trying to rapidly justify the existence of whatever Rube Goldberg machine I'm having to explain. There's always reasons behind the madness, even if they're not particularly good ones.
• derealized 3y ago
  > They're hard not to love. I often search for a clearcut answer to a technical question and I'm met with a 2 hour history lesson into a decade of company politics and failed replatforming projects. Yeah, thanks for telling me why John from accounting was a dick 10 years ago and you had to code this module in a certain way. I really don't care. I'm new to the codebase and I just want to know how it (the codebase) works. I'm currently in this situation and a colleage never gives straight answers to anything. It's always some little rant about something and when it's done, I still haven't got my answer.
  • digging 3y ago
    Easier said than done, but try telling them that. "I appreciate you sharing context, but it's too much to take in at once. It would work better for me to get a more clear-cut answer."
  • cochne 3y ago
    > I'm new to the codebase and I just want to know how it (the codebase) works. You should want to know _why_ it works that way too if you want to do any meaningful work with it. Context matters. I’ve seen many cases where the way something works seems dumb, only to learn later they had already tried the “smart” way but ran into some obscure problem which the “dumb” way solves.
    • wonderwonder 3y ago
      This is important. As soon as we come to the understanding that the coders that came before were not all idiots then we are forced to ask the "why". There is generally a decent reason why something is coded the way it is. Could be as simple as it was an emergency and meant to go back and fix it but never had the time or it could be a valid business edge case that absolutely had to be wedged in. There is almost always a why. Otherwise you tell everyone about a fix you made that 3x performance and you see their faces go gray as they explain you just shut down some archaic but mandatory process in Singapore.
      • OOPMan 3y ago
• racl101 3y ago
  This is where another one of your Pied Piper boxes would go. Okay. Let me show you the next location where we would install one of your Pied Piper boxes. ... okay
• ericmcer 3y ago
  It doesn't seem like a bad way to go. If the company is stable enough and you can cash in your engineers salary for 25 years and have crazy good job security. Seems better than being a 50 year old trying to wow a 28 year old interviewer with your skills on whatever hot new framework just came out.
  • Gigachad 3y ago
    If I’m still building webshit SaaS products at 50 then something has gone wrong. I should be either retired or some kind of high level manager.
• wwilim 3y ago
  Just one?
• bick_nyers 3y ago
  They are worth their weight in gold, truly. I had the pleasure of interacting with one such man at my previous company weekly for an extended period of time while planning/designing a rewrite. Listening, thinking, and questioning his stories and perspectives allowed me to formulate problems and architecture in a more first-principles, fundamental way and I believe that experience really helped me mature as a young engineer (as well as the project).
• foxyv 3y ago
  I usually give the history lesson to forestall the endless series of "Oh god, why do we do this?" questions.
• MilStdJunkie 3y ago
  I am slowly turning into this person and I am not sure how I feel about that.
• m463 3y ago
  https://users.cs.utah.edu/~elb/folklore/mel-annotated/node1.html https://users.cs.utah.edu/~elb/folklore/mel-annotated/node1....