19 ms·
Yes, the y2k crisis was real, or more accurately, would have been a serious crisis if people had not rushed and spent lots of money to deal with it ahead of tim
by dwheeler 7y ago
Yes, the y2k crisis was real, or more accurately, would have been a serious crisis if people had not rushed and spent lots of money to deal with it ahead of time. In many systems it would have been no big deal if unfixed, but there were a huge number of really important systems that would have been a serious problem had they not been fixed. Part of the challenge was that this was an immovable deadline, often if things don't work out you just spend more time and money, but there was no additional time that anyone could give.
The Y2K bug did not become a crisis only because people literally spent tens of billions of dollars in effort to fix it. And in the end, everything kept working, so a lot of people thought it wasn't a crisis at all. Complete nonsense.
Yes, it's true that all software occasionally has bugs. But when all the software fails at the same time, a lot of the backup systems simultaneously fail, and you lose the infrastructure to fix things.
- duxup 7y agoYeah the sheer volume of just bad data entry type situations with messed up dates could have been for some companies absolutely enormous. Whole businesses would be ground to a halt to stem the mess of bad data and if not prepared they would not have quick fix. In many cases in the age of apps and web apps we can roll out fixes fairly quickly, but in 2000 that was very much not the case for most situations.
- type0 7y agoY2K has this tragicomic portrayal in the movie Office Space
- o-__-o 7y agoahhh i remember as a teenager making sure our board of ed systems were Y2K-compliant. numerous times the principal staff would evacuate the office to take care of some action going on in the halls of the school. leaving me with access to the entire student records db and the VP login and password written on a sticky under the keyboard (not even joking!). in addition this was a laptop so RAS details were conveniently saved in network neighborhood. this left me at a crossroads. i thought about writing malware that would randomly raise peoples semester grades by 4 points (e.g. C- would became a B+). i thought about mass changing grades. i thought about altering a target group of kids i didn't like. all but the last of the scenarios ended with me getting fingered because I was the smart computer kid. if i didn't touch my grades i would have plausible deniability. i wrote the malware. then i watched office space and decided to think about my actions (and promptly forgot as a horny teenager does). soooo glad i didn't release it because years later i went back into the code and found a leftover debug that would have targeted the only 2 kids in the school who had this letter in their last name. tl;dr: office space is real
- mhh__ 7y agoPeople I've met definitely think it was an Apollo 13 type emerging problem rather than a largely preempted issue.
- weej 7y agoI'm having PTSD just thinking about updates of YY to CCYY in mainframe code. Very real and heavy investment across numerous industries from aviation to financial and more. At a macro level, I remember it being pretty well managed risk and mitigation.
- oso2k 7y agoWhen you start 3+ years before the known threshold date, you're giving yourself the best chances at success. I remember flashing BIOSes for weeks in the computer labs at my university.
- exikyut 7y agoOooh. Do you remember any notable number of systems turning into doorstops in the process? I wonder what the practical mixture of NAND flash vs EEPROMs was. I understand NAND wasn't especially stable back then.
- o-__-o 7y agoEEPROM. And there has always been a reflash method for bricked firmwares for all AMI and Phoenix BIOS' as well. Insert disk with firmware, Hold a key, power on. Once upon a time we got somewhere in the neighborhood of 1200 dell workstations and not a single one failed the bios upgrade. 10/10 would do again
- abiogenesis 7y ago> people literally spent tens of billions of dollars in effort to fix it The sceptic inside me is curious as to how much they actually accomplished in their effort to fix it. I mean, yes, they spent billions to get millions of lines of code read, but how many fixes have been made and what would be the cost (when compared to those billions of dollars spent) if they weren't fixed at all.
- TomVDB 7y agoA good friend of mine was (and still is) a civil/construction engineer. Other than some introduction courses in college, he had never done any programming before. But after losing his job in 1998, he was hired by a major European consulting company to work on the Y2K problem. With an auditorium full of others who had barely any programming experience, he was given a crash course in Cobol for a month, and then he had to go through thousands of lines of code of bank software that ran on some mainframe and identify all cases where a date was hard-coded as 2 characters. Extremely tedious work, but there were tons of fixes to be made. The problem was real, and the billions that were spent on fixing things were necessary.
- codeisawesome 7y agoI’m wondering what happened to the people in meeting rooms who suggested that this could be automated if it needed no deep understanding of the COBOL language or the banking domain..? EDIT (for clarity): that’s what I would say, and I’m pretty sure many people smarter than I had said it - so I’m just curious what objections made business push forward with the “relatively unskilled” human labour intensive practices instead, because that’s the history we have now.
- setpatchaddress 7y agoI'm not a COBOL expert, but I suspect you misunderstand both the difficulty of teaching a machine to understand the problem well enough to automate fixes for it and the available computing resources at the time. This was still the era of Windows 95, the classic Mac, and expensive workstations.
- samstave 7y agoDear god, reviving the ghosts of IT departments past!!! Yes, updating JDEdwards AS/400 systems and many a PC was a big project, but i dont recall doing it at that time as being super difficult... frack ist so long ago i cant really recall many of the details other than reporting daily on the number of systems updated. Also, fuck you google - this was when you were nascent a i converted the entire company to your “minimalist” front page vs yahoo, and a few years later you wasted 3 months of my time interviewing to tell me to expect an offer letter tomorrow and called to then reacind the offer (as i didnt have a degree) and then continuing to contact me for FIVE FUCKING YEARS for the same job that you wouldnt hire me for. /off-my-chest
- ncmncm 7y agoIt happened to all of us. But be glad you are not there now. None of those people contacting you, or phone screening, were Google employees. They were, or worked for, contractors. It didn't matter to them or to Google whether anybody they contacted was hired.
- tdy721 7y agoOh it mattered to the recruiters, right up until they realized he didn’t fit the template.
- klenwell 7y agoI remember hearing a story on NPR years ago years after Y2K with someone knowledgeable on the subject that addressed this question. He gave basically the same answer you did: a lot of effort went into avoiding disaster and a lot of people treated it after the fact as if it was hype or hysteria. I recall the interview because it changed my significantly thinking on the subject. (I wasn't in the industry at the time of Y2K.) He also noted that a lot of American firms worked with Indian firms to resolve it. Indian engineers were well suited to the problem, according to him, because they had been exposed to a lot of the legacy systems involved during their studies. He said an interesting consequence of this was that many American companies concluded they could use lower wage Indian engineers on other software projects so it helped initiate a wave of offshoring in the early 2000s. I couldn't find the story I heard in the NPR archives, but they do have a number of stories reported right around that time if you want to see how it was being discussed and treated by NPR at the time: https://www.npr.org/search?query=y2k&page=1 https://www.npr.org/search?query=y2k&page=1 I did find this NPR interview with an Indian official from 2009 that notes Y2K's impact in US-Indian economic relations: But I think the real turning point in many ways in the U.S. relations was Y2K, the 2000 - year 2000, Indian software engineers and computer engineers suddenly found themselves in the U.S. helping the U.S. in adjusting itself to the Y2K problem. And that opened a new chapter in our relations with a huge increase in services, trade and software and computer industry. https://www.npr.org/templates/story/story.php?storyId=120738550 https://www.npr.org/templates/story/story.php?storyId=120738...
- ZainRiz 7y agoIt was also really difficult to detect bugs since no good testing paradigms had been established yet. This created an exploitable opportunity for the few who knew how to write automated tests. During the Y2K panic Sun Microsystems (IIRC) announced that they would pay a bounty of ~$1,000 per Y2K bug that anyone found in their software (due to the lack of automated tests, they didn't even know how to find their Y2K bugs) James Whittaker (a college professor at the time) worked with his students to create a program that would parse Sun's binaries and discover many types of Y2K bugs. They wrote the code, hit run, and waited. And waited. And waited. One week later the code printed out it's findings: It found tens of thousands of bugs. James Whittaker went to Sun Microsystems with his lawyer. They saw the results and then brought in their own lawyers. Eventually there was some settlement. One of James' students bought a car with his share.
- skissane 7y agoFrom what I've heard, a number of IT departments used it as justification to dump legacy systems. For example, at a former employer of mine, a big justification for getting rid of the IBM-compatible mainframe and a lot of legacy systems which ran on it (various in-house apps written in COBOL and FORTRAN) was Y2K. In reality, they probably could have updated the mainframe systems to be Y2K-compliant. But, they didn't want to do that. They wanted to dump it all and replace it with an off-the-shelf solution running on Unix and/or Windows. And, for reasons which have absolutely nothing to do with Y2K itself (the expense and limitations of the mainframe platform), it probably was the right call. But, Y2K helped moved it from "wouldn't-it-be-nice-if-we" column into the "must-be-done" column.
- ZainRiz 7y agoAs they say in DC, never let a good crisis go to waste
- o-__-o 7y agoHow apropos to current events.
- fishnchips 7y agoWasn’t this originally said by Winston Churchill?
- wp381640 7y agoWorked in a project to migrate a huge and critical .gov database from an AS400 nobody understood anymore to Microsoft SQL Server. We had to guess some of the values in some bitwise fields through trial and error. Made it with a few weeks to spare - nobody noticed.
- Angostura 7y agoI'm quite surprised no-one understood an AS/400 in 2000 - they were still pretty current and an important part of the IBM range, surely.
- hinkley 7y agoAnd the dotcom bust came when it did in part because the Y2K spending was lifting all boats. A bunch of companies shifted from buying too much to buying not much.
- ghaff 7y agoI'm not sure why you're being downvoted. It wasn't really related to the dotcom bust per se. But the falloff in Y2K spend was part of the overall drop in IT spend on consultants, system refreshes, etc. We use dot-bomb as a shorthand but there were clearly a number of things that happened in the same general timeframe that made it something of a perfect storm.
- jshevek 7y ago>treated it after the fact as if it was hype or hysteria I agree with everything you and the parent said, but there was also hype and hysteria on top of it all - especially among the communities of preppers, back to the landers, and eco-extremist types. I met entire communities of technophobes who declared I was just ignorant or 'a sheep' and refused to believe me when I tried to explain that the industry was working on the problem and had a solid shot at preventing major catastrophe. Frightening predictions of power plants going offline for days, erratic traffic lights causing mayhem, food and water supplies being interrupted, even reactors melting down were not uncommon. As always, there is money to be made by exploiting people's fear. This hysteria was considered an irrelevant side show by most educated people, the way I look at the chem trail people or the way I used to view the anti-vaxxers [1], but the unjustified hysteria was real. [1] Before I learned their numbers were growing and could threaten herd immunity in some areas.
- marcus_holmes 7y agoThis. So many parallels with every "looming disaster" since. The media went into frenzy mode a few times, predicting the End Of The World As We Know It and scaring everyone witless. And a few people did take it really seriously and spend the Millennium in some remote place where they would stand a chance of surviving the collapse of Western Civilisation.
- klenwell 7y agoTrue. The first image that pops into my head when I hear Y2K is a Simpsons joke where airplanes and satellites start falling from the sky and landing in their backyard: https://www.youtube.com/watch?v=x0bBYK-7ZiU https://www.youtube.com/watch?v=x0bBYK-7ZiU Oh, and Dick Clark melts.
- swiftgoose 7y agoI was working for Hughes, now Boeing Satellite Systems, for a couple of years doing Y2K remediation. I was writing ground station code that were on VAX/VMS, basically older hardware and software. So that thought did occur to me of satellites falling from the sky. From what I understand, everything worked out ok. There might be only one instance that we know of that was unanticipated when the year roll-over.
- wickerman 7y agoI had a teacher in college who'd been part of a project to fix the Y2K bug in the bank he worked at the time. He said to us when talking about project management, that the only project he saw actually finish on time was that one. The pressure was immense. People don't realise what a huge success story the Y2K situation really was.
- foobar1962 7y agoI worked at a University that had a 28 story tower. Around 1998 they set the clock in the lift management system to 1 January 2000 and all the lifts lowered themselves to the bottom of the lift wells because they hadn't been serviced for 50 years (or something). So yes indeed stuff was broken, but it got fixed before the big date.
- bridgerj 7y agoThis sounds apocryphal. If setting the date forward triggered a maintenance warning, it sounds like the date system was working correctly. More importantly, lift systems are not sensitive to long duration dates. They do not need to be. This story reflects one of the myths that emerged from the media. Around 1997 they started discovering the topic of embedded systems and that many devices, including lifts, contained "computers." Not understanding the restricted and specialized nature of embedded systems, they then claimed all these systems were vulnerable to Y2K, and that the whole world was about to crash. Most embedded systems, in lifts and other devices, had been designed in the 1990s. If they used dates, they did not use mistakes from the 1960s.
- foobar1962 7y agoThe system was using 2-digit year and the service interval was went negative. It indeed happened, I was working in the IT department at the time. The system was probably designed in the 1970s.
- econcon 7y agoI wonder how many y2k deniers are there.
- zerkten 7y ago> Yes, it's true that all software occasionally has bugs. But when all the software fails at the same time, a lot of the backup systems simultaneously fail, and you lose the infrastructure to fix things. This is especially important if you think about what things were like in the late-90s. As a teenage geek in Northern Ireland I read about the the first tech boom, but locally there was very little exploitation of tech and you were an anomaly if you had dial-up internet at home. There was very limited expertise available to fix your computer systems in the best of time. Big companies had complex systems, but they could also afford contracts with vendors in the UK mainland, or Eire, to fix systems. It was much more limited for other companies. While people were generally not relying on personal computers too much, small businesses had just reached that tipping point where Y2K would have been painful. As a result, they took action. I only got to visit the US a few times in the early-nineties and it seemed so futuristic (I got a 286 PC in '93 to use alongside my Amiga). I imagined the Y2K problem as being much more painful in the US, and I wasn't accounting for the distribution of expertise. One great experience was summer training courses that the government had organised with a IT training company in Belfast. These were free and it was a like a rag tag army of teenagers and older geeks from both sides of what was quite a divided community. The systems we were dealing with were fairly simple. It was mostly patching Windows, Sage Accounting, upgrading productivity apps (there was more than Microsoft Office in use) etc., but the trainer was normally teaching more advanced stuff like networking and motherboard repair so we spent a lot of time on that.
- mhewett 7y agoI was a computer operator in the late 1970s and I used to read ComputerWorld which was the biggest enterprise tech "journal" (actually a newspaper) of the day. In the late 1970s they had a number of articles commenting on the need to fix the Y2K problem. So smart people were planning for it well ahead of time. On the other hand memory and storage were so limited then it was probably hard to get permission to allocate two more bytes for a 4-digit year. But people were definitely thinking about it for a long time before the year 2000.
- bridgerj 7y agoWhat you're saying here is that there was no crisis. The fact that people worked on date issues for a decade does not mean the public scare campaign in the late 90s was justified. You could equally claim airplanes would crash if they weren't refuelled, or people would die if garbage wasn't collected. In those cases, the media is educated enough to know the claims are pointless. In the case of y2k, they had little understanding of how enterprise IT worked, and lots of businesses who were happy to encourage that ignorance.