8 ms·
To add a little nuance: A global retailer you've definitely heard of that used to own stores in Germany spent a lot of time preparing for Y2K. This was a long
by Diederich 7y ago
To add a little nuance:
A global retailer you've definitely heard of that used to own stores in Germany spent a lot of time preparing for Y2K. This was a long and painful process, but it got done in time.
But problems still slipped through. These problems ended up not being that big and visible because a large percentage of the code base had just been recently vetted, and a useful minority of that had been recently updated. Every single piece had clear and current ownership.
Lastly, there was an enormous amount of vigilance on The Big Night. Most of the Information Systems Division was on eight hour shifts, with big overlap between shifts at key hours (midnight in the US), and everyone else was on call.
As midnight rolled through Germany, all of the point of sale systems stopped working.
This was immediately noticed (the stores weren't open, but managers were there, exercising their systems), the responsible team started looking into it within minutes, a fix was identified, implemented and tested within tens of minutes, and rolled out in less than an hour.
Pre-Y2K effort and planning was vital; during-Y2K focus, along with the indirect fruits of the prior work, was also critical.
- deepaksurti 7y ago>> This was immediately noticed ..., a fix was ... rolled out in less than an hour. I guess Agile/Scrum etc wasn't a thing then else the hour would be spent in a stand up!!!
- Psyladine 7y agoWe'll just send an email out EOB that restates the MVP they signed off on 18 months ago was reached per spec & additional project budgeting would have to go through normal triage management before committal.
- hyperpape 7y agoWhy the generic description? Are you unable to say which retailer?
- onychomys 7y agoThe only global retailer that we've all heard of that owned stores in Germany in 1999 but doesn't now (note the "used to own stores" in the comment) is Walmart: https://www.theguardian.com/business/2006/jul/28/retail.money https://www.theguardian.com/business/2006/jul/28/retail.mone...
- Diederich 7y agoThat's a bingo. I wasn't particularly trying to obfuscate. I'm curious how you so quickly figured it out though.
- carstenhag 7y agoAs a German I noticed that you said global retailer. Only Walmart is (was) truly global imo and left Germany.
- Diederich 7y agoJust being cute. It's WalMart Stores, Inc.
- hyperpape 7y agoThanks, just wanted to nudge you to make the example as complete as possible.
- rolandog 7y agoI love these kinds of war stories of incredibly capable people. That was great planning for the unknowns.
- Diederich 7y agoI've worked in silicon valley for the past 11 years, in startups as well as two so-called FAANG companies. There are a lot of brilliant people out here. So that is well known. What's not well known is this: there are pockets of brilliant people elsewhere, including Bentonville, Arkansas. Especially in the years running up to 2000, I had the pleasure and honor of working with some of the most talented people in the world, even though we were all looked down on by many people on the coasts.
- wglb 7y agoWhile I don't have direct knowledge of Bentonville, the general impression that I have formed is that they are one or two cuts above any other enterprise of their size.
- pjmorris 7y agoI once worked with an F500 retailer who had Bentonville 'graduates' among their IT staff. Sharp people among other sharp people. It takes a lot to keep the lights on pretty much everywhere. Walk in to any big box store and look around for how many gadgets and systems there are.
- Diederich 7y agoMost everyone would be amazed at much networked equipment is in a WalMart supercenter. By the time I left, there were four routers, four big core switches, a couple dozen 'edge' switches, and scores of access points. And that's just the networking gear. As of 2009, my team's network discovery system had identified tens of millions of separate IP addressable nodes on the global network. At that scale, everything had to be almost perfectly automated. And, to be clear, this was not a static, boring network. Stores, home offices, distribution centers, you name it, were being added, moved, expanded, deallocated 24 hours a day, all over the world. And the device configs were far from cookie cutter. An eye-wateringly complex heuristic was required to dynamically create router, switch and access point config that would allow a given store to function. And, one quarter in the early 2000s, we (Network Engineering) achieved six sigma (99.9999%) global uptime. Things did start going downhill around 2004 or so, a few years after Kevin Turner became CIO.
- fleetingmoments 7y agoCouldn't the rollover be simulated ahead of time by simply setting the date forward? Seems crazy that you had to have people on the ground fixing things as the clock struck 0.
- davesmylie 7y agoSure. On systems under your control. It's the network of interacting systems that prevents this from being so simple.
- protomyth 7y agoYep, the super fun of getting a Y2K compliant specification of how the data will come over the interface only for it to fail and have '1910' instead of '2000' in the 4 character date field that worked in test. People on site to correct problems were a needed item for some even after testing.
- salgernon 7y agoAll systems in the gigantic clockwork of interconnected legacy would need to be rolled simultaneously- and it wasn’t unreasonable to expect that there are systems in that mix that nobody knew could be affected. To this day I read about systems that come up as a surprise when they start to fail and nobody owns them and the persons responsible have long since lived on.
- Diederich 7y agoTwo decades ago, I was roughly familiar with the details, after the fact, but I can't remember now. As sibling comments have noted: these systems are an exceedingly complex pile of separate but interdependent pieces. The rollover from 31-Dec-1999 to 1-Jan-2000 had happened thousands of times before the actual day came. Some piece/assumption/etc had been overlooked.
- marcus_holmes 7y agoYou can do that with test systems, but not live ones. People object to having their bills dated 98 years ago and overdue on the day they were issued ;)
- Roboprog 7y agoI remember having to go into work at 4:00 AM to replace the shift that spent the night in the operations support center. Fortunately, the guys that spent the previous year sweating the details got it right. We had one system that printed year 100 in its log which was only used by humans. Unfortunately, I missed the party and cash the night shift got.
- Diederich 7y ago> that printed year 100 Did it print the year "100" or "19100"?
- Roboprog 7y agoI don’t remember any more. We fixed it a few days later, but it didn’t cause much trouble so it was largely forgotten.
- Scuds 7y agoAt the Living Computer Museum in Seattle they have an IBM S/360 that prints out we’re in year 19120.
- dag11 7y agoMaybe it knows something we don't...
- iso947 7y agoI’m sure I’ve seen websites in the last year or two from the 90s that still do this
- wolf550e 7y agohttps://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/getYear https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
- samstave 7y ago
- marcus_holmes 7y agoI went on holiday over actual Y2K, flew to Australia and spent the Big Night on the beach, completely incommunicado. It focused the whole team, we had to be OK for it, because if we weren't we were fucked ;) We did multiple dry runs, setting server times to 31/12/99 23:55 and waiting 5 mins to see if anything died. Mostly it was OK, most of our systems were OK, but a few were dodgy and needed some changes. The Big Night rolled around and we were OK. The planning had worked, and nothing fell over. I got the email on Jan 1st - all good.
- EvanAnderson 7y agoI spent the night in the processing center in the basement of a small regional bank helping run tests and look out for anomalies. It was the culmination of years of work and a lot of money to make sure that it was a non-event.
- samstave 7y agoWhen i worked at intel in the mid-late 90s - i recall talking to an analyst one night in SC-5 on the balcony - and they were having problems rebuilding the financial system’s fields to be long enough strings for the amounts of money intel had... Did banks go through some of the same growing pains
- hermitdev 7y agoI remember hitting a Y2K bug before 2008. We were working on a huge release, updating all of our external dependencies to the last version and moving from 32 to 64 bit. A 3rd party commercial lib we were using at the time would parse/format %y as a 2 digit year. There was a macro definition, that when defined would handle a 4-digit year, but it was off by default. We caught it early in development, but it was still a pain in the ass rebuilding all of the projects that depended on it.