11 ms·
Microsoft confirms Exchange Year 2022 problem
- sharmin123 5y ago[dead]
- edoceo 5y agoI wouldn't call one bug, in one software a "vengeance". Yea, it's widly used, and affected lots of mail but was very limited and had a trivial work-around. But, the Y2K issue, back then, was in 1000s of software from 1000s of vendors. The only Y2K affect I remember was the first issue of 2600 in 2000.
- laumars 5y agoIt also wasn’t just the tick over to the year 2000. It was various dates and times around the year 2000 (leap years et al). But the Jan 01 2000 problem was the biggest and most high profile.
- Fatnino 5y agoI remember the laser-tag place near Google (now long gone) would print out your scorecard with a date like 11/24/102
- throwawayboise 5y agoYeah I saw that sort of thing frequently on smaller websites back then as well.
- Pxtl 5y agoIIRC even phpBB would start reporting years as 19100
- withinboredom 5y agoJust imagine if the computing revolution started closer to the beginning of the century instead of the last few decades of it. It would have been cray trying to fix everything.
- dang 5y agoSubmitters: please don't editorialize titles. This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html. We've reverted the title now. (Submitted title was "Y2K problem came back with vengeance in 2022".)
- indigodaddy 5y ago
- DoctorOW 5y agoDo the mods do that?
- pc86 5y agoI don't think it's particularly widespread, and if anyone can do it it makes sense for mods to have that power. However I have on numerous times (in absolute terms) seen mods change titles to something more informative than the original. I wouldn't necessarily call that "editorializing," though.
- hn_throwaway_99 5y agoThere are mods, plural? I.e. someone else besides dang?
- AussieWog93 5y ago"dang" is not a single person a cybernetic entity - hundreds of human beings being worked to death in an underground facility against their will, all mediated by the most sophisticated AI known to man.
- Thorrez 5y agoAccording to this there are 2: dang and sctb. https://news.ycombinator.com/item?id=14283202 https://news.ycombinator.com/item?id=14283202
- ocdtrekkie 5y agoI love the solution: They changed their antimalware definition files to December 33rd, 2021, until they have a more substantial patch ready. The instructions are just to basically clear out the existing files and re-download to get rid of that pesky 2022 year.
- axjmc 5y agoPerhaps switch to unsigned int until a sane solution is implemented. Should give them a couple thousand years to work with.
- Fatnino 5y agoNo, it just kicks the can down the road another 22 years.
- axjmc 5y agoAt first glance thought they were using yyyy format, it's actually only yy. You are correct.
- ocdtrekkie 5y agoIt's pretty likely they'll change the data type somehow, but I guess the question is how fast they can test how that interacts with everywhere that might encounter that variable. Presumably the hack they implemented here buys them at least a month or two to test a real code change.
- cpeterso 5y agoSummary of the bug: Exchange tried to store "2201010001" (i.e. date time '22-01-01 00:00) in a 32-bit signed int, but INT_MAX is 2147483647 so the result is a negative number.
- sdoering 5y agoDamn. This reminds me of the alleged 'Nuclear Gandhi' [1]. Or the Swiss railway axle counter [2]. [1]: https://en.wikipedia.org/wiki/Nuclear_Gandhi https://en.wikipedia.org/wiki/Nuclear_Gandhi [2]: https://www.reddit.com/r/programming/comments/4sco75/the_axle_count_of_trains_in_switzerland_must_not/ https://www.reddit.com/r/programming/comments/4sco75/the_axl...
- alephxyz 5y agoThat reddit post is a link to a Twitter post containing a link to an imgur image containing the relevant text
- PostThisTooFast 5y ago
- manachar 5y agoThe future is weird.
- 29athrowaway 5y agoOr std::numeric_limits<int>::max()
- phkahler 5y agoOh they should have used an UNSIGNED number. /sarcasm
- jlv2 5y agoI'm sure that was the patch.
- mproud 5y agoSo many spelling mistakes!
- zinekeller 5y agoOriginally in German, this is a best-effort translation by the author.
- MauranKilom 5y agoWow, what a bug. The malware scanning service of MS Exchange crashes, because it treats a yyMMddHHmm timestamp as a signed integer when verifying a signature file. Turns out that 2201010001 is negative when treated as a 32 bit integer (the greatest positive one is 2147483647, and 2021 had fewer than 47 months). I can only assume that somebody wrote that "timestamp string as integer" code, checked that it worked correctly (at the time) and then just assumed they must be good on data type range.
- deleted 5y ago[deleted]
- crehn 5y agoSurprised this sort of hack would even pass code review.
- hn_throwaway_99 5y agoReally? Billions of lines of code is reviewed every year at Microsoft. Not hard for me to imagine that this was coerced to an int somewhere (perhaps even unintentionally) that was non-obvious during a code review.
- PostThisTooFast 5y ago
- jeroenhd 5y agoI think it probably developed the other way around, same as the version information almost every DNS server uses. The anti malware releases probably started out with definition version 1, 2, 3, and so on, until someone complained that version 37363 doesn't mean much. Some smart guy probably realised that as long as the numbers increased, they could put anything in there, so the 1201010000 version was born, incremented a day at a time. As MXToolbox states: > The serial number is an unsigned 32 bit value assigned to your SOA record must be between 1 and 4294967295. […] It has become common to set your serial number with a date format to make it easier to to manage. This format uses 10 digits to represent the date and then a two digit sequence number with the format of YYYYmmddss. Do any SOA query against hosted DNS and you'll probably see exactly this pattern appear. The fact they used a long go store the version also suggested that someone may not have realised that longs are the same size as its, depending on your platform and compiler. The effective distinction between the two in Microsoft's compiler ended in the 32 bit era, so using a long to store anything might be seen as a red flag. Based on the fix linked, the current antimalware release seems to be based on December 33rd, 2021. That will buy them enough time to write a patch that just uses long longs for these timestamps, or if their customers don't like installing updates they might just go back to incrementing the number by one at some point.
- greenyoda 5y agoThis bug was extensively discussed yesterday: Microsoft Exchange stops passing mail due to bug on 1/1/22 (677 points / 355 comments) https://news.ycombinator.com/item?id=29756714 https://news.ycombinator.com/item?id=29756714
- manigandham 5y agoIt's surprising how often this issue (ceiling for 32-bit ints) comes up way. If you're still using integers (for ids, timestamps, etc) then just go with 64-bits. It avoids any potential problem with data size in the future, and if the size stays small then it won't matter anyway. Storage is cheap and CPU cache lines are 64-bit now too.
- cube00 5y agoThis will always be a challenge for applications that start their lives as 32-bit (Exchange wasn't 64-bit until 2007)
- cm2187 5y agoThought this had to do with memory addressing/pointer size, not data types.
- dkbrk 5y agoCache lines are 64 bytes not bits. That is: a cache line can store 8 64-bit numbers.
- manigandham 5y agoAh you're right, I was thinking about 64-bit alignment for atomic operations. Either way none of this matters except under the most extreme performance conditions.
- cube00 5y agoI'm pleased the error message actually mentioned exactly what's wrong. Compare the two snippets in [1]: > Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long. > Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error. I hope it's trend that continues because when it's going pear shaped every little morsel of information is important to narrow down the problem. I'm less pleased they seemly didn't deploy this update to an internal on-prem test Exchange server before a wider release. [1]: https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-exchange-on-premises-transport-queues/ba-p/3049447 https://techcommunity.microsoft.com/t5/exchange-team-blog/em...
- irrational 5y ago> when it's going pear shaped I think this means “when it’s going bad”, but what does “pear shaped” mean? And why a pear?
- FooHentai 5y agoIt originated in the British Royal Air Force where an imperfectly-executed loop-de-loop would go 'pear shaped' instead of round as intended. Interestingly it then played off the British tendency for understatement so that something that meant 'a bit wonky' slid into meaning 'off the rails busted'.
- mizzao 5y ago> I'm less pleased they seemly didn't deploy this update Is there actually an update here, or just some code that was probably sitting around for years until 2022 rolled around?
- aoetalks 5y agoAFAIK it was a signature update, and the existing code couldn't understand the new signatures with a 2022 timestamp. IMHO, signatures still should have been deployed to an early ring before broad release. Antimalware signatures as code, anyone?
- bawolff 5y agoMaybe this will kill the last remenants of people thinking they are using spf when they are actually using senderid
- whatever1 5y agoPlease, D. Knuth just solve this dates for computers problem once and for all!
- SavantIdiot 5y agoCan you imagine being a dev on this bugfix? The bug is known, but every second wasted testing, regressing, and preparing to deploy, literally millions of emails aren't delivered. That's some serious pressure. I wonder if there were detectable drops in internet traffic due to fewer emails?
- technion 5y agoHere we have a person not running Exchange, seeing the impact on their own outbound delayed queues: https://twitter.com/taupehat/status/1477367798326120458 https://twitter.com/taupehat/status/1477367798326120458
- deleted 5y ago[deleted]
- jlv2 5y agoLong ago I learned a lesson: Don't write your own date time handling code. (because you will always miss something)
- Ueland 5y agoRelevant Tom Scott video: https://www.youtube.com/watch?v=-5wpm-gesOY https://www.youtube.com/watch?v=-5wpm-gesOY
- ghostly_s 5y agoI'm rather surprised the test suite for a product like Exchange wouldn't include setting date types to ($TODAY..$FAR_FUTURE).
- withinboredom 5y agoWe have some test suites that run with a date two weeks in the future. The last two weeks of every year always seems to be fixing these kinds of bugs! (Luckily it’s a web app and most people refresh the page after a few days)
- natch 5y agoSite has this confusing update: > Addendum: A fix ist available. Are we speaking German now? Or is that a typo for isn’t? Or is, maybe?
- K5EiS 5y agoAny reason they didn't make the int unsigned from the get go? Or would that also cause issues?
- baq 5y agothe intern who wrote the code didn't think of the future farther than the weekend party and the senior who reviewed it 10 years ago didn't catch it and doesn't work there anymore.