25 ms·
Microsoft Exchange stops passing mail due to bug on 1/1/22
- tomohawk 5y ago2022 is already a great year!
- alblue 5y agoTL;DR Microsoft released a virus definition file with a timestamp/serial 220101001, and is unable to parse it into a (signed) long value since the largest value is 2,147,483,647 and it’s possible that there’s a factor of 10 in the deserialisation logic.
- ginko 5y agoBut 220,101,001 isn't larger than 2,147,483,647.. Did they get the number slightly wrong?
- cnorthwood 5y agoIt also included a serial number following the date part of 001 (also 002, etc)
- jonny_eh 5y agoClearly this should stay a string. But I suspect they’re converting to an int to compare which definition is newer.
- turminal 5y agoThat would work just fine with a string comparison.
- priansh 5y agoYeah my first thought was, well you could lay out the timestamp in a way that supports lexicographical sorting, and then realized they had already done so. No reason this shouldn’t be a string!
- petee 5y agoIt can be both. DNS serial numbers are YYYYMMDDnn, and it works just fine for humans, it is easy to convert, more compact, and you can compare in a single operation. Internally, there is no reason for it to be a string.
- genocidicbunny 5y agoComparing with strings ain't no panacea either. There's some suspicion that Windows 9 was skipped because a lot of old software checked for Windows 95 or 98 by just doing a string compare against "Windows 9". (For the record, I kind of doubt this was much of a concern by the time Windows '9' was being worked on, but there's a certain logic to it.)
- mark-r 5y agoI think the point is that a 3 digit serial wouldn't be enough to cause overflow, it would need to be 4 digits. But who makes a post like that without using copy/paste to get it exactly right?
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- Cycl0ps 5y agoThere's three 0s before the last 1. That pushes it over the edge of what a long int can handle
- mark-r 5y agoI've gone back to the original just to double check. It's very consistent about having only 2 zeros before the last digit, not 3.
- Flocular 5y agoThe error message " Can’t Convert “2201010001” to long." makes it seem like that was the timestamp/serial used (Would be yymmddHHMM format). No factor 10 in deserialisation needed for an overflow.
- AshamedCaptain 5y ago
- f0e4c2f7 5y agoMaybe we should just leave email off for 2022.
- skofgar 5y agoThis would be a great New Years resolution. No emails.
- bpodgursky 5y agoYou won't like what comes next.
- influx 5y agoMuch prefer e-mail, which is async over messaging which isn't. Also, my e-mail has programmable filters...
- AnIdiotOnTheNet 5y ago
- thanatos519 5y ago... so if they test things, they don't do it in an environment that resembles the world of their victims^Wcustomers?
- Krasnol 5y agoI assumed it's normal now to outsource beta testing to the customer.
- hughrr 5y agoBeing a victim is exactly what I felt like when I was an MSFT customer. I shall remember that one :)
- fragmede 5y agoFor all the hate the cloud gets because of cost/whatever, I'm really glad to have a SaaS provider for email right about now.
- deleted 5y ago[deleted]
- belter 5y agoMany eons ago...Around 10 to 15 years ago...Seen a similar serialization and date processing bug with a Java based Enterprise Integration Platform used by some of the largest companies in the world. Several European Central Banks were unable to process transactions ;-). Can't provide more details, to protect the innocent. Since then, it has been a standard within the Testing group of that vendor to always have a running platform setup with a date 6 months ahead ( clock sync from a different source ). Something I added to my own Software Engineering and Testing standard practices.
- karmakaze 5y agoI came to ask if anyone was aware of systems that automate tests of dates in the future to find it already answered. > Something I added to my own Software Engineering and Testing standard practices. Noted.
- trembonator 5y agoCmon, it’s Microsoft! You shouldn’t need to do that!
- jrootabega 5y agoAgreed. Microsoft should be doing that already.
- hughrr 5y agoIsn’t this basically fuzzing? Making sure it works at the business domain level is another thing entirely.
- matsemann 5y agoNot exactly fuzzing, no. Most tests are written with a fixed date, or now(). Instead, using now()+delta can help discover future bugs and give you some time to remedy the issue. Often annoying to write tests like that, though, without them breaking for unrelated/timing issues compared to a fixed value.
- benttoothpaste 5y agoLong should be a 64 bit type on 64 bit machines, no?
- breakingcups 5y agoSee also: https://news.ycombinator.com/item?id=29757377 https://news.ycombinator.com/item?id=29757377
- technion 5y agoAs far as I can tell, there's currently no official acknowledgement of this issue. It has affected 100% of the Exchange servers I'm overseeing. In every case, a complete halt to mail was corrected by a fix only escribed on Reddit and Twitter.
- drsnow 5y ago
- hprotagonist 5y agome, too! https://techcommunity.microsoft.com/t5/exchange-team-blog/me-too/ba-p/610643 https://techcommunity.microsoft.com/t5/exchange-team-blog/me...
- bazzargh 5y agoThere is one here https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-transport-queues/ba-p/3049447 https://techcommunity.microsoft.com/t5/exchange-team-blog/em... though you'd be hard pressed to recognize it as being this bug, since it doesn't mention the error message that everyone has seen, and the instructions for workaround are not as explicit and detailed as I'd want if I was paged for this.
- YXNjaGVyZWdlbgo 5y agoThat is what happens without QA, people saving timestamps as int.
- Forge36 5y agoHow often does QA adjust time on their machines? Could be an easy test case/setup step for an automated suite
- YXNjaGVyZWdlbgo 5y agoSince the year 2k craze it's actually a rather standard practice in QA, at least on our team, to increment in decade steps up to 2038.
- k8sToGo 5y agoHow do you get to 2038 by increasing by decades only?
- YXNjaGVyZWdlbgo 5y agoYou are absolutely right the last step is most of the time not a decade and in a few year it's even under a decade I hope the QA guys can adept their workflow...
- littlecranky67 5y agokind of offtopic: I recently learned google uses a "Leap Smear" instead of using leap seconds. To avoid errors and outages by having a minute with 61 seconds, they just speed up their time on a single day to match that extra second eventually.
- PascLeRasc 5y agoThat’s really interesting, where did you hear that? I’d like to read more if you have anything
- mikehollinger 5y agohttps://developers.google.com/time/smear https://developers.google.com/time/smear :-)
- ed25519FUUU 5y agoI’d you’re using Google’s public timeservers you’re also using leap second smearing. https://developers.google.com/time/smear https://developers.google.com/time/smear
- db48x 5y agoPhone numbers are not numbers. Zip codes are not numbers. Model numbers are not numbers. Characters are not numbers. Which part of this progression escapes you? Seriously, if you can’t add, subtract, multiply, and divide them, then they aren‘t numbers and you shouldn’t use a numeric data type for them. This is CS 101 knowledge. How badly does MS run their engineering?
- fortran77 5y ago> How badly does MS run their engineering? Bad enough to be a 2.53 Trillion Dollar company.
- hulitu 5y agoJust look at Win 10 and at Teams for examples. The only engineering which counts at MS is the one which maximizes profits.
- Demiurge 5y agoAnd how is this different for everyone else in the free market economy?
- dijit 5y agoI had a weird kind of epiphany when I was watching the fireworks over Sydney last night. The harbour bridge has lights on it. For what financial benefit? The fireworks display itself, I can't understand where the financial benefit is and it costs millions of dollars. ($7 million as far as I'm aware) --- I think that you could make a financial motivation statement for making good software (it makes it less likely for people to switch, for example), but my broader point is: why does everything have to be financially motivated? Because, a lot of stuff that we enjoy doesn't seem to be primarily financial in nature.
- dylan604 5y ago>The harbour bridge has lights on it. >For what financial benefit? Same for the Opera House. Imagine all of those stock photos of Sydney. Now narrow those down to the nighttime shots of Sydney. Imagine those without lights on the bridge or opera house. For that matter, any of the lights on any of the buildings. What do you have left? A really boring photo. Nobody wants to visit a city with really boring photos. What's the financial benefit of that?
- hexo 5y agoWhats the point of using exchange? Why people can't just use regular mail servers? Whats different?
- hahajk 5y agoWhat are other good self-hosted email server options?
- hexo 5y agoI don't know... postfix?
- ananonymoususer 5y agohttps://mailinabox.email/ https://mailinabox.email/
- frays 5y agoI'm more interested in hearing why these Sys Admins haven't switched over to O365 yet?
- analog31 5y agoAre they actually different, as in the nuts and bolts, or just different shells on top of the same infrastructure?
- mymargora 5y ago
- deleted 5y ago[deleted]
- gtsop 5y agoInterested to know if there are other legacy pieces of software in any industry having similar bugs
- puyoxyz 5y agodon’t link old reddit please, anyone who wants to use old reddit already has it set as the default, and for apollo users old reddit links don’t automatically open in apollo like normal reddit.com links
- netsec_burn 5y agoIf I remember correctly, HN automatically converts Reddit links to old.reddit.com (less user hostile).
- tuwtuwtuwtuw 5y agoSounds like it should convert it to i.reddit.com then since it's even less user hostile, at least in my view
- technion 5y agoCan confirm, when I made this thread it didn't have the old. name in the URL.
- tuwtuwtuwtuw 5y agoNot sure what Apollo is, but it sounds like whatever it is it should be fixed instead of you asking people to give some alternate links to a worse version of the website.
- ZeroCool2u 5y agoApollo is a 3rd party Reddit client for iOS. It probably should be fixed in Apollo. I know Sync on Android has no issue handling any of the Reddit sub domains.
- BenjiWiebe 5y agoAnd RedReader on Android also supports old.reddit links.
- iso1210 5y ago
- jakub_g 5y agoSemi-relevant: "Practical time machines" (ironically, written by MS engineer from another department) https://textslashplain.com/2021/10/01/practical-time-machines/ https://textslashplain.com/2021/10/01/practical-time-machine...
- linker3000 5y agoWe shouldn't blame Microsoft for this; I once worked in the veterinary industry, and one Practice Management System allowed you to register a pet including their real or approximate birth date, or by entering an approximate age. If you entered a birth date, the software would calculate the pet's age, and we noticed we had the odd quirk, like a 400 year-old cat etc. I couldn't see an obvious pattern to the anomaly, and it didn't occur often, so I created a support ticket. After a couple of days, I received a response that 'dates were hard' and we should correct the ages manually as needed. So there you have it: Dates are hard. (No, the bug wasn't ever fixed.)
- goda90 5y agoI feel like after Y2K, we should've learned to never store two digit years again. If the format they used was actually YYYYMMDDhhmm, then it would've crashed as soon as someone tried to convert it to an int and they would've fixed it without another thought. Sure dates are hard, but a few simple rules cover a lot of problems.
- OisinMoran 5y agoDo you mean it would crash from that number being too big for a byte-sized int? Just checking I'm not missing anything here.
- bena 5y agoYeah, they converted the DateTime object into a string of the format YYMMddhhmm which for 2022 starts with 2201010000. Then they convert that into a signed long integer or 2,201,010,000 A long integer is 32 bits. Since it's signed, one of those bits are used for the sign and 31 are used to represent the actual number. With 31 bits, you can express numbers from 0 - 2^31 or 2,147,483,648. 2,201,010,000 is larger than 2,147,483,648. If they used the format YYYYMMddhhmm, then their numbers for every year would have overflowed the field. Because 201,801,010,000, for example, is too bit to fit in the long. Every year past year 99 would be too big. But, if it's, say 2008 and you realize that putting the thousand and hundreds will always make it overflow, 801,010,000 would not make it overflow. So why not just cut off thousands and hundreds? Because it's not like we'll run into a 1999/2000 problem before the software is replaced.
- raintrees 5y agoSet-MalwareFilteringServer -BypassFiltering $True -identity <server name> Then restart Microsoft Exchange Transport service. New year: Building a postfix-dovecot server.
- kizer 5y agoSeems like many bugs occur when programmers try to be clever unnecessarily. In this day and age, we don't really have to do tricks to save memory or time in application programs; it's just not worth it with modern silicon and compilers/runtimes. So many bugs could be avoided if people valued robust functionality with maybe a "boring" implementation over trivial applications of cleverness etc.
- zbentley 5y agoTime to trot this out again: https://infiniteundo.com/post/25326999628/falsehoods-programmers-believe-about-time https://infiniteundo.com/post/25326999628/falsehoods-program...
- tdiff 5y agoSo they are not checking security definition updates on some test systems before releasing? This is ridiculous.
- Scarblac 5y agoThe old joke "Every program grows until it can send email, except for Microsoft Exchange" is still true on 1/1/22.
- mitchs 5y agoI'm curious, what everyone considers far enough in the future for your code. Eg. are you willing to write code that puts nanoseconds in a 64 but number? That is "only" 584 years after epoch. Would you consider that ok?
- vlmutolo 5y agoWe're currently dealing with problems that arose from people assuming that their systems wouldn't ever be used for more than, say, 50 years. That assumption is frequently proven wrong. 500 years is pretty different. Odds are that computing will look nothing like what we know today. Our software will be long dead. And if it isn't, there will be yet another Y2K bug in 500 years' time, and they'll solve it. Just like we keep solving it.
- userbinator 5y agoDid anyone notice this anomaly? 220101001 2147483647 Those don't line up. It seems like it's converting a larger number (10x?), but the error message mentions the above string.
- neycoda 5y agoThis is just evidence we're in Year Zero.
- Taniwha 5y agoSurely one of the lessons here is that the big push to have languages detect arithmetic overflow are largely moot unless you actually do something useful when it happens (because here's a case where yes they did detect it but ....)
- tluyben2 5y agoAaaa. Exchange. In another universe no one would be using it. We did a project for a large university and even MS could not make Exchange run stable (for a few 100k people); they had all servers reboot at 3 am or they would crash, lose data etc. It is old school horrible as MS used to create; they improved much over the decades but I guess they did not merge that mindset in the Exchange (or SharePoint) teams.
- frupert52 5y agoThat's because they don't want to encourage the use of on-prem
- deleted 5y ago[deleted]
- mark-r 5y agoInstead of switching to unsigned 32 bit int to buy another 21 years, they should switch to 64 bit ints. Or even 64 bit double - inside every double is a 53 bit int hiding in plain sight! That would buy them another 90073970 years.