6 ms·

Confirmed: Windows Azure downtime caused by leap-day bug

103 points by panarky 15y ago

wavephorm 15y ago
I thought this sounded familiar: http://www.computerworld.com/s/article/9124638/Zune_chokes_on_leap_year_bug http://www.computerworld.com/s/article/9124638/Zune_chokes_o...
- panarky 15y ago
  The Zune leap-year bug bricked the players on December 31. Money quote: "Microsoft says it will issue a bug fix for the device so that this problem won't occur again in 2012, the next leap year."
  - tazzy531 15y ago
    Maybe that's why they cancelled the Zune in 2011 to avoid having to fix the leap year bug.
paulbaumgart 15y ago
The last line is a little unfortunate. It made me think of: http://37signals.com/svn/posts/1528-the-bullshit-of-outage-language http://37signals.com/svn/posts/1528-the-bullshit-of-outage-l...
- panarky 15y ago
  Contrast Microsoft's wishy washy "any inconvenience this may have caused" with Amazon's sincere apology. http://aws.amazon.com/message/65648/ http://aws.amazon.com/message/65648/ (Scroll to last paragraph.)
  - Mythbusters 15y ago
    I was hoping to hear a little more technical apology from Microsoft. For a suit, reading what the VP posted might make sense, but for a techie like myself I really need to know that the platform is mature and stable. A detailed description of the problem helps me judge.
    - screwt 15y ago
      To be fair, the post does say they'll provide the technical explanation when they have more info: We will post an update on this situation, including details on the root cause analysis at the end of this incident. I think that's fair enough (assuming they don't just quietly forget about this last part).
  - 4ad 15y ago
    Sincere apology? You just told us to scroll to the last paragraph, presumably because most people who clicked your link would not have seen it. That's not a sincere apology, that's a huge wall of text that could have been about five times shorter. It's PR, it's always PR.
    - abraham 15y ago
      It isn't PR, it is a thorough technical rundown of the issue and what they are doing to improve it.
      - 4ad 15y ago
        It's verbose, not thorough, and it's light on the technical side. It only superficially looks technical to people who don't work in the field. Everything is PR. Even if this was indeed a very thorough and technical rundown of the issue, it would still exist only because of PR.
- lysium 15y ago
  At least they didn't say "... may have caused".
xelfer 15y ago
How are these things happening? in Australia our Healthcare system had a bug too: http://www.itnews.com.au/News/292081,hicaps-bug-hits-health-payment-system.aspx http://www.itnews.com.au/News/292081,hicaps-bug-hits-health-... I don't recall any of this occurring in 2008 or 2004.
- RandallBrown 15y ago
  This happens all the time because of how confusing our calendar is. Lots of dates are figured out by counting the number of seconds since 1970. It's really easy to miss a little detail like a leap year, since they happen so infrequently and they didn't always exist. It's not just Microsoft that does stuff like this either. Apple regularly messes up iPhone alarms during daylight savings.
  - crcastle 15y ago
    Divisible by 4 => leap year Also divisible by 100 => not leap year unless also divisible by 400. It's really not that complicated. [edit: Oops. I messed it up. Irony. Fixed now.]
    - yuliyp 15y ago
      Also divisible by 100 => not leap year unless also divisible by 400. Complicated enough to mess up.
    - nkassis 15y ago
      Check out wikipedia's page I just noticed they have the algorithm spelled out in pseudocode ;p http://en.wikipedia.org/wiki/Leap_year http://en.wikipedia.org/wiki/Leap_year What baffles me is did they really have to rewrite that again somewhere else? Don't they have libraries in whatever language they are using with something to do the date calculations they need?
    - notatoad 15y ago
      why is anybody even doing these calculations? import an existing datetime library and use it. these problems have been solved, and tested in previous leap years. what's the point in re-solving it?
nandemo 15y ago
I don't understand. Your software works for months with 28 or 30 or 31 days. Why does it break for months with 29 days? I think unless you're messing with non-Gregorian calendars, this is a solved problem. Am I missing something?
- keypusher 15y ago
  One of the many issues I saw today was around converting times to and from different formats. Let's say you have a timestamp in epoch, and you want to calculate a year long offset. You add seconds_in_year to your epoch time,, and call your date conversion library to get some kind of iso date (dd-mm-yyyy). Yes, it's wrong because seconds_in_year is not a constant, but this stuff sneaks its way into a large codebase and nobody realizes because all the tests pass until it's leap day. This is only the beginning, there are so many possible issues related to an edge case like this, while it might be a "solved problem" that doesn't mean it's something developers are thinking about every day. And don't even get me started on daylight savings time...
  - RandallBrown 15y ago
    yeah throw in daylight savings time and time zones and it becomes almost unmanageable.
- tazzy531 15y ago
  Time in distributed cloud computing gets very complicated and very critical. Google has an interesting post on how they handled the Leap Second in 2008: http://googleblog.blogspot.com/2011/09/time-technology-and-leaping-seconds.html http://googleblog.blogspot.com/2011/09/time-technology-and-l...
- lysium 15y ago
  Although time is an important data in most applications, there is only poor library support for it. Most languages don't even have a data type for time! For example, the system call gettimeofday(2) does only return the seconds since Jan 1, 1970 (the 'epoch'), the time zone and the daylight saving time correction. No day, no month, no year. For this, you have to call some other function, eg. libc' localtime(3). libc's time(3) function says it returns the number of seconds since the epoch, but it ignores leap seconds, so it actually does not return the number of seconds since the epoch (since there have been several leap seconds since the epoch). Even if you use localtime(3) to get the actual wall clock hour and minutes, you are left on your own from there. Want to have a time point two hours from now? Do your own math, but watch out for the end of the day, which might also be the end of the month and/or the end of the year. One month from now? Do your own math, but watch out for months that have less days than your current month. You may want to resort to do calculations only in seconds since the epoch, but how many seconds are in a month? Depends on the month (and the year as we've just learned!). In a year? Depends on the year (Is it a leap year? Was / will be there a leap second? Do you have to care about leap seconds?). I just picked the C language because it is so prevalent. Other languages have their own issues or inherit them from C. In Python, for example, there is a timedelta object, but it can only handle days, seconds, and microseconds, so you still cannot calculate the date one month from now or one year ago. I find it unbelievably funny that it's 2012 and we still have to deal with this 'solved problem'. Turns out, it is not solved at all.
slardat01 15y ago
Another strange time related issue that just burned us, if your server is up for 497 days, it will stop closing sockets: http://support.microsoft.com/kb/2553549 http://support.microsoft.com/kb/2553549
- InclinedPlane 15y ago
  On a hunch I converted 497 days to seconds, and it works out to be 42.9 million. A suspiciously familiar number, as it is precisely 2^32 hundredths of a second. Since 10 ms is a common clock resolution on systems that points to an obvious cause: a 32-bit counter for time rolling over and horfing the relative age calculations, so all of the sockets that were open prior to the rollover stay open forever.
  - rjh29 15y ago
    This is very much a Windows issue however, as other operating systems have higher resolution TCP timestamps, e.g. 1ms on Linux which rolls over every 49.7 days, and yet they do not have issues with closing sockets.
  - dave1010uk 15y ago
    Windows 95 crashed after 49.7 days (2^32 ms) for similar reasons: http://news.cnet.com/2100-1040-222391.html http://news.cnet.com/2100-1040-222391.html
    - InclinedPlane 15y ago
      There are two things which are a bit off-putting about this. First, the fact that the same exact type of bug had been known in 1999 and yet they either failed to fix it in the newer code base or they reimplemented the exact same bug in new code. Second, almost certainly the reason that these bugs weren't caught earlier is because it's unusual for Windows to have such long uptime (50 days for Win 9x is impressive, and over a year for Windows server equally so). More so, almost certainly the average user has such low expectations of windows reliability that if they see the system become unstable or slow after a long period of uptime they will as a rule merely reboot the system rather than investigate. Edit: a thought occurs to me. Perhaps the "fix" for the older problem was to simply change from using milliseconds since last boot for tcp/ip socket age to using hundredths of a second. I really, really hope that wasn't the case.
mrb 15y ago
Is there any study comparing the downtime of the different cloud platforms over, say, the past few years? EC2, Azure, Google Apps, etc. That would be the ultimate tool to shame substandard cloud vendors...
- obtu 15y ago
  CloudHarmony collects these, here are stats for the past year: https://cloudharmony.com/status https://cloudharmony.com/status
  - obtu 15y ago
    Correction: it doesn't seem to show the recent Azure outage, which would be on Azure Compute regions.
pom 15y ago
This sort of thing happens every leap year, and everytime I remember that the very first exercise in the very first CS class that I took in college was to write an algorithm to decide whether or not a given year was a leap year.
weavejester 15y ago
They should really be running some test servers with clocks set ahead of time in order to get advanced notice of problems like this. I seem to recall that Amazon does this with its servers.
rachelbythebay 15y ago
They noticed it at 5:45 PM PST? $ date -d "Feb 28 2012 17:45 PST" -u Wed Feb 29 01:45:00 UTC 2012 Does that mean it took them nearly two hours to spot the problem? Or are they not running on UTC?

103 points by panarky 15y ago

I thought this sounded familiar: http://www.computerworld.com/s/article/9124638/Zune_chokes_on_leap_year_bug http://www.computerworld.com/s/article/9124638/Zune_chokes_o...

panarky 15y ago
The Zune leap-year bug bricked the players on December 31. Money quote: "Microsoft says it will issue a bug fix for the device so that this problem won't occur again in 2012, the next leap year."
- tazzy531 15y ago
  Maybe that's why they cancelled the Zune in 2011 to avoid having to fix the leap year bug.

paulbaumgart 15y ago

The last line is a little unfortunate. It made me think of: http://37signals.com/svn/posts/1528-the-bullshit-of-outage-language http://37signals.com/svn/posts/1528-the-bullshit-of-outage-l...

panarky 15y ago
Contrast Microsoft's wishy washy "any inconvenience this may have caused" with Amazon's sincere apology. http://aws.amazon.com/message/65648/ http://aws.amazon.com/message/65648/ (Scroll to last paragraph.)
- Mythbusters 15y ago
  I was hoping to hear a little more technical apology from Microsoft. For a suit, reading what the VP posted might make sense, but for a techie like myself I really need to know that the platform is mature and stable. A detailed description of the problem helps me judge.
  - screwt 15y ago
    To be fair, the post does say they'll provide the technical explanation when they have more info: We will post an update on this situation, including details on the root cause analysis at the end of this incident. I think that's fair enough (assuming they don't just quietly forget about this last part).
- 4ad 15y ago
  Sincere apology? You just told us to scroll to the last paragraph, presumably because most people who clicked your link would not have seen it. That's not a sincere apology, that's a huge wall of text that could have been about five times shorter. It's PR, it's always PR.
  - abraham 15y ago
    It isn't PR, it is a thorough technical rundown of the issue and what they are doing to improve it.
    - 4ad 15y ago
      It's verbose, not thorough, and it's light on the technical side. It only superficially looks technical to people who don't work in the field. Everything is PR. Even if this was indeed a very thorough and technical rundown of the issue, it would still exist only because of PR.
lysium 15y ago
At least they didn't say "... may have caused".

xelfer 15y ago

How are these things happening? in Australia our Healthcare system had a bug too: http://www.itnews.com.au/News/292081,hicaps-bug-hits-health-payment-system.aspx http://www.itnews.com.au/News/292081,hicaps-bug-hits-health-... I don't recall any of this occurring in 2008 or 2004.

RandallBrown 15y ago
This happens all the time because of how confusing our calendar is. Lots of dates are figured out by counting the number of seconds since 1970. It's really easy to miss a little detail like a leap year, since they happen so infrequently and they didn't always exist. It's not just Microsoft that does stuff like this either. Apple regularly messes up iPhone alarms during daylight savings.
- crcastle 15y ago
  Divisible by 4 => leap year Also divisible by 100 => not leap year unless also divisible by 400. It's really not that complicated. [edit: Oops. I messed it up. Irony. Fixed now.]
  - yuliyp 15y ago
    Also divisible by 100 => not leap year unless also divisible by 400. Complicated enough to mess up.
  - nkassis 15y ago
    Check out wikipedia's page I just noticed they have the algorithm spelled out in pseudocode ;p http://en.wikipedia.org/wiki/Leap_year http://en.wikipedia.org/wiki/Leap_year What baffles me is did they really have to rewrite that again somewhere else? Don't they have libraries in whatever language they are using with something to do the date calculations they need?
  - notatoad 15y ago
    why is anybody even doing these calculations? import an existing datetime library and use it. these problems have been solved, and tested in previous leap years. what's the point in re-solving it?

nandemo 15y ago

I don't understand. Your software works for months with 28 or 30 or 31 days. Why does it break for months with 29 days? I think unless you're messing with non-Gregorian calendars, this is a solved problem. Am I missing something?

keypusher 15y ago
One of the many issues I saw today was around converting times to and from different formats. Let's say you have a timestamp in epoch, and you want to calculate a year long offset. You add seconds_in_year to your epoch time,, and call your date conversion library to get some kind of iso date (dd-mm-yyyy). Yes, it's wrong because seconds_in_year is not a constant, but this stuff sneaks its way into a large codebase and nobody realizes because all the tests pass until it's leap day. This is only the beginning, there are so many possible issues related to an edge case like this, while it might be a "solved problem" that doesn't mean it's something developers are thinking about every day. And don't even get me started on daylight savings time...
- RandallBrown 15y ago
  yeah throw in daylight savings time and time zones and it becomes almost unmanageable.
tazzy531 15y ago
Time in distributed cloud computing gets very complicated and very critical. Google has an interesting post on how they handled the Leap Second in 2008: http://googleblog.blogspot.com/2011/09/time-technology-and-leaping-seconds.html http://googleblog.blogspot.com/2011/09/time-technology-and-l...
lysium 15y ago
Although time is an important data in most applications, there is only poor library support for it. Most languages don't even have a data type for time! For example, the system call gettimeofday(2) does only return the seconds since Jan 1, 1970 (the 'epoch'), the time zone and the daylight saving time correction. No day, no month, no year. For this, you have to call some other function, eg. libc' localtime(3). libc's time(3) function says it returns the number of seconds since the epoch, but it ignores leap seconds, so it actually does not return the number of seconds since the epoch (since there have been several leap seconds since the epoch). Even if you use localtime(3) to get the actual wall clock hour and minutes, you are left on your own from there. Want to have a time point two hours from now? Do your own math, but watch out for the end of the day, which might also be the end of the month and/or the end of the year. One month from now? Do your own math, but watch out for months that have less days than your current month. You may want to resort to do calculations only in seconds since the epoch, but how many seconds are in a month? Depends on the month (and the year as we've just learned!). In a year? Depends on the year (Is it a leap year? Was / will be there a leap second? Do you have to care about leap seconds?). I just picked the C language because it is so prevalent. Other languages have their own issues or inherit them from C. In Python, for example, there is a timedelta object, but it can only handle days, seconds, and microseconds, so you still cannot calculate the date one month from now or one year ago. I find it unbelievably funny that it's 2012 and we still have to deal with this 'solved problem'. Turns out, it is not solved at all.

slardat01 15y ago

Another strange time related issue that just burned us, if your server is up for 497 days, it will stop closing sockets: http://support.microsoft.com/kb/2553549 http://support.microsoft.com/kb/2553549

InclinedPlane 15y ago
On a hunch I converted 497 days to seconds, and it works out to be 42.9 million. A suspiciously familiar number, as it is precisely 2^32 hundredths of a second. Since 10 ms is a common clock resolution on systems that points to an obvious cause: a 32-bit counter for time rolling over and horfing the relative age calculations, so all of the sockets that were open prior to the rollover stay open forever.
- rjh29 15y ago
  This is very much a Windows issue however, as other operating systems have higher resolution TCP timestamps, e.g. 1ms on Linux which rolls over every 49.7 days, and yet they do not have issues with closing sockets.
- dave1010uk 15y ago
  Windows 95 crashed after 49.7 days (2^32 ms) for similar reasons: http://news.cnet.com/2100-1040-222391.html http://news.cnet.com/2100-1040-222391.html
  - InclinedPlane 15y ago
    There are two things which are a bit off-putting about this. First, the fact that the same exact type of bug had been known in 1999 and yet they either failed to fix it in the newer code base or they reimplemented the exact same bug in new code. Second, almost certainly the reason that these bugs weren't caught earlier is because it's unusual for Windows to have such long uptime (50 days for Win 9x is impressive, and over a year for Windows server equally so). More so, almost certainly the average user has such low expectations of windows reliability that if they see the system become unstable or slow after a long period of uptime they will as a rule merely reboot the system rather than investigate. Edit: a thought occurs to me. Perhaps the "fix" for the older problem was to simply change from using milliseconds since last boot for tcp/ip socket age to using hundredths of a second. I really, really hope that wasn't the case.

mrb 15y ago

Is there any study comparing the downtime of the different cloud platforms over, say, the past few years? EC2, Azure, Google Apps, etc. That would be the ultimate tool to shame substandard cloud vendors...

obtu 15y ago
CloudHarmony collects these, here are stats for the past year: https://cloudharmony.com/status https://cloudharmony.com/status
- obtu 15y ago
  Correction: it doesn't seem to show the recent Azure outage, which would be on Azure Compute regions.

pom 15y ago

This sort of thing happens every leap year, and everytime I remember that the very first exercise in the very first CS class that I took in college was to write an algorithm to decide whether or not a given year was a leap year.

weavejester 15y ago

They should really be running some test servers with clocks set ahead of time in order to get advanced notice of problems like this. I seem to recall that Amazon does this with its servers.

rachelbythebay 15y ago

They noticed it at 5:45 PM PST? $ date -d "Feb 28 2012 17:45 PST" -u Wed Feb 29 01:45:00 UTC 2012 Does that mean it took them nearly two hours to spot the problem? Or are they not running on UTC?