18 ms·

Glibc is still not Y2038 compliant by default

325 points by chmaynard 5y ago

phendrenad2 5y ago
There are a lot of people, even in this very thread, who will say "don't break backward compatibility!" right up until 03:14:07 UTC on 19 January 2038. At SOME point we'll have to let saner people prevail, and actually break backward compatibility. I'm just hoping it happens, you know, sooner rather than later.
- jrimbault 5y ago
  I'm exactly of this opinion, it's better for us (programmers/admins/etc) to deal with some level of breakage now (gigantic as it may be), than for actual users to deal with breakage with their future dates.
  - paledot 5y ago
    Bingo. We have the option of recompiling with an older version of glibc while we work out the bugs. Our users won't have the option of running our code in 2037 when the epoch arrives.
kazinator 5y ago
> This can be observed with the large file support extension, which is needed to handle files larger than 2GiB, you must always build your code with -D_FILE_OFFSET_BITS=64. And, similarly, if you’re on a 32-bit system, and you do not build your app with -D_TIME_BITS=64, it will not be built using an ABI that is Y2038-compliant. _TIME_BITS=64 is not working for me on an Ubuntu 18 system based on glibc 2.27 (three plus years old), and I see nothing in the header files that switches time_t. This must be something new? _FILE_OFFSET_BITS is old, on the other hand. Edit: I see in the git log that this has 2021 all over it: https://sourceware.org/git/?p=glibc.git&a=search&h=HEAD&st=commit&s=_TIME_BITS https://sourceware.org/git/?p=glibc.git&a=search&h=HEAD&st=c... Anyway, it's too early to see if this is "bad". The unknown quantity is the behavior of distro people. Distro people have the ability to override this default, such that all the packages have 64 bit time_t, and the toolchain is configured to build that. I have some faith in distro people.
yjftsjthsd-h 5y ago
> Consider libraries: what if a dependency is built with -D_TIME_BITS=64, and another dependency is built without, and they need to exchange struct timespec or similar with each other? What happens in the reverse case on Alpine (or anything using the other approach)? If I build a new program but link against a dependency that predates the switch (say, I upgraded my workstation to a new Alpine release but didn't `make clean`), will I get the same breakage?
- mlyle 5y ago
  > If I build a new program but link against a dependency that predates the switch (say, I upgraded my workstation to a new Alpine release but didn't `make clean`), will I get the same breakage? If you jump between incompatible versions of a C library without rebuilding, you can expect nothing to work.
  - DSMan195276 5y ago
    > If you jump between incompatible versions of a C library without rebuilding, you can expect nothing to work. Is there a reason musl introduced this in a non-major version? Your point here is perfectly valid, but to me the next obvious question is "what qualifies as an incompatible version?", and after doing some Googling I'm surprised it's not very clear to me. I would have assumed that going from 1.1.X to 1.2.0 was compatible, or IE. two programs compiled against 1.1.0 and 1.2.0 will work fine together, but clearly that's not actually the case.
    - formerly_proven 5y ago
      musl created new symbols for all time_t related functions. This means that an 32-bit time_t application can link against any version, because the 32-bit time_t symbols did not change. The problem is when code passes around time_t to non-musl code - you can't mix 32-bit and 64-bit time_t then, because those are different types. Source: https://musl.libc.org/time64.html https://musl.libc.org/time64.html
      - 5y ago
jmercouris 5y ago
I think glibc is right to not change the defaults out from under the feet of users. “ This is the worst possible way to do this. Consider libraries: what if a dependency is built with -D_TIME_BITS=64, and another dependency is built without, and they need to exchange struct timespec or similar with each other? ” - this argument makes no sense to me. If you need new features and new values update your flags rather than breaking any program that depends on your CLI API
- mlyle 5y ago
  > If you need new features and new values update your flags rather than breaking any program that depends on your CLI API The thing is, 2038 is less and less far away with each day that goes by. Especially since real programs often have to use future timevalues (and often further in the future than one would think). Stuff has to change at some point. A glibc major version which changes the structures and typedefs for all 32 bit code could handle it. But as it stands now, someone who wants to make a 32 bit program y2038-compliant will have a hard time doing it: they can't safely hand time values to any library that is possibly compiled differently. This punts it to whom, distro maintainers? They have to try and rebuild everything with the correct define? And the lone end-user who does gcc -o myfile myfile.c gets code that's broken against the system libraries? Or they patch glibc itself and ship a glibc that's ABI incompatible with everyone else. Meh.
- mongol 5y ago
  It is only changed from under their feet if it is done without communication. Most important is to communicate a plan. Tell the users that at from date X, all new releases will have the default changed.
- GoblinSlayer 5y ago
  >I think glibc is right to not change the defaults out from under the feet of users. Either glibc will do it, or time itself will do it. But when time will do it, it will be much harder breakage.
- throw0101a 5y ago
  The current version is 2.34: * https://www.gnu.org/software/libc/ https://www.gnu.org/software/libc/ Change the default and release "3.0" with the time_t alteration at the top of the ChangeLog. 2.x can perhaps live for a while for some overlap. The distros will eventually pick it up, and all code compiled/released for a particular distro version will deal with it in course.
tick_tock_tick 5y ago
> It should be noted that both approaches described above introduce zero friction to get the right thing. If your code is called by any other code and took in a time_t recompiling would have broken your ABI.
- mlyle 5y ago
  Sure. And the ABI needs to break sometime before 2038. So, new major libc version, and... I mean, waiting does mean there will be fewer 32 bit systems: so there's an argument to put off the transition to lower the scope of work. However, most of the ones that stay around will be embedded and long-lived: just the worst kind of thing to bite us in the future.
  - mrweasel 5y ago
    > Sure. And the ABI needs to break sometime before 2038. That really highlights the benefit of the BSD model, where the OS, libc and all the base packages are shipped as a complete system. OpenBSD switch to 64 bit time_t on all supported platforms back in 2014. We all talked about Y2038 back then, and most agreed that something should be now as quickly as possible. Then the world just sort of forgot. I mean it is solved, but if you're a new developer, you might not know that you need to do something special. The GNU and Linux world is really good at not breaking ABI compatibility and mostly that's great, but in this case it's a problem and pushing it much further is going to be a problem. It's also going to be weird if the default never change, then we just have a C library that that just keep producing defective programs, unless you add certain flags.
    - rjsw 5y ago
      OpenBSD was the slowest of the major BSD systems to switch. NetBSD made the change in 2009, it also preserves backward compatibility.
      - mrweasel 5y ago
        I did not know that. The OpenBSD change just stuck in my head somehow. Any idea how NetBSD fixed the backward compatibility?
dveeden2 5y ago
I think the best way of doing this is a 4 step approach: 1. Old situation, time_t is 32-bit 2. Migration starts, time_t is 32-bit by default, -D_TIME_BITS=64 to get 64-bits 3. Migration continues, time_t is 64-bit by default, -D_TIME_BITS=32 to get 32-bits 4. New situation, time_t is 64-bit And with a long time between these steps. So glibc is at step 2 while musl skipped step 2 and is at step 3. There should be plenty of time between steps. It might make sense to keep it at step 3 until 2037 or so for very restrictive embedded plaforms etc.
- mlyle 5y ago
  Is phase 2 that useful? Why would you -D_TIME_BITS to something other than the default? It's a hard thing to do safely, since you're choosing to be ABI incompatible (in a way that's not caught at compile or even invocation time) with the system's default. And then, all you get is a binary that will potentially handle times past 2038 safely, when the whole rest of the system won't.
  - MichaelBurge 5y ago
    I think it lets developers opt-in to fixing their glibc-dependent programs ahead of time, so there's less work to do on stage 3.
    - mlyle 5y ago
      The thing is, not too much can be expected to break from a 64 bit time_t. Only stuff that directly shoves time_t's over the network or to disk -- a bad practice -- will. Note anything that builds on x86-64 or other 64 bit architectures has already figured out how to tolerate 64 bit time_t's. The major reason it's not safe to build with a 64 bit time_t is because you don't know whether other libraries are. If the C library cuts over, when people/distributions move to the new library version, they know that's not a concern. Defaults change in tooling all the time, requiring code changes for distributions bundling newer tooling/libraries. This one would require less change than most. Of course, it's possible to survive a 64 bit time_t and still not be 2038-safe. But at least you can be correct if you have a C library and other libraries that will tolerate 64 bit time_t's.
ncmncm 5y ago
Just changing time_t to an unsigned int would take us all the way out to 2106. This has seemed to be an unpopular observation, in the past.
- thewakalix 5y ago
  That’s still a breaking change, although somewhat less of a break. Better to just do it right.
- mlyle 5y ago
  Lots of code would break because they assume they can do signed math with time_t, though. It's a less invasive change to make it wider: largely, it's just a recompile, except for code that persists a time_t directly or sends it directly over the network (and both of these should be considered harmful for other reasons). The other problem is that more unsafe code in libraries, etc, will happily cooperate with 2038-safe unsigned time_t code, but will start to do bad things shortly before 2038.
  - anfilt 5y ago
    Honestly, for network code it makes more sense when the data is received to add the time to the current era. A 64 bit timestamp is an extra 4 bytes of overhead. However, the biggest issue with a network protocol is you just can't force everyone to update everything. Now it's some in house protocol sure you can just update everything instead making the timestamp a relative value to the current era. For code running locally 64 bits is less of issue, just mainly a problem of ABI breaks. Honestly, one thing I think people overlook is file-formats... A lot them have 32 bit time fields. Also unlike a network packet the file could actually be from a previous 32 bit era. So those timestamps are ambiguous after 2038.
  - masklinn 5y ago
    > Lots of code would break because they assume they can do signed math with time_t, though. And lots of code would break because they want to talk about pre-1970 dates. I'm definitely guilty of it, as I routinely use the timeline of the Great Emu War (1932-10-02/1932-12-10) to test time-bound features.
howinteresting 5y ago
The solution here is to simply not use dynamic linking for anything outside glibc and openssl. If your program is statically linked, or if each program ships all the dynamic libraries it links against, this compatibility problem disappears.
- mlyle 5y ago
  Static linking doesn't fix it. You still need to know how every library you use was built, and choose the same setting.
  - howinteresting 5y ago
    I'm assuming that you're also building all your dependent libraries as part of your build process, as is common with languages like Rust. Closed source libraries are somewhat out of scope when talking about Linux distributions, though you can always ask the vendor to provide you a 64-bit compatible library.
    - WJW 5y ago
      > though you can always ask the vendor to provide you a 64-bit compatible library This is not always true, as the vendor may have disappeared and can no longer provide new versions of the library for any reason. Though, if you knowingly have such a black box in your code and willingly keep running with it until 2038 then you kinda deserve what is coming.
    - mlyle 5y ago
      Then it's not a distinction between dynamic linking and static linking. It's a distinction of "only link against things you yourself build and ship". It's nothing to do with closed or open source. I don't want to build a very big world, and ship a huge amount of binary code that can be found in distributions, to make a 2038-safe app.
      - howinteresting 5y ago
        It's not that big a world. And ststic linking in particular opens up a ton of other opportunities, such as monomorphization.
edflsafoiewq 5y ago
What is the reason a new call like time64_t time64(); wasn't added?
- rwaksmunski 5y ago
  Are you going to rewrite 5 decades worth of legacy software to use it?
  - alkonaut 5y ago
    What’s the alternative to rewriting? Just recompiling? Assuming the code on the receiving end stores the value in 32 bits (and not a time _t which can magically change meaning but a 32 bit integer) then it’s still doomed without rewrite? I mean even with time_t use you can’t just recompile and make it work because there could be subsequent assumptions that the size of a struct containing a time_t will be a specific size or allocations will be too small or misaligned and so on.
    - rwaksmunski 5y ago
      Yes, the alternative to rewriting is changing the size and recompiling. OpenBSD somehow managed to pull it off.
      - alkonaut 5y ago
        But the problem isn't really changing the OS to return a 64bit value where it used to return 32, the problem is all the applications that assumed it would be 32. By "pulling it off" you mean, they changed it and planes didn't crash around them?
        rwaksmunski 5y ago
        Yes, stuff broke in the ports collection. Not sure about planes but at least one monitor exploded.
Subsentient 5y ago
Thank god all my shit is 64-bit now.
- masklinn 5y ago
  If you're on a GNU/Linux system that is not actually relevant unless you've also opted into glibc's 64b time_t, and recompiled everything locally: by default, glibc uses 32b time_t even on 64b machines.
  - AnssiH 5y ago
    time_t has always been 64-bit on 64-bit glibc (https://sourceware.org/glibc/wiki/Y2038ProofnessDesign https://sourceware.org/glibc/wiki/Y2038ProofnessDesign , https://www.gnu.org/software/libc/manual/html_node/Feature-Test-Macros.html https://www.gnu.org/software/libc/manual/html_node/Feature-T...). > Currently it defaults to 64 bits on most architectures. Although it defaults to 32 bits on some traditional architectures (i686, ARM), this is planned to change and applications should not rely on this.
  - petters 5y ago
    This is certainly not true: #include <time.h> #include <stdio.h> int main() { printf("size: %lu\n", sizeof(time_t)); return 0; } prints 8 on my system when compiled with just "gcc test.c"
high_byte 5y ago
added a reminder January 18, 2038 :)
- onion2k 5y ago
  I know this is a joke but it highlights an issue with the 2038 bug - people think it'll be a problem in 2038. It's very badly named. It should be called something like "the future date problem", because it affects any date that gets converted to a timestamp that represents a time after 19 Jan 2038. I wrote code for calculating the payments on 40 year mortgages back in the early 2000s and I had to consider this problem then.
  - high_byte 5y ago
    ah interesting I hadn't realized. but for sure lots of problems will start occurring starting 2038
    - deleted 5y ago
      [deleted]
tgsovlerkhgsel 5y ago
We're 16 years away from the overflow. In other words, some of the systems being put in production right now are going to be around, untouched, unchanged, when it happens. Given this article, it seems we're still doing it wrong, and that means... 2038 will be "fun" (either the remediation, or the consequences of the lack thereof). At least most of us in the industry now have a good retirement plan: Fixing the legacy systems in 16 years...
- jandrese 5y ago
  Ironic that so many people are hoping to retire before they get asked to deal with this mess since so much medical equipment is built on ancient and poorly maintained microcontroller environments.
  - deleted 5y ago
    [deleted]
  - treesknees 5y ago
    Which part is ironic?
    - jandrese 5y ago
      The irony of retiring to avoid the problem only to have it become a problem with the equipment keeping you alive in retirement.
- Animats 5y ago
  We're 16 years away from the overflow. In other words, some of the systems being put in production right now are going to be around, untouched, unchanged, when it happens. Yes. It's getting close. Windows XP, released in 2001, still had a significant presence in 2017, 16 years after introduction.[1] 42% of companies still had some XP systems running back then. 11% of machines were still running it. [1] https://community.spiceworks.com/networking/articles/2719-windows-10-adoption-who-s-using-it-two-years-after-launch https://community.spiceworks.com/networking/articles/2719-wi...
usr1106 5y ago
Isn't the title clickbait? If it read "glibc in 32 bit user space is still not Y2038 compliant by default" I would not have clicked it.
- DarkmSparks 5y ago
  I was feeling the same, about the only thing I have using 32bit userspace these days are a few rPIs, little bit early to be panicking they may suffer from the Y2038 bug, most companies waited to about 6 months before the Y2K bug was going to hit before they started caring.
  - alkonaut 5y ago
    If you spend large amounts of money today, fixing a system that is later decommissioned in 2029 or 2032… you wasted a lot of money. The correct play could very well be to wait to the last second and pay absurd money for scarce experts to fix the systems.
    - DarkmSparks 5y ago
      The correct play is surely to be the scarce expert available at the last second.
  - throw0101a 5y ago
    > I was feeling the same, about the only thing I have using 32bit userspace these days are a few rPIs, little bit early to be panicking they may suffer from the Y2038 bug Quite the opposite if you're talking about rPIs: the small/embedded space is where things can live for a long time. They're the primary place to worrying about. Especially if you're talking about industrial processes.
  - mwcampbell 5y ago
    But if we techies get way ahead of the problem this time, maybe we can prevent it becoming a mass hysteria as Y2K was.
- turminal 5y ago
Asmod4n 5y ago
The problem doesn’t start in 16 years, it’s happening now. Try to do date/time calculation after the overflow today. This can have major issues with ensurance companies right now.
- alkonaut 5y ago
  Right. This is an overlooked aspect. At the precise date of the epochalypse, the date will roll over for the current time_t. But one second before that, it will blow up for a system that peeks 1 second into the future (e.g a control system). One day before, the date will be screwed up for any program doing addition with one day such as a rental system. In the years before 2038, systems that do calculations on year scales will fail, like perhaps insurance or mortgage and similar.
- recov 5y ago
  Yep. Also see https://twitter.com/jxxf/status/1219009308438024200 https://twitter.com/jxxf/status/1219009308438024200
einpoklum 5y ago
If I were to write this post, I would take the time to: Check _why_ glibc maintainers have made this decision, and ask them why it is reasonable for this to be the case right now.
- erk__ 5y ago
  They have a design document for it here: https://sourceware.org/glibc/wiki/Y2038ProofnessDesign https://sourceware.org/glibc/wiki/Y2038ProofnessDesign
OtomotO 5y ago
Wonderful, so I already know what I'll be doing in 2038. If I still live, if civilization is still in place... Keep the paychecks coming! Sorry for my cynicism, it's one coping strategy. I'd also prefer to build reliable software, but alas...
- speedgoose 5y ago
  I don't know, building reliable software is a lot of boring work. Most software the industry develops is not going to space or safety critical, it's okay if they have bugs like this one.
  - OtomotO 5y ago
    The engineer inside me cringes, while the Businessman cheers ;)
getcrunk 5y ago
They can at least throw some compiler warnings.
roeles 5y ago
For work I'm currently building a test-rig that has a projected lifetime of 20 years. So for me, this is very relevant. Thank you for posting.
dathinab 5y ago
Honestly over the years I started associating GNU with: - dedicated but stuck in the past people and approaches - bad or even terrible UX - bad security (either because of bad UX especially bad defaults or because of not consulting crypto experts for crypto stuff)
midjji 5y ago
Even if upgraded, its is so flawed that it is practically universally replaced anyways. There are two severe issues which must be addressed for it to be saved. First, the type should be float, not int. The former has an intuitive precision, while the latter does not. Not to mention the sign issues. It is very rare to see a timestamp int which is not used as a kind of fixed point float somewhere in practice, but doing so manually gives rise to endless bugs. float on the other hand, can always be in second, which is much more intuitive, and means the downstream apps fix if precision changes is much easier to get right. For these reasons, upgrading to float64 would be substantially better than upgrading to int64, while being better for most use cases, almost always as fast, and require less thought in downstream. However, I think we can do even better. Int64 is also too small, as the common nanosecond clocks would only give a century of range, which requires adjusting pretty much every algorithm, even if it only ever deals with a few times. And of course, if I dont care about precession, I want the floating point to deal with it for me. It almost always makes sense to use a time precision exceeding the most common clock with which it will be used, and with a range covering the most common use cases. Most regular cpus give nanosecond precision, but then float64 would only give a few years of range exactly, which is a bit too short for many applications which want exact precision, and use the standard 1973 start. The long awaited float128 however, would be sufficient for billions of years at pico precision, while maintaining low memory and compute costs. Notably cheaper than int32 was originally. float128 simply makes for a fantastic timestamp format for almost every application. Its extremely convenient compared to the clunky std::chrono, works in C and interop, and is very simple to use. The only downside is that compiler support, while getting better, remains bad.
- oshiar53-0 5y ago
  Is there something wrong in particular for using built-in fixed-point types for timestamps? Yes, overflow/underflow issues and yada yada, but virtually every type can be mised in its own way.
  - foxfluff 5y ago
    Nothing wrong with fixed point. It's only a problem because people end up hand-rolling their arithmetic (and botching it) instead of using library functions that do it right.
strenholme 5y ago
I test for these things when building open source software. • 64-bit compiles and applications have a 64-bit time_t in Linux. If your distro and binaries are 64-bit, there isn’t a problem. • 32-bit compiles and applications still have a 32-bit time_t in mainstream Linux libraries, so that old binaries still run. The timestamp is a rolling one; once Y2038 is hit, the timestamp will be a negative one, but one which is updated and is off by 136 years. The workaround is code like this (this is real production code in my Lua fork, Lunacy[1]): time_t t; int64_t tt; if (lua_isnoneornil(L, 1)) /* called without args? */ t = time(NULL); /* get current time */ else { // Lunacy only supports getting the current time lua_pushnil(L); return 1; } if(t < -1) { tt = (int64_t)t + 4294967296ULL; } else { tt = (int64_t)t; } if (t == (time_t)(-1)) lua_pushnil(L); else lua_pushnumber(L, (lua_Number)tt); return 1; This gives us accurate timestamps until 2106. If post 2106 compatibility is needed, we can add 2 ^ 32 again for timestamps with a positive value once the Y2038 rollover happens. • Legacy 32-bit Windows applications using the Posix compatibility layer are not Y2038 compliant. Once Y2038 hits us, the timestamp will always return -1 (in Windows XP, the behavior was to return a number off by 136 years, but this changed in Windows 10). The workaround is to use Windows’ proprietary API for post-Y2038 timestamps. Again, from Lunacy which has a Win32 port: /* Convert Windows "filetime" in to Lua number */ uint64_t t; FILETIME win_time = { 0, 0 }; GetSystemTimeAsFileTime(&win_time); t = win_time.dwHighDateTime & 0xffffffff; t <<= 32; t |= (win_time.dwLowDateTime & 0xffffffff); t /= 10000000; t -= 11644473600LL; lua_pushnumber(L, (lua_Number)t); return 1; Now, last time I researched this, there wasn’t a “int64_t time_64bit()” style system call in the Linux API so that newly compiled 32-bit binaries can be Y2038 compliant without breaking the ABI by using “time_64bit()” instead of “time()”. This was based on some digging around Stackoverflow just last year, and simple Google searches are still not returning pages saying, in big bold letters “-D_TIME_BITS=64 when building 32-bit apps”. https://sourceware.org/glibc/wiki/Y2038ProofnessDesign https://sourceware.org/glibc/wiki/Y2038ProofnessDesign talks about 64-bit support on 32-bit systems as if it’s a future pipe dream, but apparently -D_TIME_BITS=64 works right now. [1] https://github.com/samboy/lunacy https://github.com/samboy/lunacy [2] https://en.wikipedia.org/wiki/Time_in_Indiana https://en.wikipedia.org/wiki/Time_in_Indiana [3] To ruin a classic sadistic interview question for sys admin roles, Linux these days returns both the modification time and the mostly useless “status change” timestamp. Facebook once decided to not move forward because I said that file timestamp was “modification time” and not “status change”; if Facebook is still asking that question, their knowledge is out of date.
ck2 5y ago
Reminder to self not to fly on a plane in January 2038, maybe a few months before and after just to be sure. Whole lotta unknowing beta-testers going to be used to debug old system.
AnssiH 5y ago
The article says that users will never get 64-bit time_t on 32-bit systems on glibc. However, glibc documentation (https://www.gnu.org/software/libc/manual/html_node/Feature-Test-Macros.html https://www.gnu.org/software/libc/manual/html_node/Feature-T...) says: > If _TIME_BITS is undefined, the bit size of time_t is architecture dependent. Currently it defaults to 64 bits on most architectures. Although it defaults to 32 bits on some traditional architectures (i686, ARM), this is planned to change and applications should not rely on this. So it sounds like glibc plans to change the default, just like musl has done.
tyingq 5y ago
MySql fixed their 2038 issues with UNIX_TIMESTAMP() and FROM_UNIXTIME(), fairly recently: https://bugs.mysql.com/bug.php?id=12654 https://bugs.mysql.com/bug.php?id=12654
TacticalCoder 5y ago
But then Y2K was presented as the end of the world, with supposedly nothing being ready, and the world at large was going to experience catastrophic supply chain failures, people would be dying in hospitals, etc. Then Y2K turned out to be a nothing burger (I'm not saying some systems didn't misbehave left and right but in the grand scheme of things, it was a non event). I'm pretty sure it's going to the same with Y2038.
- GnarfGnarf 5y ago
  Y2K was not a disaster precisely because there was panic and concern, and programmers spent much effort fixing it, and it was fixed due to prioritization of resources & effort. "it's going to the same with Y2038" is like saying we can fold our umbrella in a rainstorm because we haven't gotten wet yet.
  - mbg721 5y ago
    When Y2k happened, it also seemed a little ridiculous to laypeople because they thought, "Surely we don't depend on computers that much," and there was still an institutional memory of doing important business on paper. Nobody is under that illusion now.
999900000999 5y ago
Note to self, retire by then. I sure don't want to deal with this. Although knowing corporate America, that's a problem for whoever is CEO in 2037
Aissen 5y ago
From my experience, the libc is not even the most scary step… It's the first, necessary step, but not sufficient. Converting a code base that has serialized data structures with 32 bits timestamps that are not even time_t (which is a good choice since you want serialization stability), now that makes for complex upgrade paths.

325 points by chmaynard 5y ago

There are a lot of people, even in this very thread, who will say "don't break backward compatibility!" right up until 03:14:07 UTC on 19 January 2038. At SOME point we'll have to let saner people prevail, and actually break backward compatibility. I'm just hoping it happens, you know, sooner rather than later.

jrimbault 5y ago
I'm exactly of this opinion, it's better for us (programmers/admins/etc) to deal with some level of breakage now (gigantic as it may be), than for actual users to deal with breakage with their future dates.
- paledot 5y ago
  Bingo. We have the option of recompiling with an older version of glibc while we work out the bugs. Our users won't have the option of running our code in 2037 when the epoch arrives.

kazinator 5y ago

> This can be observed with the large file support extension, which is needed to handle files larger than 2GiB, you must always build your code with -D_FILE_OFFSET_BITS=64. And, similarly, if you’re on a 32-bit system, and you do not build your app with -D_TIME_BITS=64, it will not be built using an ABI that is Y2038-compliant. _TIME_BITS=64 is not working for me on an Ubuntu 18 system based on glibc 2.27 (three plus years old), and I see nothing in the header files that switches time_t. This must be something new? _FILE_OFFSET_BITS is old, on the other hand. Edit: I see in the git log that this has 2021 all over it: https://sourceware.org/git/?p=glibc.git&a=search&h=HEAD&st=commit&s=_TIME_BITS https://sourceware.org/git/?p=glibc.git&a=search&h=HEAD&st=c... Anyway, it's too early to see if this is "bad". The unknown quantity is the behavior of distro people. Distro people have the ability to override this default, such that all the packages have 64 bit time_t, and the toolchain is configured to build that. I have some faith in distro people.

yjftsjthsd-h 5y ago

> Consider libraries: what if a dependency is built with -D_TIME_BITS=64, and another dependency is built without, and they need to exchange struct timespec or similar with each other? What happens in the reverse case on Alpine (or anything using the other approach)? If I build a new program but link against a dependency that predates the switch (say, I upgraded my workstation to a new Alpine release but didn't `make clean`), will I get the same breakage?

mlyle 5y ago
> If I build a new program but link against a dependency that predates the switch (say, I upgraded my workstation to a new Alpine release but didn't `make clean`), will I get the same breakage? If you jump between incompatible versions of a C library without rebuilding, you can expect nothing to work.
- DSMan195276 5y ago
  > If you jump between incompatible versions of a C library without rebuilding, you can expect nothing to work. Is there a reason musl introduced this in a non-major version? Your point here is perfectly valid, but to me the next obvious question is "what qualifies as an incompatible version?", and after doing some Googling I'm surprised it's not very clear to me. I would have assumed that going from 1.1.X to 1.2.0 was compatible, or IE. two programs compiled against 1.1.0 and 1.2.0 will work fine together, but clearly that's not actually the case.
  - formerly_proven 5y ago
    musl created new symbols for all time_t related functions. This means that an 32-bit time_t application can link against any version, because the 32-bit time_t symbols did not change. The problem is when code passes around time_t to non-musl code - you can't mix 32-bit and 64-bit time_t then, because those are different types. Source: https://musl.libc.org/time64.html https://musl.libc.org/time64.html
    - 5y ago

jmercouris 5y ago

I think glibc is right to not change the defaults out from under the feet of users. “ This is the worst possible way to do this. Consider libraries: what if a dependency is built with -D_TIME_BITS=64, and another dependency is built without, and they need to exchange struct timespec or similar with each other? ” - this argument makes no sense to me. If you need new features and new values update your flags rather than breaking any program that depends on your CLI API

mlyle 5y ago
> If you need new features and new values update your flags rather than breaking any program that depends on your CLI API The thing is, 2038 is less and less far away with each day that goes by. Especially since real programs often have to use future timevalues (and often further in the future than one would think). Stuff has to change at some point. A glibc major version which changes the structures and typedefs for all 32 bit code could handle it. But as it stands now, someone who wants to make a 32 bit program y2038-compliant will have a hard time doing it: they can't safely hand time values to any library that is possibly compiled differently. This punts it to whom, distro maintainers? They have to try and rebuild everything with the correct define? And the lone end-user who does gcc -o myfile myfile.c gets code that's broken against the system libraries? Or they patch glibc itself and ship a glibc that's ABI incompatible with everyone else. Meh.
mongol 5y ago
It is only changed from under their feet if it is done without communication. Most important is to communicate a plan. Tell the users that at from date X, all new releases will have the default changed.
GoblinSlayer 5y ago
>I think glibc is right to not change the defaults out from under the feet of users. Either glibc will do it, or time itself will do it. But when time will do it, it will be much harder breakage.
throw0101a 5y ago
The current version is 2.34: * https://www.gnu.org/software/libc/ https://www.gnu.org/software/libc/ Change the default and release "3.0" with the time_t alteration at the top of the ChangeLog. 2.x can perhaps live for a while for some overlap. The distros will eventually pick it up, and all code compiled/released for a particular distro version will deal with it in course.

tick_tock_tick 5y ago

> It should be noted that both approaches described above introduce zero friction to get the right thing. If your code is called by any other code and took in a time_t recompiling would have broken your ABI.

mlyle 5y ago
Sure. And the ABI needs to break sometime before 2038. So, new major libc version, and... I mean, waiting does mean there will be fewer 32 bit systems: so there's an argument to put off the transition to lower the scope of work. However, most of the ones that stay around will be embedded and long-lived: just the worst kind of thing to bite us in the future.
- mrweasel 5y ago
  > Sure. And the ABI needs to break sometime before 2038. That really highlights the benefit of the BSD model, where the OS, libc and all the base packages are shipped as a complete system. OpenBSD switch to 64 bit time_t on all supported platforms back in 2014. We all talked about Y2038 back then, and most agreed that something should be now as quickly as possible. Then the world just sort of forgot. I mean it is solved, but if you're a new developer, you might not know that you need to do something special. The GNU and Linux world is really good at not breaking ABI compatibility and mostly that's great, but in this case it's a problem and pushing it much further is going to be a problem. It's also going to be weird if the default never change, then we just have a C library that that just keep producing defective programs, unless you add certain flags.
  - rjsw 5y ago
    OpenBSD was the slowest of the major BSD systems to switch. NetBSD made the change in 2009, it also preserves backward compatibility.
    - mrweasel 5y ago
      I did not know that. The OpenBSD change just stuck in my head somehow. Any idea how NetBSD fixed the backward compatibility?

dveeden2 5y ago

I think the best way of doing this is a 4 step approach: 1. Old situation, time_t is 32-bit 2. Migration starts, time_t is 32-bit by default, -D_TIME_BITS=64 to get 64-bits 3. Migration continues, time_t is 64-bit by default, -D_TIME_BITS=32 to get 32-bits 4. New situation, time_t is 64-bit And with a long time between these steps. So glibc is at step 2 while musl skipped step 2 and is at step 3. There should be plenty of time between steps. It might make sense to keep it at step 3 until 2037 or so for very restrictive embedded plaforms etc.

mlyle 5y ago
Is phase 2 that useful? Why would you -D_TIME_BITS to something other than the default? It's a hard thing to do safely, since you're choosing to be ABI incompatible (in a way that's not caught at compile or even invocation time) with the system's default. And then, all you get is a binary that will potentially handle times past 2038 safely, when the whole rest of the system won't.
- MichaelBurge 5y ago
  I think it lets developers opt-in to fixing their glibc-dependent programs ahead of time, so there's less work to do on stage 3.
  - mlyle 5y ago
    The thing is, not too much can be expected to break from a 64 bit time_t. Only stuff that directly shoves time_t's over the network or to disk -- a bad practice -- will. Note anything that builds on x86-64 or other 64 bit architectures has already figured out how to tolerate 64 bit time_t's. The major reason it's not safe to build with a 64 bit time_t is because you don't know whether other libraries are. If the C library cuts over, when people/distributions move to the new library version, they know that's not a concern. Defaults change in tooling all the time, requiring code changes for distributions bundling newer tooling/libraries. This one would require less change than most. Of course, it's possible to survive a 64 bit time_t and still not be 2038-safe. But at least you can be correct if you have a C library and other libraries that will tolerate 64 bit time_t's.

ncmncm 5y ago

Just changing time_t to an unsigned int would take us all the way out to 2106. This has seemed to be an unpopular observation, in the past.

thewakalix 5y ago
That’s still a breaking change, although somewhat less of a break. Better to just do it right.
mlyle 5y ago
Lots of code would break because they assume they can do signed math with time_t, though. It's a less invasive change to make it wider: largely, it's just a recompile, except for code that persists a time_t directly or sends it directly over the network (and both of these should be considered harmful for other reasons). The other problem is that more unsafe code in libraries, etc, will happily cooperate with 2038-safe unsigned time_t code, but will start to do bad things shortly before 2038.
- anfilt 5y ago
  Honestly, for network code it makes more sense when the data is received to add the time to the current era. A 64 bit timestamp is an extra 4 bytes of overhead. However, the biggest issue with a network protocol is you just can't force everyone to update everything. Now it's some in house protocol sure you can just update everything instead making the timestamp a relative value to the current era. For code running locally 64 bits is less of issue, just mainly a problem of ABI breaks. Honestly, one thing I think people overlook is file-formats... A lot them have 32 bit time fields. Also unlike a network packet the file could actually be from a previous 32 bit era. So those timestamps are ambiguous after 2038.
- masklinn 5y ago
  > Lots of code would break because they assume they can do signed math with time_t, though. And lots of code would break because they want to talk about pre-1970 dates. I'm definitely guilty of it, as I routinely use the timeline of the Great Emu War (1932-10-02/1932-12-10) to test time-bound features.

howinteresting 5y ago

The solution here is to simply not use dynamic linking for anything outside glibc and openssl. If your program is statically linked, or if each program ships all the dynamic libraries it links against, this compatibility problem disappears.

mlyle 5y ago
Static linking doesn't fix it. You still need to know how every library you use was built, and choose the same setting.
- howinteresting 5y ago
  I'm assuming that you're also building all your dependent libraries as part of your build process, as is common with languages like Rust. Closed source libraries are somewhat out of scope when talking about Linux distributions, though you can always ask the vendor to provide you a 64-bit compatible library.
  - WJW 5y ago
    > though you can always ask the vendor to provide you a 64-bit compatible library This is not always true, as the vendor may have disappeared and can no longer provide new versions of the library for any reason. Though, if you knowingly have such a black box in your code and willingly keep running with it until 2038 then you kinda deserve what is coming.
  - mlyle 5y ago
    Then it's not a distinction between dynamic linking and static linking. It's a distinction of "only link against things you yourself build and ship". It's nothing to do with closed or open source. I don't want to build a very big world, and ship a huge amount of binary code that can be found in distributions, to make a 2038-safe app.
    - howinteresting 5y ago
      It's not that big a world. And ststic linking in particular opens up a ton of other opportunities, such as monomorphization.

edflsafoiewq 5y ago

What is the reason a new call like time64_t time64(); wasn't added?

rwaksmunski 5y ago
Are you going to rewrite 5 decades worth of legacy software to use it?
- alkonaut 5y ago
  What’s the alternative to rewriting? Just recompiling? Assuming the code on the receiving end stores the value in 32 bits (and not a time _t which can magically change meaning but a 32 bit integer) then it’s still doomed without rewrite? I mean even with time_t use you can’t just recompile and make it work because there could be subsequent assumptions that the size of a struct containing a time_t will be a specific size or allocations will be too small or misaligned and so on.
  - rwaksmunski 5y ago
    Yes, the alternative to rewriting is changing the size and recompiling. OpenBSD somehow managed to pull it off.
    - alkonaut 5y ago
      But the problem isn't really changing the OS to return a 64bit value where it used to return 32, the problem is all the applications that assumed it would be 32. By "pulling it off" you mean, they changed it and planes didn't crash around them?
      - rwaksmunski 5y ago
        Yes, stuff broke in the ports collection. Not sure about planes but at least one monitor exploded.

Subsentient 5y ago

Thank god all my shit is 64-bit now.

masklinn 5y ago
If you're on a GNU/Linux system that is not actually relevant unless you've also opted into glibc's 64b time_t, and recompiled everything locally: by default, glibc uses 32b time_t even on 64b machines.
- AnssiH 5y ago
  time_t has always been 64-bit on 64-bit glibc (https://sourceware.org/glibc/wiki/Y2038ProofnessDesign https://sourceware.org/glibc/wiki/Y2038ProofnessDesign , https://www.gnu.org/software/libc/manual/html_node/Feature-Test-Macros.html https://www.gnu.org/software/libc/manual/html_node/Feature-T...). > Currently it defaults to 64 bits on most architectures. Although it defaults to 32 bits on some traditional architectures (i686, ARM), this is planned to change and applications should not rely on this.
- petters 5y ago
  This is certainly not true: #include <time.h> #include <stdio.h> int main() { printf("size: %lu\n", sizeof(time_t)); return 0; } prints 8 on my system when compiled with just "gcc test.c"

high_byte 5y ago

added a reminder January 18, 2038 :)

onion2k 5y ago
I know this is a joke but it highlights an issue with the 2038 bug - people think it'll be a problem in 2038. It's very badly named. It should be called something like "the future date problem", because it affects any date that gets converted to a timestamp that represents a time after 19 Jan 2038. I wrote code for calculating the payments on 40 year mortgages back in the early 2000s and I had to consider this problem then.
- high_byte 5y ago
  ah interesting I hadn't realized. but for sure lots of problems will start occurring starting 2038
  - deleted 5y ago
    [deleted]

tgsovlerkhgsel 5y ago

We're 16 years away from the overflow. In other words, some of the systems being put in production right now are going to be around, untouched, unchanged, when it happens. Given this article, it seems we're still doing it wrong, and that means... 2038 will be "fun" (either the remediation, or the consequences of the lack thereof). At least most of us in the industry now have a good retirement plan: Fixing the legacy systems in 16 years...

jandrese 5y ago
Ironic that so many people are hoping to retire before they get asked to deal with this mess since so much medical equipment is built on ancient and poorly maintained microcontroller environments.
- deleted 5y ago
  [deleted]
- treesknees 5y ago
  Which part is ironic?
  - jandrese 5y ago
    The irony of retiring to avoid the problem only to have it become a problem with the equipment keeping you alive in retirement.
Animats 5y ago
We're 16 years away from the overflow. In other words, some of the systems being put in production right now are going to be around, untouched, unchanged, when it happens. Yes. It's getting close. Windows XP, released in 2001, still had a significant presence in 2017, 16 years after introduction.[1] 42% of companies still had some XP systems running back then. 11% of machines were still running it. [1] https://community.spiceworks.com/networking/articles/2719-windows-10-adoption-who-s-using-it-two-years-after-launch https://community.spiceworks.com/networking/articles/2719-wi...

usr1106 5y ago

Isn't the title clickbait? If it read "glibc in 32 bit user space is still not Y2038 compliant by default" I would not have clicked it.

DarkmSparks 5y ago
I was feeling the same, about the only thing I have using 32bit userspace these days are a few rPIs, little bit early to be panicking they may suffer from the Y2038 bug, most companies waited to about 6 months before the Y2K bug was going to hit before they started caring.
- alkonaut 5y ago
  If you spend large amounts of money today, fixing a system that is later decommissioned in 2029 or 2032… you wasted a lot of money. The correct play could very well be to wait to the last second and pay absurd money for scarce experts to fix the systems.
  - DarkmSparks 5y ago
    The correct play is surely to be the scarce expert available at the last second.
- throw0101a 5y ago
  > I was feeling the same, about the only thing I have using 32bit userspace these days are a few rPIs, little bit early to be panicking they may suffer from the Y2038 bug Quite the opposite if you're talking about rPIs: the small/embedded space is where things can live for a long time. They're the primary place to worrying about. Especially if you're talking about industrial processes.
- mwcampbell 5y ago
  But if we techies get way ahead of the problem this time, maybe we can prevent it becoming a mass hysteria as Y2K was.
turminal 5y ago

Asmod4n 5y ago

The problem doesn’t start in 16 years, it’s happening now. Try to do date/time calculation after the overflow today. This can have major issues with ensurance companies right now.

alkonaut 5y ago
Right. This is an overlooked aspect. At the precise date of the epochalypse, the date will roll over for the current time_t. But one second before that, it will blow up for a system that peeks 1 second into the future (e.g a control system). One day before, the date will be screwed up for any program doing addition with one day such as a rental system. In the years before 2038, systems that do calculations on year scales will fail, like perhaps insurance or mortgage and similar.
recov 5y ago
Yep. Also see https://twitter.com/jxxf/status/1219009308438024200 https://twitter.com/jxxf/status/1219009308438024200

einpoklum 5y ago

If I were to write this post, I would take the time to: Check _why_ glibc maintainers have made this decision, and ask them why it is reasonable for this to be the case right now.

erk__ 5y ago
They have a design document for it here: https://sourceware.org/glibc/wiki/Y2038ProofnessDesign https://sourceware.org/glibc/wiki/Y2038ProofnessDesign

OtomotO 5y ago

Wonderful, so I already know what I'll be doing in 2038. If I still live, if civilization is still in place... Keep the paychecks coming! Sorry for my cynicism, it's one coping strategy. I'd also prefer to build reliable software, but alas...

speedgoose 5y ago
I don't know, building reliable software is a lot of boring work. Most software the industry develops is not going to space or safety critical, it's okay if they have bugs like this one.
- OtomotO 5y ago
  The engineer inside me cringes, while the Businessman cheers ;)

getcrunk 5y ago

They can at least throw some compiler warnings.

roeles 5y ago

For work I'm currently building a test-rig that has a projected lifetime of 20 years. So for me, this is very relevant. Thank you for posting.

dathinab 5y ago

Honestly over the years I started associating GNU with: - dedicated but stuck in the past people and approaches - bad or even terrible UX - bad security (either because of bad UX especially bad defaults or because of not consulting crypto experts for crypto stuff)

midjji 5y ago

Even if upgraded, its is so flawed that it is practically universally replaced anyways. There are two severe issues which must be addressed for it to be saved. First, the type should be float, not int. The former has an intuitive precision, while the latter does not. Not to mention the sign issues. It is very rare to see a timestamp int which is not used as a kind of fixed point float somewhere in practice, but doing so manually gives rise to endless bugs. float on the other hand, can always be in second, which is much more intuitive, and means the downstream apps fix if precision changes is much easier to get right. For these reasons, upgrading to float64 would be substantially better than upgrading to int64, while being better for most use cases, almost always as fast, and require less thought in downstream. However, I think we can do even better. Int64 is also too small, as the common nanosecond clocks would only give a century of range, which requires adjusting pretty much every algorithm, even if it only ever deals with a few times. And of course, if I dont care about precession, I want the floating point to deal with it for me. It almost always makes sense to use a time precision exceeding the most common clock with which it will be used, and with a range covering the most common use cases. Most regular cpus give nanosecond precision, but then float64 would only give a few years of range exactly, which is a bit too short for many applications which want exact precision, and use the standard 1973 start. The long awaited float128 however, would be sufficient for billions of years at pico precision, while maintaining low memory and compute costs. Notably cheaper than int32 was originally. float128 simply makes for a fantastic timestamp format for almost every application. Its extremely convenient compared to the clunky std::chrono, works in C and interop, and is very simple to use. The only downside is that compiler support, while getting better, remains bad.

oshiar53-0 5y ago
Is there something wrong in particular for using built-in fixed-point types for timestamps? Yes, overflow/underflow issues and yada yada, but virtually every type can be mised in its own way.
- foxfluff 5y ago
  Nothing wrong with fixed point. It's only a problem because people end up hand-rolling their arithmetic (and botching it) instead of using library functions that do it right.

strenholme 5y ago

I test for these things when building open source software. • 64-bit compiles and applications have a 64-bit time_t in Linux. If your distro and binaries are 64-bit, there isn’t a problem. • 32-bit compiles and applications still have a 32-bit time_t in mainstream Linux libraries, so that old binaries still run. The timestamp is a rolling one; once Y2038 is hit, the timestamp will be a negative one, but one which is updated and is off by 136 years. The workaround is code like this (this is real production code in my Lua fork, Lunacy[1]): time_t t; int64_t tt; if (lua_isnoneornil(L, 1)) /* called without args? */ t = time(NULL); /* get current time */ else { // Lunacy only supports getting the current time lua_pushnil(L); return 1; } if(t < -1) { tt = (int64_t)t + 4294967296ULL; } else { tt = (int64_t)t; } if (t == (time_t)(-1)) lua_pushnil(L); else lua_pushnumber(L, (lua_Number)tt); return 1; This gives us accurate timestamps until 2106. If post 2106 compatibility is needed, we can add 2 ^ 32 again for timestamps with a positive value once the Y2038 rollover happens. • Legacy 32-bit Windows applications using the Posix compatibility layer are not Y2038 compliant. Once Y2038 hits us, the timestamp will always return -1 (in Windows XP, the behavior was to return a number off by 136 years, but this changed in Windows 10). The workaround is to use Windows’ proprietary API for post-Y2038 timestamps. Again, from Lunacy which has a Win32 port: /* Convert Windows "filetime" in to Lua number */ uint64_t t; FILETIME win_time = { 0, 0 }; GetSystemTimeAsFileTime(&win_time); t = win_time.dwHighDateTime & 0xffffffff; t <<= 32; t |= (win_time.dwLowDateTime & 0xffffffff); t /= 10000000; t -= 11644473600LL; lua_pushnumber(L, (lua_Number)t); return 1; Now, last time I researched this, there wasn’t a “int64_t time_64bit()” style system call in the Linux API so that newly compiled 32-bit binaries can be Y2038 compliant without breaking the ABI by using “time_64bit()” instead of “time()”. This was based on some digging around Stackoverflow just last year, and simple Google searches are still not returning pages saying, in big bold letters “-D_TIME_BITS=64 when building 32-bit apps”. https://sourceware.org/glibc/wiki/Y2038ProofnessDesign https://sourceware.org/glibc/wiki/Y2038ProofnessDesign talks about 64-bit support on 32-bit systems as if it’s a future pipe dream, but apparently -D_TIME_BITS=64 works right now. [1] https://github.com/samboy/lunacy https://github.com/samboy/lunacy [2] https://en.wikipedia.org/wiki/Time_in_Indiana https://en.wikipedia.org/wiki/Time_in_Indiana [3] To ruin a classic sadistic interview question for sys admin roles, Linux these days returns both the modification time and the mostly useless “status change” timestamp. Facebook once decided to not move forward because I said that file timestamp was “modification time” and not “status change”; if Facebook is still asking that question, their knowledge is out of date.

ck2 5y ago

Reminder to self not to fly on a plane in January 2038, maybe a few months before and after just to be sure. Whole lotta unknowing beta-testers going to be used to debug old system.

AnssiH 5y ago

The article says that users will never get 64-bit time_t on 32-bit systems on glibc. However, glibc documentation (https://www.gnu.org/software/libc/manual/html_node/Feature-Test-Macros.html https://www.gnu.org/software/libc/manual/html_node/Feature-T...) says: > If _TIME_BITS is undefined, the bit size of time_t is architecture dependent. Currently it defaults to 64 bits on most architectures. Although it defaults to 32 bits on some traditional architectures (i686, ARM), this is planned to change and applications should not rely on this. So it sounds like glibc plans to change the default, just like musl has done.

tyingq 5y ago

MySql fixed their 2038 issues with UNIX_TIMESTAMP() and FROM_UNIXTIME(), fairly recently: https://bugs.mysql.com/bug.php?id=12654 https://bugs.mysql.com/bug.php?id=12654

TacticalCoder 5y ago

But then Y2K was presented as the end of the world, with supposedly nothing being ready, and the world at large was going to experience catastrophic supply chain failures, people would be dying in hospitals, etc. Then Y2K turned out to be a nothing burger (I'm not saying some systems didn't misbehave left and right but in the grand scheme of things, it was a non event). I'm pretty sure it's going to the same with Y2038.

GnarfGnarf 5y ago
Y2K was not a disaster precisely because there was panic and concern, and programmers spent much effort fixing it, and it was fixed due to prioritization of resources & effort. "it's going to the same with Y2038" is like saying we can fold our umbrella in a rainstorm because we haven't gotten wet yet.
- mbg721 5y ago
  When Y2k happened, it also seemed a little ridiculous to laypeople because they thought, "Surely we don't depend on computers that much," and there was still an institutional memory of doing important business on paper. Nobody is under that illusion now.

999900000999 5y ago

Note to self, retire by then. I sure don't want to deal with this. Although knowing corporate America, that's a problem for whoever is CEO in 2037

Aissen 5y ago

From my experience, the libc is not even the most scary step… It's the first, necessary step, but not sufficient. Converting a code base that has serialized data structures with 32 bits timestamps that are not even time_t (which is a good choice since you want serialization stability), now that makes for complex upgrade paths.