12 ms·

Lots of bugs in 32-bit x86 Linux entry code

153 points by swills 7y ago

purplezooey 7y ago
Segments... ha ha
dang 7y ago
Url changed from https://lwn.net/Articles/805539/ https://lwn.net/Articles/805539/, which points to this. Submitted title was "Essentially no upstream development resources dedicated to x86_32 Linux"
- swills 7y ago
  Thank you!
human_banana 7y ago
Maybe because nobody uses 32-bit x86 anymore?
- heavyset_go 7y ago
  There were 32-bit netbooks being sold the last time I looked in 2015.
  - peteradio 7y ago
    Jesus was closer to dinosaur times than we are to 2015.
    - grmn 7y ago
      Are you Him?
    - heavyset_go 7y ago
      Casual computer users don't buy new computers each year. Pretty sure set top boxes and routers with 32-bit x86 processors have been sold since then, too.
  - gdxhyrd 7y ago
    Sure, but the ones that pay the bills are servers and companies which don't care one bit about i386.
    - heavyset_go 7y ago
      Agreed, however open source has always relied on enthusiasts for consumer-grade hardware support.
  - opencl 7y ago
    The really fun part was when they launched the first 64 bit Atom CPUs (Bay Trail) and then a bunch of them got shipped with only 32 bit UEFI and cannot boot in 64 bit mode.
    - zozbot234 7y ago
      They have to _boot_ with a 32-bit UEFI bootloader, but can run a 64-bit kernel and userland. Fedora Linux even allows for this while keeping secure boot on; Debian supports it as well but the interaction with secure boot is buggy, so you need to turn it off. Not sure about other distros, however.
    - heavyset_go 7y ago
      Early 64-bit Intel Macs also shipped with a 32-bit UEFI limitation, but they were able to boot into a 64-bit system.
  - makomk 7y ago
    Unfortunately, upstream doesn't really notice if 32-bit x86 breaks. This isn't just a kernel problem; glibc managed to push out a release back in 2017 that broke memchr on 32-bit Atom in a way that tended to cause programs calling it to segfault. It turns out that amongst a few other things, including Python, the glibc build process itself relies on memchr working... I don't think they actually tested the code path in question after modifying it.
    - exikyut 7y ago
      Of course it was tested [on the developers' x86_64 laptop(s)]!
- bryanlarsen 7y ago
  No, that's the i386 architecture, which is still well supported AFAICT. x86_32 is a funny architecture that requires 64 bit chips but uses 32 bit pointers. http://www.h-online.com/open/features/Kernel-Log-x32-ABI-gets-around-64-bit-drawbacks-1342061.html http://www.h-online.com/open/features/Kernel-Log-x32-ABI-get...
  - comex 7y ago
    No, that’s “x32”. “x86_32” is not an official name for anything, but based on the contents of the post, I believe Andy Lutomirski is using it to refer to 32-bit x86, aka i386.
    - rurban 7y ago
      And I believe he meant i686. Latest Intel CPU's with 32bit pointers.
      - the8472 7y ago
        x86-64 procesors can generally be run in IA32 compatibility mode. And intel still makes some IA32-only chips for embedded.
  - treeform 7y ago
    Your comment should be the top comment. I had no clue i386 and x86_32 where different. I though the article was saying 32bit linux is broken.... but no some strange "alternative" mode on linux is broken.... who cares.
    - comex 7y ago
      The comment is wrong. It is about i386.
  - geofft 7y ago
    This post is about i386 / x86_32. What you're referring to is "x32," a different thing entirely, which uses the x86-64 instruction set. You can tell this post is not about "x32" because it talks about "segments, gates, and other delights from the i386 era," which don't exist on the x86-64 architecture regardless of how long your pointers are, and it says things are "fine on a 64-bit kernel" (x32 uses a 64-bit kernel, it's a userspace ABI only).
- thrower123 7y ago
  If only... I have been hoping for a native x64 version of Visual Studio for almost a decade, and signs do not look good for that ever coming to fruition. As it is, loading up solutions that consume nearly 2GB of RAM are still bringing the IDE to its knees and making it thrash and page.
- mike_hock 7y ago
  My old Atom laptops still run perfectly fine. Thanks, Debian.
jdsully 7y ago
I remember being an early adopter of 64-bit Linux with my Athlon64 back in the day. Lots of stuff was broken and I got a lot of debate on whether it was any faster or worthwhile at all. It's really cool to see the technology curve go full circle.
- cptskippy 7y ago
  Completely off topic but... Althlon64 was a swift kick in the nuts for Intel, but AMD really didn't followup until Ryzen. I wonder if they're going to stick around and fight this time.
  - topbanana 7y ago
    They've got momentum on their side now, but intel won't bow down without a fight. I expect we'll see some real innovation over the next five years.
    - cptskippy 7y ago
      The real question isn't if Intel will fight back but how. They've been known to innovate shady marketing tactics just as much as their CPU architecture.
  - kbenson 7y ago
    Sometimes when the swift kick in the nuts is delivered to an 800-pound gorilla, it doesn't matter if you stay around to fight, you're outclassed. This time, AMD may have stepped up with a steel pipe. We'll see how well they can use it...
    - lsofzz 7y ago
      > This time, AMD may have stepped up with a steel pipe. We'll see how well they can use it... :-))))
  - tyingq 7y ago
    Back then, AMD kind of had to license the x64 stuff to Intel for cross license reasons. They aren't under any pressure this time to give away anything.
jhoechtl 7y ago
Hah! > And the developers in question should have an appropriate degree of nostalgic adoration of segments, gates, and other delights from the i386 era.
cpeterso 7y ago
"I strongly suspect that there is at least one bug left." :)
- ShinTakuya 7y ago
  That was a cheeky statement by the author if I've ever seen one.
- saagarjha 7y ago
  That probably would be a valid addition to any bug fix in a large project, to be honest.
geofft 7y ago
Relatedly, Ubuntu no longer builds for i386 as a complete architecture: https://lists.ubuntu.com/archives/ubuntu-devel-announce/2019-June/001261.html https://lists.ubuntu.com/archives/ubuntu-devel-announce/2019... They are still building i386 packages for use on 64-bit kernels, for the sake of precompiled 32-bit software (games etc). That's scenario B from Simon McVittie's reply in this thread, and it sounds like that's not affected.
- yingw787 7y ago
  I was able to upgrade from Lubuntu 16.04 to 18.04 for my 32bit IBM Thinkpad T42 using do_release_upgrade. It’s not downloadable on the website, but I think for 18.04 the build is still there.
  - gizmo686 7y ago
    You can download Ubuntu going back to 4.10 here: http://old-releases.ubuntu.com/releases/ http://old-releases.ubuntu.com/releases/
    - Piskvorrr 7y ago
      Not sure how well that would work for installing things that aren't in the release ISOs...
  - geofft 7y ago
    I think they've stopped building them in 19.10. 18.04 is an LTS supported for 5 years, so users of i386-only CPUs are fine until 2023.
    - mavhc 7y ago
      I have a load of 32bit laptops, "I'll put ChromiumOS on them", nope, that's not a thing any more. OK, err Linux? Apparently not. Could put Windows 10 on them I guess.
      - 7y ago
skissane 7y ago
> We need real CI resources It would be easy to boot the 32-bit kernel in a VM as part of a CI pipeline and then run some automated tests to catch issues like this. (Unless by "real" he means non-virtualised? Is there anything about these bugs that would only reproduce on bare metal?)
- TD-Linux 7y ago
  I think the issue is not the hardware needed but simply the cost of the extra compute. Linux doesn't have an "official" CI server but relies on third parties running it, and none of them want to spend $ on compute for an architecture they will never use.
  - amluto 7y ago
    The hardware cost should be negligible. The relevant cost is labor.
- vetrom 7y ago
  Not always. Cache speculation bugs for example I dont think are covered by emulations. (At least not explicitly. If the mmu model is the same and transparent enough I suppose those could 'pass through' but only if they're implemented on the same plat form.
  - bonzini 7y ago
    You can certainly exploit Meltdown from a VM. The hypervisor tries hard to stay out of the way, and on a properly configured VM the vmexits should be few enough while running compute-bound code that microarchitectural side channels are very well exploitable.
    - inetknght 7y ago
      For clarity, in this context I think @ vetrom was talking about emulation, not a virtual machine. A virtual machine makes use of the native instruction set (relying on the hardware) to create a machine within a machine and naturally can only create virtual machines of identical hardware capabilities (or a subset thereof). Emulation does all of that in software and could emulate even a completely different architecture.
Ericson2314 7y ago
Oh, the freeloading. How much more CI do we need...
sockalocka 7y ago
Edit: I am possibly wrong here, and confusing this with an issue that was reported earlier

153 points by swills 7y ago

Segments... ha ha

dang 7y ago

Url changed from https://lwn.net/Articles/805539/ https://lwn.net/Articles/805539/, which points to this. Submitted title was "Essentially no upstream development resources dedicated to x86_32 Linux"

swills 7y ago
Thank you!

human_banana 7y ago

Maybe because nobody uses 32-bit x86 anymore?

heavyset_go 7y ago
There were 32-bit netbooks being sold the last time I looked in 2015.
- peteradio 7y ago
  Jesus was closer to dinosaur times than we are to 2015.
  - grmn 7y ago
    Are you Him?
  - heavyset_go 7y ago
    Casual computer users don't buy new computers each year. Pretty sure set top boxes and routers with 32-bit x86 processors have been sold since then, too.
- gdxhyrd 7y ago
  Sure, but the ones that pay the bills are servers and companies which don't care one bit about i386.
  - heavyset_go 7y ago
    Agreed, however open source has always relied on enthusiasts for consumer-grade hardware support.
- opencl 7y ago
  The really fun part was when they launched the first 64 bit Atom CPUs (Bay Trail) and then a bunch of them got shipped with only 32 bit UEFI and cannot boot in 64 bit mode.
  - zozbot234 7y ago
    They have to _boot_ with a 32-bit UEFI bootloader, but can run a 64-bit kernel and userland. Fedora Linux even allows for this while keeping secure boot on; Debian supports it as well but the interaction with secure boot is buggy, so you need to turn it off. Not sure about other distros, however.
  - heavyset_go 7y ago
    Early 64-bit Intel Macs also shipped with a 32-bit UEFI limitation, but they were able to boot into a 64-bit system.
- makomk 7y ago
  Unfortunately, upstream doesn't really notice if 32-bit x86 breaks. This isn't just a kernel problem; glibc managed to push out a release back in 2017 that broke memchr on 32-bit Atom in a way that tended to cause programs calling it to segfault. It turns out that amongst a few other things, including Python, the glibc build process itself relies on memchr working... I don't think they actually tested the code path in question after modifying it.
  - exikyut 7y ago
    Of course it was tested [on the developers' x86_64 laptop(s)]!
bryanlarsen 7y ago
No, that's the i386 architecture, which is still well supported AFAICT. x86_32 is a funny architecture that requires 64 bit chips but uses 32 bit pointers. http://www.h-online.com/open/features/Kernel-Log-x32-ABI-gets-around-64-bit-drawbacks-1342061.html http://www.h-online.com/open/features/Kernel-Log-x32-ABI-get...
- comex 7y ago
  No, that’s “x32”. “x86_32” is not an official name for anything, but based on the contents of the post, I believe Andy Lutomirski is using it to refer to 32-bit x86, aka i386.
  - rurban 7y ago
    And I believe he meant i686. Latest Intel CPU's with 32bit pointers.
    - the8472 7y ago
      x86-64 procesors can generally be run in IA32 compatibility mode. And intel still makes some IA32-only chips for embedded.
- treeform 7y ago
  Your comment should be the top comment. I had no clue i386 and x86_32 where different. I though the article was saying 32bit linux is broken.... but no some strange "alternative" mode on linux is broken.... who cares.
  - comex 7y ago
    The comment is wrong. It is about i386.
- geofft 7y ago
  This post is about i386 / x86_32. What you're referring to is "x32," a different thing entirely, which uses the x86-64 instruction set. You can tell this post is not about "x32" because it talks about "segments, gates, and other delights from the i386 era," which don't exist on the x86-64 architecture regardless of how long your pointers are, and it says things are "fine on a 64-bit kernel" (x32 uses a 64-bit kernel, it's a userspace ABI only).
thrower123 7y ago
If only... I have been hoping for a native x64 version of Visual Studio for almost a decade, and signs do not look good for that ever coming to fruition. As it is, loading up solutions that consume nearly 2GB of RAM are still bringing the IDE to its knees and making it thrash and page.
mike_hock 7y ago
My old Atom laptops still run perfectly fine. Thanks, Debian.

jdsully 7y ago

I remember being an early adopter of 64-bit Linux with my Athlon64 back in the day. Lots of stuff was broken and I got a lot of debate on whether it was any faster or worthwhile at all. It's really cool to see the technology curve go full circle.

cptskippy 7y ago
Completely off topic but... Althlon64 was a swift kick in the nuts for Intel, but AMD really didn't followup until Ryzen. I wonder if they're going to stick around and fight this time.
- topbanana 7y ago
  They've got momentum on their side now, but intel won't bow down without a fight. I expect we'll see some real innovation over the next five years.
  - cptskippy 7y ago
    The real question isn't if Intel will fight back but how. They've been known to innovate shady marketing tactics just as much as their CPU architecture.
- kbenson 7y ago
  Sometimes when the swift kick in the nuts is delivered to an 800-pound gorilla, it doesn't matter if you stay around to fight, you're outclassed. This time, AMD may have stepped up with a steel pipe. We'll see how well they can use it...
  - lsofzz 7y ago
    > This time, AMD may have stepped up with a steel pipe. We'll see how well they can use it... :-))))
- tyingq 7y ago
  Back then, AMD kind of had to license the x64 stuff to Intel for cross license reasons. They aren't under any pressure this time to give away anything.

jhoechtl 7y ago

Hah! > And the developers in question should have an appropriate degree of nostalgic adoration of segments, gates, and other delights from the i386 era.

cpeterso 7y ago

"I strongly suspect that there is at least one bug left." :)

ShinTakuya 7y ago
That was a cheeky statement by the author if I've ever seen one.
saagarjha 7y ago
That probably would be a valid addition to any bug fix in a large project, to be honest.

geofft 7y ago

Relatedly, Ubuntu no longer builds for i386 as a complete architecture: https://lists.ubuntu.com/archives/ubuntu-devel-announce/2019-June/001261.html https://lists.ubuntu.com/archives/ubuntu-devel-announce/2019... They are still building i386 packages for use on 64-bit kernels, for the sake of precompiled 32-bit software (games etc). That's scenario B from Simon McVittie's reply in this thread, and it sounds like that's not affected.

yingw787 7y ago
I was able to upgrade from Lubuntu 16.04 to 18.04 for my 32bit IBM Thinkpad T42 using do_release_upgrade. It’s not downloadable on the website, but I think for 18.04 the build is still there.
- gizmo686 7y ago
  You can download Ubuntu going back to 4.10 here: http://old-releases.ubuntu.com/releases/ http://old-releases.ubuntu.com/releases/
  - Piskvorrr 7y ago
    Not sure how well that would work for installing things that aren't in the release ISOs...
- geofft 7y ago
  I think they've stopped building them in 19.10. 18.04 is an LTS supported for 5 years, so users of i386-only CPUs are fine until 2023.
  - mavhc 7y ago
    I have a load of 32bit laptops, "I'll put ChromiumOS on them", nope, that's not a thing any more. OK, err Linux? Apparently not. Could put Windows 10 on them I guess.
    - 7y ago

skissane 7y ago

> We need real CI resources It would be easy to boot the 32-bit kernel in a VM as part of a CI pipeline and then run some automated tests to catch issues like this. (Unless by "real" he means non-virtualised? Is there anything about these bugs that would only reproduce on bare metal?)

TD-Linux 7y ago
I think the issue is not the hardware needed but simply the cost of the extra compute. Linux doesn't have an "official" CI server but relies on third parties running it, and none of them want to spend $ on compute for an architecture they will never use.
- amluto 7y ago
  The hardware cost should be negligible. The relevant cost is labor.
vetrom 7y ago
Not always. Cache speculation bugs for example I dont think are covered by emulations. (At least not explicitly. If the mmu model is the same and transparent enough I suppose those could 'pass through' but only if they're implemented on the same plat form.
- bonzini 7y ago
  You can certainly exploit Meltdown from a VM. The hypervisor tries hard to stay out of the way, and on a properly configured VM the vmexits should be few enough while running compute-bound code that microarchitectural side channels are very well exploitable.
  - inetknght 7y ago
    For clarity, in this context I think @ vetrom was talking about emulation, not a virtual machine. A virtual machine makes use of the native instruction set (relying on the hardware) to create a machine within a machine and naturally can only create virtual machines of identical hardware capabilities (or a subset thereof). Emulation does all of that in software and could emulate even a completely different architecture.

Ericson2314 7y ago

Oh, the freeloading. How much more CI do we need...

sockalocka 7y ago

Edit: I am possibly wrong here, and confusing this with an issue that was reported earlier