10 ms·

A warning about 5.12-rc1

217 points by eplanit 6y ago

abraae 6y ago
Linus really does write beautifully. Love his style, concise, easily readable, frank, honest with nice human touches.
- kelnos 6y ago
  That email was many things, but concise is definitely not one of them.
  - RegW 6y ago
    ... but remarkably good humoured! (Or are those teeth gritted?:-)
    - OskarS 6y ago
      I thought so too! Linus was even like "don't blame the developers, this one was particularly nasty" and everything! Good to see nice and sensible Linus once in a while!
- hjek 6y ago
  > So I'm not blaming the developers in question, Much kinder than past Linus writings. I loved it too. "Double ungood" works at least as good as the swearing.
  - rob74 6y ago
    For anyone who didn't recognize it, that's a reference to Orwell's Newspeak (https://en.wikipedia.org/wiki/Newspeak https://en.wikipedia.org/wiki/Newspeak). Which makes me wonder about two things: - is Linus hinting that he is censoring himself, and if he wasn't, what expression would he have used instead? - for what kind of incident does he reserve the attribute "double plus ungood"? Probably if such a bug would get into the production kernel?
    - jlgaddis 6y ago
      > ... if he wasn't, what expression would he have used instead? My first guess would be something that includes the word "fuck"... but, on second thought, this isn't Nvidia -- so perhaps not.
    - matthewmacleod 6y ago
      You’re reading way too much into it - it’s just a colourful expression. Anyway, “double ungood” would not be a valid Newspeak construction. The base is “good” and “ungood” is the negation; intensification would be “plusungood” and further intensification “doubleplusungood”.
  - speeder 6y ago
    In my view seeing him writing double ungood worries me. Don't need to swear, but he could have written "very bad" or something like that.
  - jlgaddis 6y ago
    I can recall a colleague, many years ago, sometimes using "doubleplusungood", often shortened -- in (typical lazy UNIX admin style -- to just ++ungood Later, another cow-orker tried using a slight variant ... ++!good ... although that one never really caught on -- apparently it didn't compile or something (maybe because we were using Red Hat's version "2.96" of gcc at the time?). -- (+10 Internet points if you get the reference(s))
    - rob74 6y ago
      Oops, sorry for giving it away in my comment ;)
      - jlgaddis 6y ago
        It's okay, that's not the only one!
    - kps 6y ago
      ++ can't operate on !good because it's not an lvalue (i.e. not assignable). typedef double winston; winston smith(int good) { return (double)+!good; }
  - fractalb 6y ago
    I think that the word ‘Double ungood’ was generated by his auto correction program which automatically replaces all the swear words with some less offensive words
deleted 6y ago
[deleted]
encryptluks2 6y ago
I guess because I use a swapfile I am not normal according to Linus? Haha, I just find it easier to use a swap file when dual-booting Windows and my Linux install is encrypted. Otherwise, it'd me more work to ensure my swap partition is encrypted.
- asymptosis 6y ago
  I recently ended up using swapfiles because I refactored my disk layout and it was just simpler to chuck the swap on an existing nvme partition. I realised an extra benefit: you can make it a sparse file, so it never even uses disk space at all until you actually need it, which could be never.
  - muxator 6y ago
    "refactoring" means changing the (often implicit) structure of code without changing its external behaviour. In this case "reorganized" or "repartitioned" could probably be more apt. If we'd talked about software here (and not disk layout), I'd have considered less leniently the use of "refactoring", and a hint of unclear intentions. Sorry for being so nitpicky, but this is a sore thumb from too many "refactorings" that are, indeed, somenthing else.
    - asymptosis 6y ago
      Refactoring has the meaning that we impute to it. God didn't hand down golden tablets in the 60s with the words "thou shalt only use 'refactor' in these prescribed ways". Instead of being sorry for being nitpicky, maybe just keep your nitpicks to yourself in future.
  - qiqitori 6y ago
    But if you suddenly need it and don't have it, you'll get segfaults (I assume). Sounds a bit too dangerous for me :p
    - NavinF 6y ago
skissane 6y ago
I was interested to see the code. Linus's message doesn't provide commit IDs, so you have to go looking yourself. I believe this is the commit which introduces the bug: https://github.com/torvalds/linux/commit/48d15436fde6feebcded7bd0fdc8ea4a9181b8fa https://github.com/torvalds/linux/commit/48d15436fde6feebcde... And I believe this is the commit which fixes it: https://github.com/torvalds/linux/commit/caf6912f3f4af7232340d500a4a2008f81b93f14 https://github.com/torvalds/linux/commit/caf6912f3f4af723234...
- CyberRabbi 6y ago
  Why do you believe that’s the commit that caused the bug?
  - exikyut 6y ago
    The second commit self-disambiguates with "swap: fix swapfile read/write offset" and the commit message notes "fixes: 48d1543", which is the first linked commit.
    - CyberRabbi 6y ago
      Thanks for answering the question
  - dgellow 6y ago
    Because of the commit message of the fix?
- N48 6y ago
  The fix makes me feel a little uneasy. It changes the behaviour of swap_page_sector() to take into account the offset. But swap_page_sector() was already used before the bug was introduced, a few lines earlier in the bdev_write_page() fast path, as introduced by this commit from 2014: https://github.com/torvalds/linux/commit/dd6bd0d9c7dbb3951005db4e424abbd37c4ff801 https://github.com/torvalds/linux/commit/dd6bd0d9c7dbb395100... So either: 1. The bdev_write_page() call was also broken, and has been for years, or 2. The bdev_write_page() uses the sector value differently, worked fine with the old swap_page_sector(), but is now broken with the new version, or 3. The change in swap_page_sector() somehow doesn't affect bdev_write_page(); maybe it's a noop when using swap files?
walrus01 6y ago
this is unlikely to hurt you unless you use a swap file, rather than a dedicated swap partition. for performance reasons most any modern linux installation would be using a dedicated swap partition, that is defined as swap-only during the install process with mkswap In my experience some of the reasons for using a swap file were more common maybe 10 or 15 years ago. Nowadays with disk space being less costly and rare, and systems having a lot more RAM (so the likelihood of going deep into swap is less), it's unsurprising that none of the testers encountered it. https://man7.org/linux/man-pages/man8/mkswap.8.html https://man7.org/linux/man-pages/man8/mkswap.8.html
- ylyn 6y ago
  Swapfiles are quite common when you use FDE (say, LUKS) and you don't want your swap to be in plaintext on disk. (Although, you could just have an entire LUKS-encrypted swap volume instead of a swapfile...)
  - walrus01 6y ago
    most people I know who are paranoid to really go into the details of a FDE linux desktop/workstation environment choose to have no swap file, and at minimum 64GB of RAM. even safer if the swap doesn't exist.
  - dathinab 6y ago
    While maybe not the most performant my favorite is LVM2 on top of a single large LUKS-encrypted partition (+no boot partition and a single (custom platform key) signed EFI bootable kernel blob containing the initram fs in the EFI partition). This: - Allows a encrypted swap partition (on in lvm2 on top of luks, sure not perfect but I don't really use swap) - Allows hibernation (which I don't really use tbh.) - Fully encrypted disc - Fast boot (compared to e.g. encrypted grub, at least last time I checked) - Easy resizing of partitions on demand - Which in turn makes it easier to use more partitions (for /, /home, etc.), which is necessary for taking advantage of certain mount flags like noexec for security. Anyway I think the reason Ubuntu uses a swap file is because it means you don't need to resize a partition when "upgrading" your RAM or similar (for hibernation to work). Through tbh. they should make the choice in the installer dependent of the hardware.
xpressvideoz 6y ago
> And, as far as I know, all the normal distributions set things up with swap partitions, not files I guess Ubuntu is not a "normal" distribution then? Because it uses a swap file by default.
- dathinab 6y ago
  While it's the default setup of the auto installer it common (and IMHO best) practice to change it to install with a swap partition. It's just that it's much easier for anyone who just gets started with Linux to have a swap file. For example for hibernation you swap partition needs to have the right size. But people do upgrade their desktop RAM and as such would need to resize the partition, which isn't easy (because likely it means shrinking another partition)... So for a distribution like ubuntu the "non-normal" case became the default. Still this mail was targeting people which install pre-release versions of Linux kernerls which is a group of people relatively unlikely to use a swap file.
  - pbhjpbhj 6y ago
    I used to use a swap partition, but then if you get the partition size wrong you're lumbered with it. Why wouldn't you use a file. At one point in the past I used the default (or recommended) /boot size and got stuck a couple of years later juggling at updates to fit the boot files into the partition. I just use a monolithic partition per OS per disk now and swap files within that. Is that not good practice (for home systems)?
    - samus 6y ago
      `/home` should be separate from the system so you can reinstall it without trouble. Same for `/var` (or subdirectories) if you value the data of particular services. Other than that, the bootloader and EFI must be able to access certain directories like `/boot` and `/boot/efi`. As long as it works for your use cases, everything is alright. Edit: A swap file has a little bit of overhead because of the underlying file system. But at the point you notice this, your system is pretty much frozen anyways.
1996 6y ago
As no one did a TLDR: if you use a swapfile, it could overwrite part of the the filesystem where the swapfile exist. Don't use swapfiles if you can. Use a dedicated partition, there are many advantages (including automatic RAID0 if you have a swap partition on each drive)
- travv0 6y ago
  If you click the link, there's a TL;DR right up at the top, under the title.
kzrdude 6y ago
Automatic testing could have caught this, maybe.
- noobermin 6y ago
  The email mentions that normal testing didn't catch it as it's not a typical use-case for the devs apparently.
  - gpvos 6y ago
    Which is exactly why automated testing exists, to catch cases like this.
    - _-___________-_ 6y ago
      If the devs foresaw the set of conditions that trigger the bug and wrote a test that produces those conditions.
    - unionpivo 6y ago
      Automated testing is not magic. it only tests things you make it test. And since kernel is basically involved into everything that computer does, you would have to test everything, to find every error. Linux kernel is also extremely customizable, so that it can run from watches, to mars rowers, to HPC and cloud. Linux kernels are continuously heavily tested on various different hardware architectures, by various organizations (RH, IBM, Oracle, Google). So there are automated tests done. It's just seems nobody tested this.
      - gpvos 6y ago
        Of course. Thanks for stating a lot of mostly obvious things. Assuming there is a test suite (I really don't know enough about Linux kernel testing, sorry), it is a matter of testing that on a representative set of configurations. Apparently no-one thought of having one with a swap file instead of a swap partition. That's okay, it's an understandable oversight, but one that clearly should be remedied.
fomine3 6y ago
This is the root cause of fs corruption that Phoronix reported. https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.12-rc2-Likely-Soon https://www.phoronix.com/scan.php?page=news_item&px=Linux-5....
akavel 6y ago
> it didn't even show up in normal testing, exactly because swapfiles just aren't normal. Uhh, does it mean swapfiles are undertested in the Linux kernel? Or is there some "non-normal" extra testing that is being run for non-rc releases, but not for rc ones? (!?)
- exmadscientist 6y ago
  I fear you grossly overestimate how detailed the kernel's testing is. (Do please note that the word "detailed" above was carefully chosen.)
  - encryptluks2 6y ago
    While the Kernel team directly may not be as involved in testing the Linux Kernel to the extent that dedicated QA team would be, there are multiple billion dollar organizations like IBM (Red Hat), Oracle, etc that do a lot of automated testing. Given the amount of issues with Windows as well, especially for Insider builds, it doesn't make me lose trust in the Linux kernel anymore than I'd trust a Windows operating system. In fact, because it was caught before it would regular distro users it gives me more trust.
- cmeacham98 6y ago
  > Uhh, does it mean swapfiles are undertested in the Linux kernel? Or is there some "non-normal" extra testing that is being run for non-rc releases, but not for rc ones? (!?) My understanding is that is the point of RC builds, they are the builds used for testing before a stable release.
deleted 6y ago
[deleted]
londons_explore 6y ago
Why are swap files/partitions constant size? Most of the time, I don't need any swap at all, and any swap file/partition is wasted space. While hibernated, I need ~the amount of system ram as swap. While running some horrendous matlab script, I need about 1TB of swap. Yet linux won't dynamically allocate it like other files. I'm forced to constantly resize it manually to save space or be able to do more things.
- megous 6y ago
  Possibly because it's hard to guarantee re-allocating/sizing will not require more RAM during the process on any given filesystem...
- dspillett 6y ago
  I assume the argument to be that it adds complexity (out of swap, what do I do, grow it or something else, if set to grow what if there isn't enough space, if set to grow and there are multiple swap locations which do I grow, ...) that needs to be maintained & tested, but isn't commonly enough needed for someone working on the kernel to decide it warrants that effort, given there are workarounds (i.e. dynamically adding swap partitions/volumes/files) for most of the cases where it might be needed. > I need about 1TB of swap. I don't think most people[†] would ever want that much to be dynamically allocated. If something has needed that much memory unexpectedly then it is likely to be something like a massive leak or accidental fork bomb. It could quickly get to the point where a performance drop due to swap thrashing means you can't interact with the system to kill the offending process so your system is effectively locked until either hit the BRB or it runs out of the massive maximum storage allocated to swap. Of course that adds more complexity: how do you decide how much too much is? If it is a user decision and you sometimes want it to be able to balloon that much but still want the OOM killer to deal with accidentally leaky processes at other times, then you are back to manually managing swap allocation before/after processes that are expected to heavily use it. [†] not that it is wrong to have a use case that warrants it, but if there is a safe workaround for such unusual cases that complicates life just a little, then IMO that is preferable to the risk of adding complexity that will likely go relative untested because it isn't commonly used. > linux Of course other OSs have made different decisions on the matter, so maybe there is room for further discussion.
the_duke 6y ago
I used to assume that Linux had a huge test suite with hundreds of thousands of tests, given the crazy feature matrix that the kernel has to support. And that they would have started requiring tests for new features and bug fixes. Alas, that's not the case. There are a few external test suites like the Linux Test Project [1], but nothing that looks very extensive. The process seems to mostly rely on maintainers giving patches a good look, plus developers and first line users, including hardware manufacturers and corps like IBM or Red Hat, running their own software on kernel pre-releases and reporting the bugs. I realize that most of Linux is device drivers or code that is tightly coupled to hardware, which is more or less impossible to test without a huge test farm and lot's of manual labor, but it's still surprising to me that they don't do it for the more or less device independent functionality. This particular bug looks like it might have been caught by a relatively straight-forward swap file test case. [1] https://github.com/linux-test-project/ltp https://github.com/linux-test-project/ltp
- ddevault 6y ago
  It is important to remember that this is the first release candidate of a series which is usually 7 or 8 entries strong. It's pretty far removed from shipping. This is a testing procedure, though not one which looks the way you're used to. In general, Linux ships pretty reliable releases once the rc process is completed.
- lumpa 6y ago
  Comment rom "Helping Out with LTS Kernel Releases"[0]: "I find it amusing that the answer to "how can I help test" is "that people test the -rc releases when I announce them, and let me know if they work or not for their systems/workloads/tests/whatever". No smoke tests, no automated tools, no suggested workloads, no fuzzing, just "fool around and report back". There probably are lots of kernel testing tools around, but you wouldn't learn that from the linked article." It seems to be conscious choice, to the point of developers stating that "the process works". Which, looking at the results, it does seem to work. But wouldn't it work better with more tests, easily run by users wanting to help? I mean, if a downstream patch in related code regresses the issue, is there a new test added with this fix that would catch that? Edited to add: it seems that the situation is changing for the better, with gregkh saying in a comment: "`make kselftest` seems to do what you want today, if there are any gaps the kernel developers are glad to take more tests." [0] https://news.ycombinator.com/item?id=26021962 https://news.ycombinator.com/item?id=26021962 [1] https://lwn.net/Articles/848352/ https://lwn.net/Articles/848352/
JediPig 6y ago
unpopular opinion here... maybe if linus would fix kernel bugs ( i am looking at pulse audio right now ) even though it would break ABI, we would not have these long term issues. Userland is a utter mess, because of fragmentation and the insanely deceptive "rule" of no breaking userland. Linux has 26 million lines of BLOAT, CORPORATE BLOAT. Eventually linux will be considered to bloated to continue. As a hobbiest, I have no desire to work on the kernel, I may go work on a bsd kernel , cause they own both the kernel and userland. I am allowed to to change both, and make both better. It is why kqueue is vastly superior to epoll crap. Sincerly, A fat ass guy that has written more code than linus.
mschuster91 6y ago
I'd find it interesting to read an analysis how this bug could have happened from a technical point. Like... is swap file support going shortcuts in the kernel so that it can corrupt the file system the swap file is residing on? Because (at least in my hope) no matter of write() or whatever calls on a file handle should cause a filesystem corruption...
- cesarb 6y ago
  The swap file code does not go through the filesystem. It's the exact same code which is used for a swap partition or a swap file, and it writes directly to the block device (the partition). The main difference is that, if you have a swap file, the "swapon" call asks the filesystem to tell it which ranges in the block device contain the swap, instead of using the whole block device.
  - bombcar 6y ago
    I did not know this - it makes sense, but it certainly a surprise.
egberts 6y ago
Yep, encrypted swap space is also an added configuration and that too got hosed bad in 5.12 so that’s an issue i’ve yet to file a bug for because i’m still stepping through the kdb. This is a dedicated data partition that LVM managed to procure a data volume that user land encryption performs on for a swap partition.
MR4D 6y ago
This is one of the best examples of leadership I’ve ever seen... Huge impactful problem.... Gets past the gatekeepers... Things go bad... Issue gets discovered & corrected.... Leader writes an honest and open note to explain the situation and actually apologizes. Corporate America could learn something from this.
pavon 6y ago
> Additionally, he is asking maintainers to not start branches from 5.12-rc1 to avoid future situations where people land in the buggy code while bisecting problems. Nice. Everywhere I've worked, once the fix has been committed the job is done. I should be more conscientious about preventing bad commits from being in branch histories like this.
sys_64738 6y ago
This is why life on the bleeding edge is sometimes very bloody.

217 points by eplanit 6y ago

abraae 6y ago

Linus really does write beautifully. Love his style, concise, easily readable, frank, honest with nice human touches.

kelnos 6y ago
That email was many things, but concise is definitely not one of them.
- RegW 6y ago
  ... but remarkably good humoured! (Or are those teeth gritted?:-)
  - OskarS 6y ago
    I thought so too! Linus was even like "don't blame the developers, this one was particularly nasty" and everything! Good to see nice and sensible Linus once in a while!
hjek 6y ago
> So I'm not blaming the developers in question, Much kinder than past Linus writings. I loved it too. "Double ungood" works at least as good as the swearing.
- rob74 6y ago
  For anyone who didn't recognize it, that's a reference to Orwell's Newspeak (https://en.wikipedia.org/wiki/Newspeak https://en.wikipedia.org/wiki/Newspeak). Which makes me wonder about two things: - is Linus hinting that he is censoring himself, and if he wasn't, what expression would he have used instead? - for what kind of incident does he reserve the attribute "double plus ungood"? Probably if such a bug would get into the production kernel?
  - jlgaddis 6y ago
    > ... if he wasn't, what expression would he have used instead? My first guess would be something that includes the word "fuck"... but, on second thought, this isn't Nvidia -- so perhaps not.
  - matthewmacleod 6y ago
    You’re reading way too much into it - it’s just a colourful expression. Anyway, “double ungood” would not be a valid Newspeak construction. The base is “good” and “ungood” is the negation; intensification would be “plusungood” and further intensification “doubleplusungood”.
- speeder 6y ago
  In my view seeing him writing double ungood worries me. Don't need to swear, but he could have written "very bad" or something like that.
- jlgaddis 6y ago
  I can recall a colleague, many years ago, sometimes using "doubleplusungood", often shortened -- in (typical lazy UNIX admin style -- to just ++ungood Later, another cow-orker tried using a slight variant ... ++!good ... although that one never really caught on -- apparently it didn't compile or something (maybe because we were using Red Hat's version "2.96" of gcc at the time?). -- (+10 Internet points if you get the reference(s))
  - rob74 6y ago
    Oops, sorry for giving it away in my comment ;)
    - jlgaddis 6y ago
      It's okay, that's not the only one!
  - kps 6y ago
    ++ can't operate on !good because it's not an lvalue (i.e. not assignable). typedef double winston; winston smith(int good) { return (double)+!good; }
- fractalb 6y ago
  I think that the word ‘Double ungood’ was generated by his auto correction program which automatically replaces all the swear words with some less offensive words

deleted 6y ago

[deleted]

encryptluks2 6y ago

I guess because I use a swapfile I am not normal according to Linus? Haha, I just find it easier to use a swap file when dual-booting Windows and my Linux install is encrypted. Otherwise, it'd me more work to ensure my swap partition is encrypted.

asymptosis 6y ago
I recently ended up using swapfiles because I refactored my disk layout and it was just simpler to chuck the swap on an existing nvme partition. I realised an extra benefit: you can make it a sparse file, so it never even uses disk space at all until you actually need it, which could be never.
- muxator 6y ago
  "refactoring" means changing the (often implicit) structure of code without changing its external behaviour. In this case "reorganized" or "repartitioned" could probably be more apt. If we'd talked about software here (and not disk layout), I'd have considered less leniently the use of "refactoring", and a hint of unclear intentions. Sorry for being so nitpicky, but this is a sore thumb from too many "refactorings" that are, indeed, somenthing else.
  - asymptosis 6y ago
    Refactoring has the meaning that we impute to it. God didn't hand down golden tablets in the 60s with the words "thou shalt only use 'refactor' in these prescribed ways". Instead of being sorry for being nitpicky, maybe just keep your nitpicks to yourself in future.
- qiqitori 6y ago
  But if you suddenly need it and don't have it, you'll get segfaults (I assume). Sounds a bit too dangerous for me :p
  - NavinF 6y ago

skissane 6y ago

I was interested to see the code. Linus's message doesn't provide commit IDs, so you have to go looking yourself. I believe this is the commit which introduces the bug: https://github.com/torvalds/linux/commit/48d15436fde6feebcded7bd0fdc8ea4a9181b8fa https://github.com/torvalds/linux/commit/48d15436fde6feebcde... And I believe this is the commit which fixes it: https://github.com/torvalds/linux/commit/caf6912f3f4af7232340d500a4a2008f81b93f14 https://github.com/torvalds/linux/commit/caf6912f3f4af723234...

CyberRabbi 6y ago
Why do you believe that’s the commit that caused the bug?
- exikyut 6y ago
  The second commit self-disambiguates with "swap: fix swapfile read/write offset" and the commit message notes "fixes: 48d1543", which is the first linked commit.
  - CyberRabbi 6y ago
    Thanks for answering the question
- dgellow 6y ago
  Because of the commit message of the fix?
N48 6y ago
The fix makes me feel a little uneasy. It changes the behaviour of swap_page_sector() to take into account the offset. But swap_page_sector() was already used before the bug was introduced, a few lines earlier in the bdev_write_page() fast path, as introduced by this commit from 2014: https://github.com/torvalds/linux/commit/dd6bd0d9c7dbb3951005db4e424abbd37c4ff801 https://github.com/torvalds/linux/commit/dd6bd0d9c7dbb395100... So either: 1. The bdev_write_page() call was also broken, and has been for years, or 2. The bdev_write_page() uses the sector value differently, worked fine with the old swap_page_sector(), but is now broken with the new version, or 3. The change in swap_page_sector() somehow doesn't affect bdev_write_page(); maybe it's a noop when using swap files?

walrus01 6y ago

this is unlikely to hurt you unless you use a swap file, rather than a dedicated swap partition. for performance reasons most any modern linux installation would be using a dedicated swap partition, that is defined as swap-only during the install process with mkswap In my experience some of the reasons for using a swap file were more common maybe 10 or 15 years ago. Nowadays with disk space being less costly and rare, and systems having a lot more RAM (so the likelihood of going deep into swap is less), it's unsurprising that none of the testers encountered it. https://man7.org/linux/man-pages/man8/mkswap.8.html https://man7.org/linux/man-pages/man8/mkswap.8.html

ylyn 6y ago
Swapfiles are quite common when you use FDE (say, LUKS) and you don't want your swap to be in plaintext on disk. (Although, you could just have an entire LUKS-encrypted swap volume instead of a swapfile...)
- walrus01 6y ago
  most people I know who are paranoid to really go into the details of a FDE linux desktop/workstation environment choose to have no swap file, and at minimum 64GB of RAM. even safer if the swap doesn't exist.
- dathinab 6y ago
  While maybe not the most performant my favorite is LVM2 on top of a single large LUKS-encrypted partition (+no boot partition and a single (custom platform key) signed EFI bootable kernel blob containing the initram fs in the EFI partition). This: - Allows a encrypted swap partition (on in lvm2 on top of luks, sure not perfect but I don't really use swap) - Allows hibernation (which I don't really use tbh.) - Fully encrypted disc - Fast boot (compared to e.g. encrypted grub, at least last time I checked) - Easy resizing of partitions on demand - Which in turn makes it easier to use more partitions (for /, /home, etc.), which is necessary for taking advantage of certain mount flags like noexec for security. Anyway I think the reason Ubuntu uses a swap file is because it means you don't need to resize a partition when "upgrading" your RAM or similar (for hibernation to work). Through tbh. they should make the choice in the installer dependent of the hardware.

xpressvideoz 6y ago

> And, as far as I know, all the normal distributions set things up with swap partitions, not files I guess Ubuntu is not a "normal" distribution then? Because it uses a swap file by default.

dathinab 6y ago
While it's the default setup of the auto installer it common (and IMHO best) practice to change it to install with a swap partition. It's just that it's much easier for anyone who just gets started with Linux to have a swap file. For example for hibernation you swap partition needs to have the right size. But people do upgrade their desktop RAM and as such would need to resize the partition, which isn't easy (because likely it means shrinking another partition)... So for a distribution like ubuntu the "non-normal" case became the default. Still this mail was targeting people which install pre-release versions of Linux kernerls which is a group of people relatively unlikely to use a swap file.
- pbhjpbhj 6y ago
  I used to use a swap partition, but then if you get the partition size wrong you're lumbered with it. Why wouldn't you use a file. At one point in the past I used the default (or recommended) /boot size and got stuck a couple of years later juggling at updates to fit the boot files into the partition. I just use a monolithic partition per OS per disk now and swap files within that. Is that not good practice (for home systems)?
  - samus 6y ago
    `/home` should be separate from the system so you can reinstall it without trouble. Same for `/var` (or subdirectories) if you value the data of particular services. Other than that, the bootloader and EFI must be able to access certain directories like `/boot` and `/boot/efi`. As long as it works for your use cases, everything is alright. Edit: A swap file has a little bit of overhead because of the underlying file system. But at the point you notice this, your system is pretty much frozen anyways.

1996 6y ago

As no one did a TLDR: if you use a swapfile, it could overwrite part of the the filesystem where the swapfile exist. Don't use swapfiles if you can. Use a dedicated partition, there are many advantages (including automatic RAID0 if you have a swap partition on each drive)

travv0 6y ago
If you click the link, there's a TL;DR right up at the top, under the title.

kzrdude 6y ago

Automatic testing could have caught this, maybe.

noobermin 6y ago
The email mentions that normal testing didn't catch it as it's not a typical use-case for the devs apparently.
- gpvos 6y ago
  Which is exactly why automated testing exists, to catch cases like this.
  - _-___________-_ 6y ago
    If the devs foresaw the set of conditions that trigger the bug and wrote a test that produces those conditions.
  - unionpivo 6y ago
    Automated testing is not magic. it only tests things you make it test. And since kernel is basically involved into everything that computer does, you would have to test everything, to find every error. Linux kernel is also extremely customizable, so that it can run from watches, to mars rowers, to HPC and cloud. Linux kernels are continuously heavily tested on various different hardware architectures, by various organizations (RH, IBM, Oracle, Google). So there are automated tests done. It's just seems nobody tested this.
    - gpvos 6y ago
      Of course. Thanks for stating a lot of mostly obvious things. Assuming there is a test suite (I really don't know enough about Linux kernel testing, sorry), it is a matter of testing that on a representative set of configurations. Apparently no-one thought of having one with a swap file instead of a swap partition. That's okay, it's an understandable oversight, but one that clearly should be remedied.

fomine3 6y ago

This is the root cause of fs corruption that Phoronix reported. https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.12-rc2-Likely-Soon https://www.phoronix.com/scan.php?page=news_item&px=Linux-5....

akavel 6y ago

> it didn't even show up in normal testing, exactly because swapfiles just aren't normal. Uhh, does it mean swapfiles are undertested in the Linux kernel? Or is there some "non-normal" extra testing that is being run for non-rc releases, but not for rc ones? (!?)

exmadscientist 6y ago
I fear you grossly overestimate how detailed the kernel's testing is. (Do please note that the word "detailed" above was carefully chosen.)
- encryptluks2 6y ago
  While the Kernel team directly may not be as involved in testing the Linux Kernel to the extent that dedicated QA team would be, there are multiple billion dollar organizations like IBM (Red Hat), Oracle, etc that do a lot of automated testing. Given the amount of issues with Windows as well, especially for Insider builds, it doesn't make me lose trust in the Linux kernel anymore than I'd trust a Windows operating system. In fact, because it was caught before it would regular distro users it gives me more trust.
cmeacham98 6y ago
> Uhh, does it mean swapfiles are undertested in the Linux kernel? Or is there some "non-normal" extra testing that is being run for non-rc releases, but not for rc ones? (!?) My understanding is that is the point of RC builds, they are the builds used for testing before a stable release.

deleted 6y ago

[deleted]

londons_explore 6y ago

Why are swap files/partitions constant size? Most of the time, I don't need any swap at all, and any swap file/partition is wasted space. While hibernated, I need ~the amount of system ram as swap. While running some horrendous matlab script, I need about 1TB of swap. Yet linux won't dynamically allocate it like other files. I'm forced to constantly resize it manually to save space or be able to do more things.

megous 6y ago
Possibly because it's hard to guarantee re-allocating/sizing will not require more RAM during the process on any given filesystem...
dspillett 6y ago
I assume the argument to be that it adds complexity (out of swap, what do I do, grow it or something else, if set to grow what if there isn't enough space, if set to grow and there are multiple swap locations which do I grow, ...) that needs to be maintained & tested, but isn't commonly enough needed for someone working on the kernel to decide it warrants that effort, given there are workarounds (i.e. dynamically adding swap partitions/volumes/files) for most of the cases where it might be needed. > I need about 1TB of swap. I don't think most people[†] would ever want that much to be dynamically allocated. If something has needed that much memory unexpectedly then it is likely to be something like a massive leak or accidental fork bomb. It could quickly get to the point where a performance drop due to swap thrashing means you can't interact with the system to kill the offending process so your system is effectively locked until either hit the BRB or it runs out of the massive maximum storage allocated to swap. Of course that adds more complexity: how do you decide how much too much is? If it is a user decision and you sometimes want it to be able to balloon that much but still want the OOM killer to deal with accidentally leaky processes at other times, then you are back to manually managing swap allocation before/after processes that are expected to heavily use it. [†] not that it is wrong to have a use case that warrants it, but if there is a safe workaround for such unusual cases that complicates life just a little, then IMO that is preferable to the risk of adding complexity that will likely go relative untested because it isn't commonly used. > linux Of course other OSs have made different decisions on the matter, so maybe there is room for further discussion.

the_duke 6y ago

I used to assume that Linux had a huge test suite with hundreds of thousands of tests, given the crazy feature matrix that the kernel has to support. And that they would have started requiring tests for new features and bug fixes. Alas, that's not the case. There are a few external test suites like the Linux Test Project [1], but nothing that looks very extensive. The process seems to mostly rely on maintainers giving patches a good look, plus developers and first line users, including hardware manufacturers and corps like IBM or Red Hat, running their own software on kernel pre-releases and reporting the bugs. I realize that most of Linux is device drivers or code that is tightly coupled to hardware, which is more or less impossible to test without a huge test farm and lot's of manual labor, but it's still surprising to me that they don't do it for the more or less device independent functionality. This particular bug looks like it might have been caught by a relatively straight-forward swap file test case. [1] https://github.com/linux-test-project/ltp https://github.com/linux-test-project/ltp

ddevault 6y ago
It is important to remember that this is the first release candidate of a series which is usually 7 or 8 entries strong. It's pretty far removed from shipping. This is a testing procedure, though not one which looks the way you're used to. In general, Linux ships pretty reliable releases once the rc process is completed.
lumpa 6y ago
Comment rom "Helping Out with LTS Kernel Releases"[0]: "I find it amusing that the answer to "how can I help test" is "that people test the -rc releases when I announce them, and let me know if they work or not for their systems/workloads/tests/whatever". No smoke tests, no automated tools, no suggested workloads, no fuzzing, just "fool around and report back". There probably are lots of kernel testing tools around, but you wouldn't learn that from the linked article." It seems to be conscious choice, to the point of developers stating that "the process works". Which, looking at the results, it does seem to work. But wouldn't it work better with more tests, easily run by users wanting to help? I mean, if a downstream patch in related code regresses the issue, is there a new test added with this fix that would catch that? Edited to add: it seems that the situation is changing for the better, with gregkh saying in a comment: "`make kselftest` seems to do what you want today, if there are any gaps the kernel developers are glad to take more tests." [0] https://news.ycombinator.com/item?id=26021962 https://news.ycombinator.com/item?id=26021962 [1] https://lwn.net/Articles/848352/ https://lwn.net/Articles/848352/

JediPig 6y ago

unpopular opinion here... maybe if linus would fix kernel bugs ( i am looking at pulse audio right now ) even though it would break ABI, we would not have these long term issues. Userland is a utter mess, because of fragmentation and the insanely deceptive "rule" of no breaking userland. Linux has 26 million lines of BLOAT, CORPORATE BLOAT. Eventually linux will be considered to bloated to continue. As a hobbiest, I have no desire to work on the kernel, I may go work on a bsd kernel , cause they own both the kernel and userland. I am allowed to to change both, and make both better. It is why kqueue is vastly superior to epoll crap. Sincerly, A fat ass guy that has written more code than linus.

mschuster91 6y ago

I'd find it interesting to read an analysis how this bug could have happened from a technical point. Like... is swap file support going shortcuts in the kernel so that it can corrupt the file system the swap file is residing on? Because (at least in my hope) no matter of write() or whatever calls on a file handle should cause a filesystem corruption...

cesarb 6y ago
The swap file code does not go through the filesystem. It's the exact same code which is used for a swap partition or a swap file, and it writes directly to the block device (the partition). The main difference is that, if you have a swap file, the "swapon" call asks the filesystem to tell it which ranges in the block device contain the swap, instead of using the whole block device.
- bombcar 6y ago
  I did not know this - it makes sense, but it certainly a surprise.

egberts 6y ago

Yep, encrypted swap space is also an added configuration and that too got hosed bad in 5.12 so that’s an issue i’ve yet to file a bug for because i’m still stepping through the kdb. This is a dedicated data partition that LVM managed to procure a data volume that user land encryption performs on for a swap partition.

MR4D 6y ago

This is one of the best examples of leadership I’ve ever seen... Huge impactful problem.... Gets past the gatekeepers... Things go bad... Issue gets discovered & corrected.... Leader writes an honest and open note to explain the situation and actually apologizes. Corporate America could learn something from this.

pavon 6y ago

> Additionally, he is asking maintainers to not start branches from 5.12-rc1 to avoid future situations where people land in the buggy code while bisecting problems. Nice. Everywhere I've worked, once the fix has been committed the job is done. I should be more conscientious about preventing bad commits from being in branch histories like this.

sys_64738 6y ago

This is why life on the bleeding edge is sometimes very bloody.