10 ms·

File system corruption is unforgivable mistake, but FS work is really hard to get right. Even stuff like VMware corrupts shared folders ... wherever they try to

by harikb 6y ago

File system corruption is unforgivable mistake, but FS work is really hard to get right. Even stuff like VMware corrupts shared folders ... wherever they try to bypass a driver translation layer. Just Google for “vmware shared folder corruption”

rleigh 6y ago
Is this the reason why they have deprecated shared folders in Workstation 16?
- Qerub 6y ago
  I don't think they deprecated shared folders, only shared VMs (a function that enables Workstation to act as a virtualization server). Source: https://en.wikipedia.org/wiki/VMware_Workstation#Version_history https://en.wikipedia.org/wiki/VMware_Workstation#Version_his...
  - nikanj 6y ago
    That’s a shame. I have a high-powered desktop and sometimes it’s nice to work from the patio by opening a few VMs on my laptop. I get the oomph of the big box with the mobility of the laptop. To be fair, this feature always felt..rickety. But it was very nice.
    - wila 6y ago
      You can still use remote desktop for accessing remote VMs. You would be missing the remote power operations. FWIW, I am working on a product called Vimarun [0] that is aimed to replace most of that missing functionality over time. No remote power operations yet, but that will come. [0] https://vimarun.com https://vimarun.com
      - nikanj 6y ago
        I'd also be missing the ability to plug in USB devices, which is very central to the work I do. RDP has some device passover support, but it's not even close to being able to replace vmware's USB support.
        wila 6y ago
        You are talking about redirecting USB via workstation as well? eg. connecting USB devices remotely to your VM? (asking as normal USB pass-through doesn't work for a shared VM, see [0]) I hadn't even considered that. Connecting USB devices remotely works well with vSphere, but never tried that with Workstation. They are going to completely removing the hostd engine from VMware Workstation. That would indeed also include that part and it is most likely not easy to get that working without hostd. [0] http://kb.vmware.com/kb/2005585 http://kb.vmware.com/kb/2005585
        acct776 6y ago
        Qemu/Spice
        wila 6y ago
        Not a bad suggestion, but the host running the VM would have to run Linux in that case, not Windows.
dmw_ng 6y ago
Reminds me of an infuriating old bug in VirtualBox where it wouldn't notice the size of a file had changed. Devs were working in some Windows editor then using Git within VirtualBox to commit changes, resulting in inexplicable trailing nulls and garbage turning up in the repo Took quite some time to figure out what was causing it. "Everyone is somehow corrupting files except for me, wtf?" Magic filesystem translation layers always suck
- ficklepickle 6y ago
  I'm now in the habit of doing a git diff before every commit. Even if I don't thoroughly read the diff, I at least skim it for sanity. I can't recall what made me start doing that, but now it's a habit.
  - withinboredom 6y ago
    I do git add -up Same thing, sorta but all in one go.
sys_64738 6y ago
FS corruption is the worst as you lose total confidence in the product. This is a mega process escape that the WSL team would need to transparently detail why it happened, what the remedy is, and why it'll never, ever happen again.
- AaronFriel 6y ago
  My guess is it's the result of shutting down the virtual machine that runs the Linux kernel too soon, leaving writes to the virtual hard disk in an inconsistent state. This coincides with "shutdown /r /t 0" being able to cause it as well as blue screens or power loss. And explains why I've never seen it, despite using WSL2 on insider preview builds on multiple machines daily: I almost never shut down, only for updates and new builds.
  - admax88q 6y ago
    The kernel should and can handle unexpected ahutdowns without corruption. Data loss sometimes, but not corruption.
    - ylyn 6y ago
      This is data loss. A few object files in the Git directory are truncated. Which results in a corrupted repository, from Git's point of view.
      - TylerE 6y ago
        That’s corruption. Data loss would indicate an entire fs transaction getting dropped entirely - but not breaking the principle of atomic ops. A write should either happen entirely or not at all.
        stretchcat 6y ago
        Does git actually attempt to write all objects atomically?
Someone 6y ago
Is this file system corruption? The article isn’t 100% clear it ‘only’ loses some data writes, but it also doesn’t explicitly say files not being written are affected, directory structures are corrupted, etc. So, to me, it looks like WSL2 not completely flushing writes to the underlying file system. Bad, but not as bad as file system corruption (which could lead to losing all data on the disk)
- Gibbon1 6y ago
  Not my area but I seem to remember bitches that Linux lies about fsync. As in it'll swear up and down that it flushed everything to disk, but it's lying. Also over the years it seems like everyone I've seen that habitually edits files remotely ends up with this sort of pain and butthurt.
- cmurf 6y ago
  There's no enough information to know if all the reported problems are the result of the same defect. But in: https://github.com/microsoft/WSL/issues/5895 https://github.com/microsoft/WSL/issues/5895 The first instance of a problem is: [ 1.956835] JBD2: Invalid checksum recovering block 97441 in log And that's corruption that leads to log replay failing, i.e. rejecting it because honoring the replay in the face of checksum errors could make things much worse. Subsequently mount fails: [ 21.151232] ERROR: MountExt4:1659: mount(/dev/sdb) failed 5 That's good because the purpose of journal replay is to make the file system consistent following a crash/power fail. And if the file system is dirty, replay is called for, but can't happen due to a corrupt journal, so now an fsck is required. i.e. it is in an inconsistent (you could say partly broken) state and needs repair. I haven't seen syslog/systemd journal for other cases to know if there's instances of ext4 log replay that succeeds, but with missing files. That's not file system corruption, even if it leads to an inconsistent state in a git repository (or even a database). But this is still concerning, because to get a situation where log replay is clean but files are missing suggests an entire transaction was just dropped. It never made it to stable media, and even the metadata was not partially written to the ext4 journal. qemu-kvm has a (host) cache setting called "unsafe". Default is typically "none" or "write back". The unsafe mode can result in file system corruption if the host crashes or has a power failure. The guest's IO is faster with this mode, but the write ordering expected by the file system is not guaranteed if the host crashes. i.e. writes can hit stable media out of order. If the guest crashes, my experience has been that things are fine - subsequent log replay (in the guest) is successful, because the guest writes that made it to the host cache do make it to stable media by the same the guest reboots. The out of order writes don't matter... unless the host crashes, and then it's a big problem. The other qemu cache modes have rather different flush/fua policies that can still keep a guest file system consistent following a host crash. But they are slower. So it makes me suspicious that for performance reasons, WSL2 might be using a possibly volatile host side caching policy. Merely for additional data point, it might be interesting to try to reproduce this problem using e.g. Btrfs for the guest file system. If write order is honored and flushed to stable media appropriate for default out of the box configuration of a VM, I'd expect Btrfs never complains, but might drop up to 30s of writes. But if there's out of order writes making it to stable media, Btrfs will also complain, I'd expect transid errors which are also a hallmark of drive firmware not consistently honoring flush/fua and then you get a badly timed crash. (And similar for ZFS for that matter - nothing is impervious to having its write order expectations blown up.)
wila 6y ago
Interesting. I basically live at the VMware community forums and could not remember this issue... FWIW I am a user moderator down there and VMware desktop products (Workstation/Fusion/Player) has my focus and as a result I basically read almost every post that would report this issue. I just DDG'ed it and see one report from 2014 [0], ok some more from around 2008 when using google. Looks like this was resolved in 2015 as that was the last time I see it being mentioned. [0] https://communities.vmware.com/thread/485062 https://communities.vmware.com/thread/485062
smileybarry 6y ago
My experience was locked files or lost shared folders (VM doesn't see them until you say the magic words, aka randomly stop and start services until it works). Both happened so (relatively) frequently that I just lost confidence in the feature until version 16 where read-only folders works fine. (I stopped using it in version 9)
mehrdadn 6y ago
I could swear I ran into that corruption (or a similar one?) almost a decade ago. They still haven't fixed it?! It was quite reproducible too, it just happened when I transferred a large file...

by harikb 6y ago

rleigh 6y ago

Is this the reason why they have deprecated shared folders in Workstation 16?

Qerub 6y ago
I don't think they deprecated shared folders, only shared VMs (a function that enables Workstation to act as a virtualization server). Source: https://en.wikipedia.org/wiki/VMware_Workstation#Version_history https://en.wikipedia.org/wiki/VMware_Workstation#Version_his...
- nikanj 6y ago
  That’s a shame. I have a high-powered desktop and sometimes it’s nice to work from the patio by opening a few VMs on my laptop. I get the oomph of the big box with the mobility of the laptop. To be fair, this feature always felt..rickety. But it was very nice.
  - wila 6y ago
    You can still use remote desktop for accessing remote VMs. You would be missing the remote power operations. FWIW, I am working on a product called Vimarun [0] that is aimed to replace most of that missing functionality over time. No remote power operations yet, but that will come. [0] https://vimarun.com https://vimarun.com
    - nikanj 6y ago
      I'd also be missing the ability to plug in USB devices, which is very central to the work I do. RDP has some device passover support, but it's not even close to being able to replace vmware's USB support.
      - wila 6y ago
        You are talking about redirecting USB via workstation as well? eg. connecting USB devices remotely to your VM? (asking as normal USB pass-through doesn't work for a shared VM, see [0]) I hadn't even considered that. Connecting USB devices remotely works well with vSphere, but never tried that with Workstation. They are going to completely removing the hostd engine from VMware Workstation. That would indeed also include that part and it is most likely not easy to get that working without hostd. [0] http://kb.vmware.com/kb/2005585 http://kb.vmware.com/kb/2005585
      - acct776 6y ago
        Qemu/Spice
        wila 6y ago
        Not a bad suggestion, but the host running the VM would have to run Linux in that case, not Windows.

dmw_ng 6y ago

Reminds me of an infuriating old bug in VirtualBox where it wouldn't notice the size of a file had changed. Devs were working in some Windows editor then using Git within VirtualBox to commit changes, resulting in inexplicable trailing nulls and garbage turning up in the repo Took quite some time to figure out what was causing it. "Everyone is somehow corrupting files except for me, wtf?" Magic filesystem translation layers always suck

ficklepickle 6y ago
I'm now in the habit of doing a git diff before every commit. Even if I don't thoroughly read the diff, I at least skim it for sanity. I can't recall what made me start doing that, but now it's a habit.
- withinboredom 6y ago
  I do git add -up Same thing, sorta but all in one go.

sys_64738 6y ago

FS corruption is the worst as you lose total confidence in the product. This is a mega process escape that the WSL team would need to transparently detail why it happened, what the remedy is, and why it'll never, ever happen again.

AaronFriel 6y ago
My guess is it's the result of shutting down the virtual machine that runs the Linux kernel too soon, leaving writes to the virtual hard disk in an inconsistent state. This coincides with "shutdown /r /t 0" being able to cause it as well as blue screens or power loss. And explains why I've never seen it, despite using WSL2 on insider preview builds on multiple machines daily: I almost never shut down, only for updates and new builds.
- admax88q 6y ago
  The kernel should and can handle unexpected ahutdowns without corruption. Data loss sometimes, but not corruption.
  - ylyn 6y ago
    This is data loss. A few object files in the Git directory are truncated. Which results in a corrupted repository, from Git's point of view.
    - TylerE 6y ago
      That’s corruption. Data loss would indicate an entire fs transaction getting dropped entirely - but not breaking the principle of atomic ops. A write should either happen entirely or not at all.
      - stretchcat 6y ago
        Does git actually attempt to write all objects atomically?

Someone 6y ago

Is this file system corruption? The article isn’t 100% clear it ‘only’ loses some data writes, but it also doesn’t explicitly say files not being written are affected, directory structures are corrupted, etc. So, to me, it looks like WSL2 not completely flushing writes to the underlying file system. Bad, but not as bad as file system corruption (which could lead to losing all data on the disk)

Gibbon1 6y ago
Not my area but I seem to remember bitches that Linux lies about fsync. As in it'll swear up and down that it flushed everything to disk, but it's lying. Also over the years it seems like everyone I've seen that habitually edits files remotely ends up with this sort of pain and butthurt.
cmurf 6y ago
There's no enough information to know if all the reported problems are the result of the same defect. But in: https://github.com/microsoft/WSL/issues/5895 https://github.com/microsoft/WSL/issues/5895 The first instance of a problem is: [ 1.956835] JBD2: Invalid checksum recovering block 97441 in log And that's corruption that leads to log replay failing, i.e. rejecting it because honoring the replay in the face of checksum errors could make things much worse. Subsequently mount fails: [ 21.151232] ERROR: MountExt4:1659: mount(/dev/sdb) failed 5 That's good because the purpose of journal replay is to make the file system consistent following a crash/power fail. And if the file system is dirty, replay is called for, but can't happen due to a corrupt journal, so now an fsck is required. i.e. it is in an inconsistent (you could say partly broken) state and needs repair. I haven't seen syslog/systemd journal for other cases to know if there's instances of ext4 log replay that succeeds, but with missing files. That's not file system corruption, even if it leads to an inconsistent state in a git repository (or even a database). But this is still concerning, because to get a situation where log replay is clean but files are missing suggests an entire transaction was just dropped. It never made it to stable media, and even the metadata was not partially written to the ext4 journal. qemu-kvm has a (host) cache setting called "unsafe". Default is typically "none" or "write back". The unsafe mode can result in file system corruption if the host crashes or has a power failure. The guest's IO is faster with this mode, but the write ordering expected by the file system is not guaranteed if the host crashes. i.e. writes can hit stable media out of order. If the guest crashes, my experience has been that things are fine - subsequent log replay (in the guest) is successful, because the guest writes that made it to the host cache do make it to stable media by the same the guest reboots. The out of order writes don't matter... unless the host crashes, and then it's a big problem. The other qemu cache modes have rather different flush/fua policies that can still keep a guest file system consistent following a host crash. But they are slower. So it makes me suspicious that for performance reasons, WSL2 might be using a possibly volatile host side caching policy. Merely for additional data point, it might be interesting to try to reproduce this problem using e.g. Btrfs for the guest file system. If write order is honored and flushed to stable media appropriate for default out of the box configuration of a VM, I'd expect Btrfs never complains, but might drop up to 30s of writes. But if there's out of order writes making it to stable media, Btrfs will also complain, I'd expect transid errors which are also a hallmark of drive firmware not consistently honoring flush/fua and then you get a badly timed crash. (And similar for ZFS for that matter - nothing is impervious to having its write order expectations blown up.)

wila 6y ago

Interesting. I basically live at the VMware community forums and could not remember this issue... FWIW I am a user moderator down there and VMware desktop products (Workstation/Fusion/Player) has my focus and as a result I basically read almost every post that would report this issue. I just DDG'ed it and see one report from 2014 [0], ok some more from around 2008 when using google. Looks like this was resolved in 2015 as that was the last time I see it being mentioned. [0] https://communities.vmware.com/thread/485062 https://communities.vmware.com/thread/485062

smileybarry 6y ago

My experience was locked files or lost shared folders (VM doesn't see them until you say the magic words, aka randomly stop and start services until it works). Both happened so (relatively) frequently that I just lost confidence in the feature until version 16 where read-only folders works fine. (I stopped using it in version 9)

mehrdadn 6y ago

I could swear I ran into that corruption (or a similar one?) almost a decade ago. They still haven't fixed it?! It was quite reproducible too, it just happened when I transferred a large file...