6 ms·

That's really cool. When I tried WSL a while back, file system access was pretty slow especially for scanning large folder trees with many files (important for

by thorum 5y ago

That's really cool. When I tried WSL a while back, file system access was pretty slow especially for scanning large folder trees with many files (important for programming projects). Is this still an issue or has it improved in that area?

craigaloewen 5y ago
We improved the file system performance overall when we introduced WSL 2 by about 3-6x in general, and 20x for specific disk heavy operations. This still does bring its own challenges (You'll need to move your files to the Linux file system, and Cross OS file perf is slower in WSL 2) but we are working on these as well! :)
- Negitivefrags 5y ago
  From our perspective WSL2 only made file system performance worse. The caviet of moving files to the Linux file system is kind of a big one. We just want to Rsync some files from a Linux server to a windows build server without garbage performance and somehow there isn’t a way to do that still.
  - e12e 5y ago
    With native opensshd in windows, and a windows build of rsync - why would wsl be needed?
    - Negitivefrags 5y ago
      What windows build of Rsync are you referring to? I haven’t found one that doesn’t use Cygwin (slow filesystem).
      - e12e 5y ago
        I thought there was a mingw build, but apparently not. :/ You see that slow file access with cygwin? Last time I had such a use case for windows, I used unision - but isn't rsync, of course. https://www.cis.upenn.edu/~bcpierce/unison/ https://www.cis.upenn.edu/~bcpierce/unison/ Judging by this scoop.sh issue, there are no obvious not-cygwin builds/ports: https://github.com/lukesampson/scoop/issues/2295 https://github.com/lukesampson/scoop/issues/2295 Looks like there are libraries for rust and go - but not clients that are a drop in replacement (yet).
        Negitivefrags 5y ago
        Cygwin is faster than WSL2 but not faster than WSL1 from the last time we tested. Our builds involve syncing directories that have several hundred thousand files in them so the speed of stat() is really the bottleneck. I’ll give unison a look, I haven’t tried that. The libraries are an interesting idea, I’ll look into that too.
      - binarycrusader 5y ago
        Haven't tried this commercial client myself and I'm not endorsing it, but this one claims to be native rsync without cygwin: https://acrosync.com/windows.html https://acrosync.com/windows.html
        Negitivefrags 5y ago
        I’ve looked into it. It’s not really the right use case for pushing data to and from a build server. It’s more designed for a continuous backup situation with a GUI.
  - madeofpalk 5y ago
    > The caviet of moving files to the Linux file system is kind of a big one. Trying not to sound reductive, but either it's an issue for you or not. It's really going to depend on your use case which should be super apparent for you. I do all my web development within WSL2 and its great. I suspect this is the main use case they're optimising for.
eknkc 5y ago
WSL2 has its own filesystem (ext4 and everything). Accessing it is fast, accessing windows filesystem from wsl and vice versa is still slow. This is a big deal because, say you wanted to use intellij idea on wsl, using it on windows side and accessing the wsl filesystem does not really work. It is too slow. Now you can run it in linux, access the native fs. I imagine it would perform pretty good.
- voidfunc 5y ago
  Perf is decent, I already do this with x410 and IntelliJ. I have noticed some quirks tho with slow syncs between editor and actual file contents that ive never noticed on a host running just Linux
- _-david-_ 5y ago
  I haven't tried this yet, but would something like this https://blog.jetbrains.com/blog/2021/03/11/projector-is-out/ https://blog.jetbrains.com/blog/2021/03/11/projector-is-out/ help out?
- hnbad 5y ago
  FWIW this is why VSCode has the Remote extension, which includes support for running VSCode headless in WSL and connecting to that from Windows. Theoretically this change would eliminate the need for that but it's nice to have the same VSCode on Windows as in WSL.
jnwatson 5y ago
File system access definitely had issues for WSL 1. WSL 2, however, is much perkier. FYI, WSL 1 and WSL 2 are completely different architectures, so you have to explicitly upgrade your instances from 1 to 2.
ohthehugemanate 5y ago
That's an issue with WSL1, which translated system calls from Linux to Windows. Unfortunately the Linux and Windows filesystem drivers are radically different; many operations that are fast/cheap on linux (eg checking file metadata, which is stored in memory) are super exensive on the NT kernel (eg metadata involves reading the file from disk, and running any programs that have registered event handlers for file reads... like virus scanners and context menu items). So Git, for example, which on Linux keeps its state spread across thousands of files' metadata, was a dog. WSL2 takes a different approach: HyperV was always good at thin VMs... they took it to the next level and made WSL2 a super thin VM. Now performance is near native, depending on your benchmark. Bias: I work at MS in an unrelated area. I run Arch as my daily driver, but am damn impressed by the engineering in WSL2.
- forgotpwd16 5y ago
  Are those operations an issue on the NT kernel or the NTFS filesystem?
  - dataflow 5y ago
    The former.

by thorum 5y ago

craigaloewen 5y ago

We improved the file system performance overall when we introduced WSL 2 by about 3-6x in general, and 20x for specific disk heavy operations. This still does bring its own challenges (You'll need to move your files to the Linux file system, and Cross OS file perf is slower in WSL 2) but we are working on these as well! :)

Negitivefrags 5y ago
From our perspective WSL2 only made file system performance worse. The caviet of moving files to the Linux file system is kind of a big one. We just want to Rsync some files from a Linux server to a windows build server without garbage performance and somehow there isn’t a way to do that still.
- e12e 5y ago
  With native opensshd in windows, and a windows build of rsync - why would wsl be needed?
  - Negitivefrags 5y ago
    What windows build of Rsync are you referring to? I haven’t found one that doesn’t use Cygwin (slow filesystem).
    - e12e 5y ago
      I thought there was a mingw build, but apparently not. :/ You see that slow file access with cygwin? Last time I had such a use case for windows, I used unision - but isn't rsync, of course. https://www.cis.upenn.edu/~bcpierce/unison/ https://www.cis.upenn.edu/~bcpierce/unison/ Judging by this scoop.sh issue, there are no obvious not-cygwin builds/ports: https://github.com/lukesampson/scoop/issues/2295 https://github.com/lukesampson/scoop/issues/2295 Looks like there are libraries for rust and go - but not clients that are a drop in replacement (yet).
      - Negitivefrags 5y ago
        Cygwin is faster than WSL2 but not faster than WSL1 from the last time we tested. Our builds involve syncing directories that have several hundred thousand files in them so the speed of stat() is really the bottleneck. I’ll give unison a look, I haven’t tried that. The libraries are an interesting idea, I’ll look into that too.
    - binarycrusader 5y ago
      Haven't tried this commercial client myself and I'm not endorsing it, but this one claims to be native rsync without cygwin: https://acrosync.com/windows.html https://acrosync.com/windows.html
      - Negitivefrags 5y ago
        I’ve looked into it. It’s not really the right use case for pushing data to and from a build server. It’s more designed for a continuous backup situation with a GUI.
- madeofpalk 5y ago
  > The caviet of moving files to the Linux file system is kind of a big one. Trying not to sound reductive, but either it's an issue for you or not. It's really going to depend on your use case which should be super apparent for you. I do all my web development within WSL2 and its great. I suspect this is the main use case they're optimising for.

eknkc 5y ago

WSL2 has its own filesystem (ext4 and everything). Accessing it is fast, accessing windows filesystem from wsl and vice versa is still slow. This is a big deal because, say you wanted to use intellij idea on wsl, using it on windows side and accessing the wsl filesystem does not really work. It is too slow. Now you can run it in linux, access the native fs. I imagine it would perform pretty good.

voidfunc 5y ago
Perf is decent, I already do this with x410 and IntelliJ. I have noticed some quirks tho with slow syncs between editor and actual file contents that ive never noticed on a host running just Linux
_-david-_ 5y ago
I haven't tried this yet, but would something like this https://blog.jetbrains.com/blog/2021/03/11/projector-is-out/ https://blog.jetbrains.com/blog/2021/03/11/projector-is-out/ help out?
hnbad 5y ago
FWIW this is why VSCode has the Remote extension, which includes support for running VSCode headless in WSL and connecting to that from Windows. Theoretically this change would eliminate the need for that but it's nice to have the same VSCode on Windows as in WSL.

jnwatson 5y ago

File system access definitely had issues for WSL 1. WSL 2, however, is much perkier. FYI, WSL 1 and WSL 2 are completely different architectures, so you have to explicitly upgrade your instances from 1 to 2.

ohthehugemanate 5y ago

That's an issue with WSL1, which translated system calls from Linux to Windows. Unfortunately the Linux and Windows filesystem drivers are radically different; many operations that are fast/cheap on linux (eg checking file metadata, which is stored in memory) are super exensive on the NT kernel (eg metadata involves reading the file from disk, and running any programs that have registered event handlers for file reads... like virus scanners and context menu items). So Git, for example, which on Linux keeps its state spread across thousands of files' metadata, was a dog. WSL2 takes a different approach: HyperV was always good at thin VMs... they took it to the next level and made WSL2 a super thin VM. Now performance is near native, depending on your benchmark. Bias: I work at MS in an unrelated area. I run Arch as my daily driver, but am damn impressed by the engineering in WSL2.

forgotpwd16 5y ago
Are those operations an issue on the NT kernel or the NTFS filesystem?
- dataflow 5y ago
  The former.