4 ms·

WinFsp – FUSE for Windows – faster Than Dokany

35 points by billziss 10y ago

iokevins 10y ago
Title seems a bit off, per caveat below...why not use the original title, "WinFsp Performance Testing" (?) "The comparison to NTFS is very important to establish a baseline. It is also very misleading because NTFS is a disk file system and MEMFS (either the WinFsp or Dokany variants) is an in memory file system. The tests will show that MEMFS is faster than NTFS. This should not be taken to mean that we are trying to make the (obvious) claim that an in memory file system is faster than a disk file system, but to show that the approach of writing a file system in user mode is a valid proposition and can be efficient."
- billziss 10y ago
  Please suggest a more appropriate title and I will be happy to change it if it better conveys the results. Please also note that I want to convey that my solution is faster than Dokany, which is a competitor solution.
  - IsBisu 10y ago
    You should better say that YOUR MEMFS implementation is better on Winfsp that YOUR MEMFS implementation on Dokan. This is even not a far comparison.
loeg 10y ago
Fantastic to see a FUSE-compatible system for Windows. This has been something I've wanted for a long time as an occasional Windows user.
- kazinator 10y ago
  I could use this with SSH, like right now.
  - billziss 10y ago
    Here is my port of SSHFS to Windows: https://github.com/billziss-gh/sshfs-win https://github.com/billziss-gh/sshfs-win
  - IsBisu 10y ago
    SSHFS has been on windows since a moment with Dokan https://github.com/Foreveryone-cz/win-sshfs https://github.com/Foreveryone-cz/win-sshfs
ysleepy 10y ago
Super misleading title and benchmark. Why did they not use a RAM Drive for the NTFS test like one of these https://en.wikipedia.org/wiki/List_of_RAM_drive_software#Non-proprietary https://en.wikipedia.org/wiki/List_of_RAM_drive_software#Non... Are there any on-disk filesystem implementations based on WinFsp so a comparison can be made? I count this as intentionally misleading, since the submitter is also the Project author and performed the benchmark and should know better.
- billziss 10y ago
  I am the author of WinFsp and I believe you are missing the point or you did not actually read the article. Early on I say that: << This performance testing shows that WinFsp has excellent performance in all tested scenarios. It outperforms NTFS in most scenarios (an unfair comparison as NTFS is a disk file system and WinFsp is tested with an in-memory file system). It also outperforms Dokany in all scenarios, often by an order of magnitude. >> My intent was to measure performance of WinFsp against NTFS in order to establish a baseline and to show that building file systems in user mode is not as inefficient as people think. You should also note that a hacker news submission is only 80 characters and it is hard to clearly communicate one's intentions fully.
  - ysleepy 10y ago
    I read the article, and the quoted sentence. This is the reason why I find your HN submission title intentionally misleading. It's a cool project, and this deception is entirely unnecessary. Why not claim "FASTER than Dokany". An interesting comparison would be ntfs-3g on WinFst vs. native NTFS either both on RAM drive or Disk. You also did not answer any of the questions in my comment.
    - billziss 10y ago
      I will change the title to say "Faster than Dokany" without mentioning NTFS. The deception is not intentional. For me it was important to verify that my project is not slower than NTFS when using my test file system, which is an in-memory file system. You may notice that Dokany is actually slower or on par with NTFS in many scenarios even with an in memory file system. Trying NTFS on a RAM disk is a good idea. I simply did not think of it. There is no on disk implementation of WinFsp that I can share at this moment. Please let me know if you have other questions.
IsBisu 10y ago
You say in your bench that Dokan support does not support memory mapped files but it does as I can see with fsx (memory mapped read/write enabled) running on their CI https://ci.appveyor.com/project/Maxhy/dokany/build/1.0.1-48/job/axakvm0u2tqqgf6a#L1086 https://ci.appveyor.com/project/Maxhy/dokany/build/1.0.1-48/... It is your implementation that doesn't support it. You add lock (that will slow down the result) on Dokan implementation only https://github.com/billziss-gh/memfs-dokany/commit/03ebaebb1d5f53823fa23709cc9e710ff4f0e540 https://github.com/billziss-gh/memfs-dokany/commit/03ebaebb1... After that you see that dokan is faster to list files (https://github.com/billziss-gh/winfsp/commit/595a77bd2e45347809ed46ab419a1a0df8e11b2e#diff-499e702a5a0648c27ee9e9e622f71521R109 https://github.com/billziss-gh/winfsp/commit/595a77bd2e45347...) You improve YOUR MEMFS winfsp implementation https://github.com/billziss-gh/winfsp/commit/f7ca9f05221040a5a88df33d8d58b87b21c1e863 https://github.com/billziss-gh/winfsp/commit/f7ca9f05221040a... but not on dokan ? You also improve memory allocation on winfsp memfs implementation and again not on dokan https://github.com/billziss-gh/winfsp/commit/d12234bb01b3620b4fc201b9a197ef48cbefe359 https://github.com/billziss-gh/winfsp/commit/d12234bb01b3620... And last point...running it on VirtualBox O_o ? Even running it 5x time and take the best time. You cannot be sure that virtualbox has not corrupt your test. For me a benchmark on a VM is not reliable at any point. My point is even if Winfsp is fast on your test with an unfair FS, I am pretty sure there is not difference with real applications on a non-virtual machine.
- billziss 10y ago
  > You say in your bench that Dokan support does not support memory mapped files but it does as I can see with fsx (memory mapped read/write enabled) It was me that ported FSX to windows so I am quite familiar with this test. Here is the code that fails in fsbench: Mapping = CreateFileMappingW(Handle, 0, PAGE_READWRITE, 0, FileSize, 0); I would be happy if you explained to me how Dokany/MEMFS fails this code. > You add lock (that will slow down the result) on Dokan implementation only https://github.com/billziss-gh/memfs-dokany/commit/03ebaebb1.. https://github.com/billziss-gh/memfs-dokany/commit/03ebaebb1.... The lock exists on WinFsp/MEMFS. It is just taken in a different place: https://github.com/billziss-gh/winfsp/blob/master/src/dll/fsop.c#L60 > After that you see that dokan is faster to list files (https://github.com/billziss-gh/winfsp/commit/595a77bd2e45347... https://github.com/billziss-gh/winfsp/commit/595a77bd2e45347...) You improve YOUR MEMFS winfsp implementation https://github.com/billziss-gh/winfsp/commit/f7ca9f05221040a.. https://github.com/billziss-gh/winfsp/commit/f7ca9f05221040a.... but not on dokan ? The WinFsp/MEMFS had indeed a problem which I fixed, but it was not WinFsp related (it was MEMFS related - aka my test user mode file system). I attempted to apply the same fix to Dokany/MEMFS, but I could not because Dokany does not pass an Offset parameter to its FindFiles operation. > You also improve memory allocation on winfsp memfs implementation and again not on dokan https://github.com/billziss-gh/winfsp/commit/d12234bb01b3620.. https://github.com/billziss-gh/winfsp/commit/d12234bb01b3620.... Actually that is incorrect. https://github.com/billziss-gh/memfs-dokany/blob/master/memfs-dokany.cpp#L88 > My point is even if Winfsp is fast on your test with an unfair FS, I am pretty sure there is not difference with real applications on a non-virtual machine. No. WinFsp is faster, because it is faster. Feel free to send me any pull requests that you want for Dokany/MEMFS and I will incorporate them into the project and re-run the tests.
lamastronaute 10y ago
OK since it is the author of Winfsp that has made memfs of dokan and winfsp and also the tools used to bench/compare them, I wanted to be sure to have the same result on my side using this tool. For now, I have been able to make it work on NTFS but when using on winfsp/memfs-dokany it just crash >< I feel very weird about this article and that I am losing my time here.
- billziss 10y ago
  MEMFS limits how many files you can create (up to 1024 by default). Try running it with the parameter -n set to 100000, which will allow you to create up to 100000 files. Examples command lines: WinFsp: memfs -t 0 -n 100000 -m Z: # no caching memfs -t 1 -n 100000 -m Z: # enables metadata caching memfs -t -1 -n 100000 -m Z: # enables data and metadata caching Dokany: memfs-dokany -n 100000 -m Z:
deleted 10y ago
[deleted]
sngan 10y ago
winfsp performance numbers look pretty good. if the implementation is stable too, a clear winner here and i will give it a try.
- billziss 10y ago
  Thanks. The implementation is very stable because WinFsp is tested regularly and rigorously. Please read about the WinFsp Testing Strategy here: https://github.com/billziss-gh/winfsp/wiki/WinFsp-Testing https://github.com/billziss-gh/winfsp/wiki/WinFsp-Testing

35 points by billziss 10y ago

Title seems a bit off, per caveat below...why not use the original title, "WinFsp Performance Testing" (?) "The comparison to NTFS is very important to establish a baseline. It is also very misleading because NTFS is a disk file system and MEMFS (either the WinFsp or Dokany variants) is an in memory file system. The tests will show that MEMFS is faster than NTFS. This should not be taken to mean that we are trying to make the (obvious) claim that an in memory file system is faster than a disk file system, but to show that the approach of writing a file system in user mode is a valid proposition and can be efficient."

billziss 10y ago
Please suggest a more appropriate title and I will be happy to change it if it better conveys the results. Please also note that I want to convey that my solution is faster than Dokany, which is a competitor solution.
- IsBisu 10y ago
  You should better say that YOUR MEMFS implementation is better on Winfsp that YOUR MEMFS implementation on Dokan. This is even not a far comparison.

loeg 10y ago

Fantastic to see a FUSE-compatible system for Windows. This has been something I've wanted for a long time as an occasional Windows user.

kazinator 10y ago
I could use this with SSH, like right now.
- billziss 10y ago
  Here is my port of SSHFS to Windows: https://github.com/billziss-gh/sshfs-win https://github.com/billziss-gh/sshfs-win
- IsBisu 10y ago
  SSHFS has been on windows since a moment with Dokan https://github.com/Foreveryone-cz/win-sshfs https://github.com/Foreveryone-cz/win-sshfs

ysleepy 10y ago

Super misleading title and benchmark. Why did they not use a RAM Drive for the NTFS test like one of these https://en.wikipedia.org/wiki/List_of_RAM_drive_software#Non-proprietary https://en.wikipedia.org/wiki/List_of_RAM_drive_software#Non... Are there any on-disk filesystem implementations based on WinFsp so a comparison can be made? I count this as intentionally misleading, since the submitter is also the Project author and performed the benchmark and should know better.

billziss 10y ago
I am the author of WinFsp and I believe you are missing the point or you did not actually read the article. Early on I say that: << This performance testing shows that WinFsp has excellent performance in all tested scenarios. It outperforms NTFS in most scenarios (an unfair comparison as NTFS is a disk file system and WinFsp is tested with an in-memory file system). It also outperforms Dokany in all scenarios, often by an order of magnitude. >> My intent was to measure performance of WinFsp against NTFS in order to establish a baseline and to show that building file systems in user mode is not as inefficient as people think. You should also note that a hacker news submission is only 80 characters and it is hard to clearly communicate one's intentions fully.
- ysleepy 10y ago
  I read the article, and the quoted sentence. This is the reason why I find your HN submission title intentionally misleading. It's a cool project, and this deception is entirely unnecessary. Why not claim "FASTER than Dokany". An interesting comparison would be ntfs-3g on WinFst vs. native NTFS either both on RAM drive or Disk. You also did not answer any of the questions in my comment.
  - billziss 10y ago
    I will change the title to say "Faster than Dokany" without mentioning NTFS. The deception is not intentional. For me it was important to verify that my project is not slower than NTFS when using my test file system, which is an in-memory file system. You may notice that Dokany is actually slower or on par with NTFS in many scenarios even with an in memory file system. Trying NTFS on a RAM disk is a good idea. I simply did not think of it. There is no on disk implementation of WinFsp that I can share at this moment. Please let me know if you have other questions.

IsBisu 10y ago

You say in your bench that Dokan support does not support memory mapped files but it does as I can see with fsx (memory mapped read/write enabled) running on their CI https://ci.appveyor.com/project/Maxhy/dokany/build/1.0.1-48/job/axakvm0u2tqqgf6a#L1086 https://ci.appveyor.com/project/Maxhy/dokany/build/1.0.1-48/... It is your implementation that doesn't support it. You add lock (that will slow down the result) on Dokan implementation only https://github.com/billziss-gh/memfs-dokany/commit/03ebaebb1d5f53823fa23709cc9e710ff4f0e540 https://github.com/billziss-gh/memfs-dokany/commit/03ebaebb1... After that you see that dokan is faster to list files (https://github.com/billziss-gh/winfsp/commit/595a77bd2e45347809ed46ab419a1a0df8e11b2e#diff-499e702a5a0648c27ee9e9e622f71521R109 https://github.com/billziss-gh/winfsp/commit/595a77bd2e45347...) You improve YOUR MEMFS winfsp implementation https://github.com/billziss-gh/winfsp/commit/f7ca9f05221040a5a88df33d8d58b87b21c1e863 https://github.com/billziss-gh/winfsp/commit/f7ca9f05221040a... but not on dokan ? You also improve memory allocation on winfsp memfs implementation and again not on dokan https://github.com/billziss-gh/winfsp/commit/d12234bb01b3620b4fc201b9a197ef48cbefe359 https://github.com/billziss-gh/winfsp/commit/d12234bb01b3620... And last point...running it on VirtualBox O_o ? Even running it 5x time and take the best time. You cannot be sure that virtualbox has not corrupt your test. For me a benchmark on a VM is not reliable at any point. My point is even if Winfsp is fast on your test with an unfair FS, I am pretty sure there is not difference with real applications on a non-virtual machine.

billziss 10y ago
> You say in your bench that Dokan support does not support memory mapped files but it does as I can see with fsx (memory mapped read/write enabled) It was me that ported FSX to windows so I am quite familiar with this test. Here is the code that fails in fsbench: Mapping = CreateFileMappingW(Handle, 0, PAGE_READWRITE, 0, FileSize, 0); I would be happy if you explained to me how Dokany/MEMFS fails this code. > You add lock (that will slow down the result) on Dokan implementation only https://github.com/billziss-gh/memfs-dokany/commit/03ebaebb1.. https://github.com/billziss-gh/memfs-dokany/commit/03ebaebb1.... The lock exists on WinFsp/MEMFS. It is just taken in a different place: https://github.com/billziss-gh/winfsp/blob/master/src/dll/fsop.c#L60 > After that you see that dokan is faster to list files (https://github.com/billziss-gh/winfsp/commit/595a77bd2e45347... https://github.com/billziss-gh/winfsp/commit/595a77bd2e45347...) You improve YOUR MEMFS winfsp implementation https://github.com/billziss-gh/winfsp/commit/f7ca9f05221040a.. https://github.com/billziss-gh/winfsp/commit/f7ca9f05221040a.... but not on dokan ? The WinFsp/MEMFS had indeed a problem which I fixed, but it was not WinFsp related (it was MEMFS related - aka my test user mode file system). I attempted to apply the same fix to Dokany/MEMFS, but I could not because Dokany does not pass an Offset parameter to its FindFiles operation. > You also improve memory allocation on winfsp memfs implementation and again not on dokan https://github.com/billziss-gh/winfsp/commit/d12234bb01b3620.. https://github.com/billziss-gh/winfsp/commit/d12234bb01b3620.... Actually that is incorrect. https://github.com/billziss-gh/memfs-dokany/blob/master/memfs-dokany.cpp#L88 > My point is even if Winfsp is fast on your test with an unfair FS, I am pretty sure there is not difference with real applications on a non-virtual machine. No. WinFsp is faster, because it is faster. Feel free to send me any pull requests that you want for Dokany/MEMFS and I will incorporate them into the project and re-run the tests.

lamastronaute 10y ago

OK since it is the author of Winfsp that has made memfs of dokan and winfsp and also the tools used to bench/compare them, I wanted to be sure to have the same result on my side using this tool. For now, I have been able to make it work on NTFS but when using on winfsp/memfs-dokany it just crash >< I feel very weird about this article and that I am losing my time here.

billziss 10y ago
MEMFS limits how many files you can create (up to 1024 by default). Try running it with the parameter -n set to 100000, which will allow you to create up to 100000 files. Examples command lines: WinFsp: memfs -t 0 -n 100000 -m Z: # no caching memfs -t 1 -n 100000 -m Z: # enables metadata caching memfs -t -1 -n 100000 -m Z: # enables data and metadata caching Dokany: memfs-dokany -n 100000 -m Z:

deleted 10y ago

[deleted]

sngan 10y ago

winfsp performance numbers look pretty good. if the implementation is stable too, a clear winner here and i will give it a try.

billziss 10y ago
Thanks. The implementation is very stable because WinFsp is tested regularly and rigorously. Please read about the WinFsp Testing Strategy here: https://github.com/billziss-gh/winfsp/wiki/WinFsp-Testing https://github.com/billziss-gh/winfsp/wiki/WinFsp-Testing