4 ms·

> NTFS is really horrible handling many small files. To pile onto NTFS, it's performance is so notoriously bad that there are developer teams working on Window

> NTFS is really horrible handling many small files. To pile onto NTFS, it's performance is so notoriously bad that there are developer teams working on Windows projects that configure their build farm to do cross builds from Linux to Windows just to avoid the performance penalty.

ta23948234 2y ago
As an anecdote, we had a really long build time for our pipeline (going back prob 15 years). I argued for a Linux laptop, and the boss said, "OK, prove it. Here's two equivalent laptops, time it.". Turns out there was zero difference, or negligible (Windows won), between compilation times. That has always annoyed me.
- rapind 2y ago
  Love that your boss was open to the idea but also wanted data to back it up. (imagine a slow clap)
  - ta9345908345098 2y ago
    Yep, was a good experience. I'm sure with fine tuning it (Linux) could've been better, but I ate that humble pie.
  - goodpoint 2y ago
    A single datapoint from an experiment done by only one person you call it data? What's not to love...
    - tssva 2y ago
      It was the only data point which mattered for the decision that needed to be made.
- chipdart 2y ago
  > Turns out there was zero difference, or negligible (Windows won), between compilation times. I think there was something seriously flawed in your test. If you Google for a minute, you find multiple posts on how moving the same builds to Linux led to performance improvements in the range of 40% drops in build times. Some anecdotes even compare doing the same builds in Ubunto with NTFS to see double-digit gains. NTFS is notoriously awful in scenarios involving reading/writing many small projects. This is the bottleneck in Windows builds. There is a myriad of benchmarks documenting this problem. Nowadays there are plenty of cross-platform projects to serve as benchmarks. Checking this can be as easy as checking out a project, start a full rebuild, and check how long it takes.
  - bee_rider 2y ago
    It seems possible that their pipeline just had large files, or something like that.
  - psd1 2y ago
    That's survivor bias. Fewer people blog about unsuccessful initiatives.
    - chipdart 2y ago
      > That's survivor bias. Fewer people blog about unsuccessful initiatives. I don't understand what point you're trying to make. Should we give equal credit to unsuccessful initiatives when their failure is due to screwing up a critical part of the project? I mean, the successful attempts document what is possible. What do you expect to report when you couldn't even manage to get something working?
      - psd1 2y ago
        Say there are N initiatives to move to cross-compilation. Of those, there are M that deliver the hoped-for benefits, where M <= N. But practically, M < N significantly, say, M < 0.9 N. Assume bloggers blog mainly about content that contains a positive message. I'm asserting that people blog more readily about their success than their failures. So when you look at the blog literature, your population is not N, it is M. You don't see the failures because they don't tell the tale.
- ta456456456456 2y ago
  Yeah sorry, I left out a lot of details for a quick anecdote. It was Java compilation, so the different JDK's may've made a difference (almost certainly). But I figured the better file system handling etc etc would've gone in favour of Linux, versus Windows. In any case it wasn't much of a fun codebase. But I think a good lesson was, always test it, always measure. Not casting shade on either OS.
  - freedomben 2y ago
    Do you remember which file system you used on Linux?
- zamadatix 2y ago
  For the most part the two are pretty equivalent in performance, it's particularly constant blocking on small file IO where Windows falls way behind. Also a bit in the "tons of cores" and "tons of networking" categories but not as nearly as bad as small file IO.
  - account42 2y ago
    Well yeah, if you avoid the OS (syscalls) as much as possible then all operating systems will have about the same performance.
- manishsharan 2y ago
  You can't have windows laptop without running a ton of Antivirus and other crapware .e.g MS store, HP/Dell diagnostics and MS Teams. If you factor that in, Linux will win everytime.
  - psd1 2y ago
    If your company has windows and Linux but only guards against threats on windows, then you may have a different problem. I believe you can lock a Linux box down tighter than a windows box, but then you're trading compile times for other costs.
  - ta12301240124 2y ago
    Not the case unfortunately; you can. I'm not defending Windows here (bitter hatred of Microsoft here, since they started charging 2k for the compilation tools back in the 90s). In your particular corporate environment that might be the case, but not in this case, I had free run of a fresh install and no offensive AV there, and detuned to remove the crap. Other posters have said certain optimizations (which I'm not sure would help, it was pure compilation, no large files that I'm aware of). Just saying, always good to keep an open mind.
- mceachen 2y ago
  Were all your tests pure unit tests? Then sure—the OS isn’t getting in the way (and if your Linux distribution didn’t set the correct cpu scheduler, windows could be faster!) If, however, your tests make any filesystem calls or fork a child process, there’s slim chances that Linux doesn’t absolutely trounce Windows. To throw in a morsel of anecdata: same laptop with a dual boot runs the PhotoStructure core test suite (some 11,000 tests which have a pleasing melange of system and integration tests, not just unit tests) in 8m30s on Windows 11. Almost all the same tests complete in 3m30s on Linux.
- account42 2y ago
  Seems weird to base the argument on compiler performance alone. I'd think that developer tools will be a much bigger factor for the time difference but that depends on what you are used to. Anecdotally though, git is unbearably slow under Windows and compiles make all filesystem operations lag while I have never seen such problems under Linux.
g15jv2dp 2y ago
Didn't they introduced ReFS ("dev drives") to alleviate this?
jraph 2y ago
Why wouldn't they work on improving NTFS performance instead?
- chris_wot 2y ago
  Because likely they aren’t a. allowed to, and/or b. have no file system code experience or understanding.
- chipdart 2y ago
  > Why wouldn't they work on improving NTFS performance instead? There are other old-school techniques which are far easier to implement and maintain, such as using RAM drives/partitions. Expensing 32GB of RAM is simpler than maintaining weird NTFS configurations. Splitting your project into submodules/subpackages also helps amortize the impact of long build times. You can run multiple builds in parallel and then have a last build task to aggregate the all. Everyone can live with builds that take 5 minutes instead of 3.
- pjc50 2y ago
  Much of the cost is imposed by semantics and the security model; there's no silver bullet solution that can be imposed without turning it into a different filesystem with different semantics and/or on disk representation. At one point they planned to replace it with a database filesystem, but that was too complicated and abandoned. That was probably the end of replacement work on NTFS. https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS
  - nullindividual 2y ago
    WinFS was never designed as a replacement as it rode on top of NTFS. It was a fancy SQL Server database that exposed files as .NET Objects for API interaction. Sans the API piece, think of it like storing blobs in SQL Server, just like SharePoint does. I was lucky enough to play around with beta 1. Not much you could do with it, though.
- TiredOfLife 2y ago
  Even Microsoft doesn't touch NTFS code (if they haven't lost it). All new file system features like new compression methods are implemented on layers above ntfs.

ta23948234 2y ago

As an anecdote, we had a really long build time for our pipeline (going back prob 15 years). I argued for a Linux laptop, and the boss said, "OK, prove it. Here's two equivalent laptops, time it.". Turns out there was zero difference, or negligible (Windows won), between compilation times. That has always annoyed me.

rapind 2y ago
Love that your boss was open to the idea but also wanted data to back it up. (imagine a slow clap)
- ta9345908345098 2y ago
  Yep, was a good experience. I'm sure with fine tuning it (Linux) could've been better, but I ate that humble pie.
- goodpoint 2y ago
  A single datapoint from an experiment done by only one person you call it data? What's not to love...
  - tssva 2y ago
    It was the only data point which mattered for the decision that needed to be made.
chipdart 2y ago
> Turns out there was zero difference, or negligible (Windows won), between compilation times. I think there was something seriously flawed in your test. If you Google for a minute, you find multiple posts on how moving the same builds to Linux led to performance improvements in the range of 40% drops in build times. Some anecdotes even compare doing the same builds in Ubunto with NTFS to see double-digit gains. NTFS is notoriously awful in scenarios involving reading/writing many small projects. This is the bottleneck in Windows builds. There is a myriad of benchmarks documenting this problem. Nowadays there are plenty of cross-platform projects to serve as benchmarks. Checking this can be as easy as checking out a project, start a full rebuild, and check how long it takes.
- bee_rider 2y ago
  It seems possible that their pipeline just had large files, or something like that.
- psd1 2y ago
  That's survivor bias. Fewer people blog about unsuccessful initiatives.
  - chipdart 2y ago
    > That's survivor bias. Fewer people blog about unsuccessful initiatives. I don't understand what point you're trying to make. Should we give equal credit to unsuccessful initiatives when their failure is due to screwing up a critical part of the project? I mean, the successful attempts document what is possible. What do you expect to report when you couldn't even manage to get something working?
    - psd1 2y ago
      Say there are N initiatives to move to cross-compilation. Of those, there are M that deliver the hoped-for benefits, where M <= N. But practically, M < N significantly, say, M < 0.9 N. Assume bloggers blog mainly about content that contains a positive message. I'm asserting that people blog more readily about their success than their failures. So when you look at the blog literature, your population is not N, it is M. You don't see the failures because they don't tell the tale.
ta456456456456 2y ago
Yeah sorry, I left out a lot of details for a quick anecdote. It was Java compilation, so the different JDK's may've made a difference (almost certainly). But I figured the better file system handling etc etc would've gone in favour of Linux, versus Windows. In any case it wasn't much of a fun codebase. But I think a good lesson was, always test it, always measure. Not casting shade on either OS.
- freedomben 2y ago
  Do you remember which file system you used on Linux?
zamadatix 2y ago
For the most part the two are pretty equivalent in performance, it's particularly constant blocking on small file IO where Windows falls way behind. Also a bit in the "tons of cores" and "tons of networking" categories but not as nearly as bad as small file IO.
- account42 2y ago
  Well yeah, if you avoid the OS (syscalls) as much as possible then all operating systems will have about the same performance.
manishsharan 2y ago
You can't have windows laptop without running a ton of Antivirus and other crapware .e.g MS store, HP/Dell diagnostics and MS Teams. If you factor that in, Linux will win everytime.
- psd1 2y ago
  If your company has windows and Linux but only guards against threats on windows, then you may have a different problem. I believe you can lock a Linux box down tighter than a windows box, but then you're trading compile times for other costs.
- ta12301240124 2y ago
  Not the case unfortunately; you can. I'm not defending Windows here (bitter hatred of Microsoft here, since they started charging 2k for the compilation tools back in the 90s). In your particular corporate environment that might be the case, but not in this case, I had free run of a fresh install and no offensive AV there, and detuned to remove the crap. Other posters have said certain optimizations (which I'm not sure would help, it was pure compilation, no large files that I'm aware of). Just saying, always good to keep an open mind.
mceachen 2y ago
Were all your tests pure unit tests? Then sure—the OS isn’t getting in the way (and if your Linux distribution didn’t set the correct cpu scheduler, windows could be faster!) If, however, your tests make any filesystem calls or fork a child process, there’s slim chances that Linux doesn’t absolutely trounce Windows. To throw in a morsel of anecdata: same laptop with a dual boot runs the PhotoStructure core test suite (some 11,000 tests which have a pleasing melange of system and integration tests, not just unit tests) in 8m30s on Windows 11. Almost all the same tests complete in 3m30s on Linux.
account42 2y ago
Seems weird to base the argument on compiler performance alone. I'd think that developer tools will be a much bigger factor for the time difference but that depends on what you are used to. Anecdotally though, git is unbearably slow under Windows and compiles make all filesystem operations lag while I have never seen such problems under Linux.

g15jv2dp 2y ago

Didn't they introduced ReFS ("dev drives") to alleviate this?

jraph 2y ago

Why wouldn't they work on improving NTFS performance instead?

chris_wot 2y ago
Because likely they aren’t a. allowed to, and/or b. have no file system code experience or understanding.
chipdart 2y ago
> Why wouldn't they work on improving NTFS performance instead? There are other old-school techniques which are far easier to implement and maintain, such as using RAM drives/partitions. Expensing 32GB of RAM is simpler than maintaining weird NTFS configurations. Splitting your project into submodules/subpackages also helps amortize the impact of long build times. You can run multiple builds in parallel and then have a last build task to aggregate the all. Everyone can live with builds that take 5 minutes instead of 3.
pjc50 2y ago
Much of the cost is imposed by semantics and the security model; there's no silver bullet solution that can be imposed without turning it into a different filesystem with different semantics and/or on disk representation. At one point they planned to replace it with a database filesystem, but that was too complicated and abandoned. That was probably the end of replacement work on NTFS. https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS
- nullindividual 2y ago
  WinFS was never designed as a replacement as it rode on top of NTFS. It was a fancy SQL Server database that exposed files as .NET Objects for API interaction. Sans the API piece, think of it like storing blobs in SQL Server, just like SharePoint does. I was lucky enough to play around with beta 1. Not much you could do with it, though.
TiredOfLife 2y ago
Even Microsoft doesn't touch NTFS code (if they haven't lost it). All new file system features like new compression methods are implemented on layers above ntfs.