3 ms·

C# is essentially Windows only and Windows is not a good platform for heavy data analysis. Windows has many problems but one of the worst is poor file system p

by tgflynn 14y ago

C# is essentially Windows only and Windows is not a good platform for heavy data analysis. Windows has many problems but one of the worst is poor file system performance. On Windows you waste a lot of time moving large files around when you would hardly notice these times on Linux.

kevingadd 14y ago
C# has been usable on other platforms like Linux for years and years now. Yes, Mono isn't as good of a runtime as the MS CLR but it's perfectly usable. Moving a file is a O(1) operation on Windows just like on any other reasonable operating system. If you're referring to copying between volumes, there's nothing magical about how Linux copies 1GB of data between volumes - Windows does the same thing. It's true that the Windows I/O stack has some performance deficits (the stat() equivalent is slower, for one) but don't be ridiculous, moving files is not a bottleneck.
- tgflynn 14y ago
  Anyone who has actually spent much time working on Windows vs. Linux with large files and probably more importantly large numbers of files is aware of how much more of pain this is on Windows. Theoretical issues aside Windows FS implementations don't hold a candle to Linux performance-wise. As for Mono the last thing I want as a developer is to be investing my time in a language with second rate support. Note that I certainly don't advocate Java for big data either, C++ is the way to go until something better comes along (and I'm not holding my breath).
  - WayneDB 14y ago
    This sounds somewhat specious. What are you doing with the large or numerous set of files? How big or how many files are we talking about?
- Uchikoma 14y ago
  In two companies I've now compared development with Eclipse/Intellij and Maven between Linux and Windows. The Windows file system seems to be much slower (~2x) for Java development (working in IDE, building, unit tests etc). Identical hardware.
  - kevingadd 14y ago
    That's largely the stat() speed penalty. Linux-oriented software tends to call stat() way more often than it needs to since it's comparatively quite fast on Linux (not sure why - more aggressive caching?). The most obvious example of this phenomenon is version control software like svn or git. If you profile them you'll see a ton of time spent just querying the attributes of files on Windows.

by tgflynn 14y ago

kevingadd 14y ago

C# has been usable on other platforms like Linux for years and years now. Yes, Mono isn't as good of a runtime as the MS CLR but it's perfectly usable. Moving a file is a O(1) operation on Windows just like on any other reasonable operating system. If you're referring to copying between volumes, there's nothing magical about how Linux copies 1GB of data between volumes - Windows does the same thing. It's true that the Windows I/O stack has some performance deficits (the stat() equivalent is slower, for one) but don't be ridiculous, moving files is not a bottleneck.

tgflynn 14y ago
Anyone who has actually spent much time working on Windows vs. Linux with large files and probably more importantly large numbers of files is aware of how much more of pain this is on Windows. Theoretical issues aside Windows FS implementations don't hold a candle to Linux performance-wise. As for Mono the last thing I want as a developer is to be investing my time in a language with second rate support. Note that I certainly don't advocate Java for big data either, C++ is the way to go until something better comes along (and I'm not holding my breath).
- WayneDB 14y ago
  This sounds somewhat specious. What are you doing with the large or numerous set of files? How big or how many files are we talking about?
Uchikoma 14y ago
In two companies I've now compared development with Eclipse/Intellij and Maven between Linux and Windows. The Windows file system seems to be much slower (~2x) for Java development (working in IDE, building, unit tests etc). Identical hardware.
- kevingadd 14y ago
  That's largely the stat() speed penalty. Linux-oriented software tends to call stat() way more often than it needs to since it's comparatively quite fast on Linux (not sure why - more aggressive caching?). The most obvious example of this phenomenon is version control software like svn or git. If you profile them you'll see a ton of time spent just querying the attributes of files on Windows.