10 ms·
Benchmarking OS primitives
- blitmap 9y agoWould the Windows-equivalent of fork() be CreateProcess() ?
- pjmlp 9y agoNo, because the semantics aren't the same. CreateProcess() is like posix_spawn(), or if you prefer fork()/exec(). Windows is a thread based OS, not process based, hence why the focus on thread performance, not on process creation.
- zokier 9y ago> Windows is a thread based OS, not process based, hence why the focus on thread performance, not on process creation. Which, somewhat ironically, leads NT to have worse numbers in the create thread test than linux in the create process one (25.6us vs 18us). The redeeming factor of NT is their async IO model which afaik is the best among mainstream OS.
- udp 9y agoIOCP is very complicated to code against, though. kqueue can do nearly all the same things and is both much cleaner and more portable.
- trentnelson 9y agoIt's a very different paradigm to wrap your head around, but once you grok the NT kernel's approach to I/O (packet based IRPs, inherently asynchronous, thread-agnostic), and thread scheduling, I/O completion ports are very powerful constructs. The key difference is that I/O completion ports can be used to achieve asynchronous I/O on any underlying object, e.g. files and sockets, and they have this nifty built-in concept of concurrency, such that the kernel can ensure there is always one running thread per CPU core (which is optimal from a scheduling perspective). You can't use file descriptors with epoll/kqueue, and you certainly can't say "ensure every core only has one active thread running". "The key to understanding what makes asynchronous I/O in Windows special is...": https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores?slide=54 https://speakerdeck.com/trent/pyparallel-how-we-removed-the-... "Thread-agnostic I/O with IOCP": https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores?slide=62 https://speakerdeck.com/trent/pyparallel-how-we-removed-the-...
- arghwhat 9y agoThere is no such thing as a "thread based OS". The statement simply doesn't make sense. The concepts of processes and threads work just the same in Linux and Windows (and internally just map to the execution unit of the scheduler, together with resource mappings and privileges), and user-space expectations are similar for the two. The main difference is that fork() is not available on Windows, but fork() is a terrible idea anyway. Fast spawn of processes isn't used for performance critical things on either OS, as process spawning is considered slow on Linux and entirely useless on Windows. Fast spawn of threads is also generally avoided, as even that is usually considered too slow. Windows is slow at creating processes (and most other things involving the kernel) not because of differences in OS use-case, but simply due to performance apparently not being a priority for Microsoft.
- pjmlp 9y agoThen you should spend some time educating yourself about such OSes, like Windows. A thread based OS is an OS where threads are the core unit of execution, and processes are just a kind of execution capsule with one thread executing by default. The kernel scheduler only understands threads. This by opposition to process based OSes like UNIX, where there is a clear distinction between a process and thread execution. The kernel scheduler handles processes and threads separately. In many UNIX platforms, a process that doesn't perform any thread related API call, won't have any thread running on its context. This was quite clear during the days when UNIX systems where still researching how to adapt threads into the process execution model. And in many cases the impedance mismatch is still visible in modern UNIX systems, like for example what happens to any given thread when a signal is triggered, or to the whole process when a thread decides to fork. You can start by getting yourself a copy of "Windows Internals" book. Here is an old version of "Processes, Threads, and Jobs in the Windows" chapter in the 5th edition. https://www.microsoftpressstore.com/articles/printerfriendly/2233328 https://www.microsoftpressstore.com/articles/printerfriendly...
- noselasd 9y agoNote that in linux, threads are the unit of scheduling, not processes.
- CJefferson 9y agoMicrosoft really need to fix windows defender and search indexing. I have myself benchmarked horrible pains similar to this, 7-10x slowdowns doing things like copying many files around. It can make the Linux Subsystem almost unusuable.
- quickben 9y agoAwesome work. Otherwise, the filesystem bench is pointless without: - SSDs type. Tlc? SLC? Are they same, different? - Linux filesystem type and fstab flags.
- mbitsnbites 9y agoSSD types: for the machines that are identical, they are the same Linux filesystem: stock ext4
- zokier 9y agoThe memory allocation test seems bit out of place, considering that the allocator is provided by libc and not the OS. Testing something like mmap/virtualalloc might have made more sense
- gpderetta 9y agoYou are not wrong, but a) at least on unix the libc is certainly considered part of the OS and b) malloc has to get the memory from the OS eventually, via sbrk or mmap.
- bjg 9y agoI'm not sure I agree with (a), any app that is focused on perf is likely going to use jemalloc or any of the other high performance allocators (tcmalloc), and completely bypass the libc implementation...
- zokier 9y ago> a) at least on unix the libc is certainly considered part of the OS Maybe on proper UNIX (and BSDs) where you have the whole base system as more or less one package, but on Linux certainly not. Honestly the whole concept of "OS" does not make that much sense on traditional Linux distros. You could consider a whole distro an OS, but then the phrase is so all-encompassing that it becomes meaningless. On the other hand calling just the kernel "OS" is not quite right either. > b) malloc has to get the memory from the OS eventually, via sbrk or mmap. Yes, eventually and occasionally. But there is fairly big disconnect between malloc and OS. You got me curious, so I did run the code from the article with few different NUM_ALLOCS values to see how it behaves. These are the results from glibc malloc on my system: Benchmark: Allocate/free 10000000 memory chunks (4-128 bytes)... 70.310378 ns / alloc % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 0.023622 4 5996 brk Benchmark: Allocate/free 1000000 memory chunks (4-128 bytes)... 171.745062 ns / alloc % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 0.250319 17 15004 brk Benchmark: Allocate/free 100000 memory chunks (4-128 bytes)... 33.700466 ns / alloc % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 0.000203 3 63 brk Benchmark: Allocate/free 10000 memory chunks (4-128 bytes)... 30.589104 ns / alloc % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 0.00 0.000000 0 9 brk Benchmark: Allocate/free 1000 memory chunks (4-128 bytes)... 26.941299 ns / alloc % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 0.000008 2 4 brk Note how the syscall count blows up at NUM_ALLOCS=1000000, which just happens to be the original value from the article. Yes, I checked and glibc did not fall back to mmap in any of these cases. You can already start seeing why this might not be the best of benchmarks. Then just for fun, I tried using jemalloc, which is a drop-in replacement for standard malloc. These are the results: Benchmark: Allocate/free 10000000 memory chunks (4-128 bytes)... 68.486404 ns / alloc % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 0.000186 4 47 mmap 0.00 0.000000 0 2 brk Benchmark: Allocate/free 1000000 memory chunks (4-128 bytes)... 59.097052 ns / alloc % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 96.45 0.000544 16 33 mmap 3.55 0.000020 10 2 brk Benchmark: Allocate/free 100000 memory chunks (4-128 bytes)... 54.659843 ns / alloc % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 0.00 0.000000 0 24 mmap 0.00 0.000000 0 2 brk Well, well, well. It certainly paints a very different picture.
- bla2 9y ago"Launching Programs" should use posix_spawn at least on macOS, it's a distinct syscall there and faster than fork + exec.
- deleted 9y ago[deleted]
- c12 9y agoAnyone else see the create file test results and think of a node_modules joke?
- stephencanon 9y agoLots of confounds due to non-uniform hardware, etc, but more importantly these are very artificial micro-benchmarks; systems are (ideally) tuned for performance on the sorts of loads that they will actually be running under, not artificial tests like "create 65k files of 32B each". In artificial tests like these, you frequently get the best performance by flushing data out as fast as possible, while in most "real-world" scenarios you have some temporal locality that makes keeping data around a win. Optimizing for these sorts of benchmarks can actually harm performance. Still, fun.
- self_awareness 9y agoNot very objective, since the operating systems were in unknown state, i.e. there was a third party antivirus installed on one Windows machine. In such conditions this benchmark doesn't provide any meaningful information.
- humanrebar 9y agoDepends on what the measurement is for. If you want to help developers pick a faster development setup, you want something realistic.
- vardump 9y agoI find the article very informative from point of view of developing software packages installable by the end user. Benchmarking typical environments vs artificially lean is much more helpful in practice.
- mbitsnbites 9y agoI didn't think that these benchmarks would get this much attention. I personally made them because I could not believe that identical Windows and Linux machines were performing so different in typical software development tasks (Git, CMake, GCC, file copying, ...). I threw in a few more machines (Raspberry, Mac, ...) into the mix to get some perspective. Most machines were stock configured (Ubuntu ext4 install for Linux, Windows 10 w/ Windows Defender, macOS X stock install, etc), so I believe that they should be representative for average users. The benchmark suite is open source and easy to run on your own hardware if you like to get more accurate/representative figures for a particular setup.
- dzdt 9y agoFrome experience git is slow on windows when dealing with 10's of thousands of files even with a SSD drive. This is due to the filesystem (NTFS) being rather slow, especially for various stat operations. (If you watch in windows taskmanager this shows up in the "other i/o" column -- not reads or writes.
- arghwhat 9y agoI am not sure if it is correct to attribute it to NTFS, rather than the Windows VFS layer. IIRC, NTFS is a reasonably sane filesystem.
- blattimwind 9y agoI don't know why, but NTFS metadata performance just was never very good. I always assumed that was due to the more complex/capable data model compared to what Linux usually uses, but I'm not so sure about that any more given the complexity and yet still better performance of ZFS (btrfs remains too much of a mixed bag to mention here). However, poor or bad applications also play a role here. For some reason explorer.exe requires 1-2 orders of magnitude more I/O time than dir.exe, and somehow the local search is slower than a human binary search. However they managed to do that bad of a job remains a mystery.
- paulmd 9y agoYeah, poor search performance is definitely an application problem here. Lately I've taken to using Cygwin find instead and it runs just as fast as you'd expect. eg find /cygdrive/c/ -type f -iname '*somefile.tar'
- swebs 9y agoIf you haven't already, I highly recommend reading "I Contribute to the Windows Kernel. We Are Slower Than Other Operating Systems. Here Is Why." http://blog.zorinaq.com/i-contribute-to-the-windows-kernel-we-are-slower-than-other-oper/ http://blog.zorinaq.com/i-contribute-to-the-windows-kernel-w...
- deleted 9y ago[deleted]
- justin66 9y agoIt's awful that something as misleading as that anonymous, superficial rant is held up as important. The only lesson there is pretty meta: you actually can issue a retraction for a rant, but nobody will care.
- meuk 9y agoI can't help but start a rant about Windows upon reading this. One of my major complaints with Windows is that things just 'feel slow'. I have to wait very often. Opening an FTP location? Wait for 5 seconds, (and it also opens in a new window, leaving the old window open in an unusable state - very confusing). Starting a GUI? Wait for 5 seconds. My laptop and raspberry pi at home both work a lot smoother than the hi-end (it's a brand new Dell XPS machine with 8 GB RAM - which I consider hi-end) laptop I have at work. I still find it hard to comprehend that people are buying ridiculously overpowered Windows computers for tasks like browsing and document editing. Developers are at fault too - if it runs smoothly on your $1000+ machine with 32GB RAM, that does not mean that the average user will be able to even use it. Everyone and their mother is jumping at the sustainability hype, but at the same time developers assume that everyone buys a new computer and phone every other year, for the same tasks we've been doing for decades. Once you realize this it's hard to use a Windows system and not cringe at the mess of laggy/unresponsive GUI's.
- justin66 9y ago> Starting a GUI? Wait for 5 seconds. You're holding it wrong. > My laptop and raspberry pi at home both work a lot smoother than the hi-end (it's a brand new Dell XPS machine with 8 GB RAM - which I consider hi-end) laptop I have at work. If your RPi is faster at comparable tasks than that Windows PC, your Windows PC has some extremely serious setup problems. (if your employer uses anything like the commercial security software mine does, that's one potential problem)
- meuk 9y agoI'd argue that it's more of a culture problem. I regularly have multiple instances of Visual Studio and Visual Studio Code open (together with many windows explorer, notepad, notepad++ and whatever). Visual Studio in itself is just slow, and often crashes, even if I just have one instance open. On Linux, I have several terminals open, which is just so much more lightweight. Of course, from a performance point-of-view, these are not comparable, but that is exactly my point. On windows, everything has a GUI, and everything seems to assume a much more hi-end machine. P.S. I admit that it was a bit misleading to post this rant under this article, since the article measures raw OS performance, while my point is that lower performance is usually sufficient too, if you use tools that have a single purpose instead of trying to be an OS in itself.
- drewg123 9y agoEver notice how much slower building software using configure is on MacOS than Linux? The results here point out why: Fork + exec is ~10x slower on MacOS. However, this isn't exactly new information. The general slowness of the OSX kernel has been known for years, via other benchmarks like lmbench. Its one of the reasons they were the first to implement a vdso-like interface for things like gettimeofday().
- mastax 9y agoAh yes, Windows Defender. I always forget about it until it makes some trivial operation take 5x too long. Make sure to add your compilers and build tools to the exclusions list.
- mbitsnbites 9y agoThat's the easy case. In a typical enterprise environment a software developer may be faced with several roundtrips to the centralized/outsourced it support to get their build folders white listed in the company approved (and forcibly installed) AV software to make a simple CMake run take less than 10 minutes (something that takes about 5 seconds on a stock Linux machine).
- justin66 9y agoBenchmarking things like file creation without noting the filesystem or how it was mounted is... interesting.
- cozzyd 9y agoYeah also SELinux / AppArmor state would be interesting to know as well.
- snvzz 9y agoNot surprised. Note the Linux results, while looking good compared to what's even worse, are still terrible. Missing are the results for the BSDs. I'm particularly interested on Dragonfly BSD. Maybe I'll try them myself when 5.2 is out, which will be soon.