5 ms·
Apparently if you're on Windows you don't deserve performance: #[cfg(windows)] const MAX_THREADS: usize = 16; #[cfg(not(windows))] const MAX_T
by jiggawatts 7y ago
Apparently if you're on Windows you don't deserve performance:
#[cfg(windows)]
const MAX_THREADS: usize = 16;
#[cfg(not(windows))]
const MAX_THREADS: usize = 128;
From: https://github.com/spacejam/sled/blob/bf37bd120fbf62f78408e04970705ea57563acfa/src/threadpool.rs#L14-L18 https://github.com/spacejam/sled/blob/bf37bd120fbf62f78408e0...
- rapsey 7y agoWell threadripper tests showed windows does not scale that high.
- jiggawatts 7y agoIt does, but the client code needs NUMA support, which Rust does not currently have. It doesn't help that the default behaviour in Windows is just stupid: "By default, an application is constrained to a single group, which should provide ample processing capability for the typical application." This is the new "640KB should be enough for everybody", except now it is "64 threads should be enough". See: https://docs.microsoft.com/en-us/windows/win32/procthread/processor-groups https://docs.microsoft.com/en-us/windows/win32/procthread/pr... https://www.anandtech.com/show/15483/amd-threadripper-3990x-review/3 https://www.anandtech.com/show/15483/amd-threadripper-3990x-...
- tjoff 7y agoSounds like a really sensible default to me.
- hinkley 7y agoWhich surprises me, because formal object ownership and read/write access sound like exactly the sorts of things you’d like to have for a NUMA machine.
- hinkley 7y ago> 64 threads should be enough. It’s more that if you have a purely sequential workflow, you get access to less than 2% of the CPU on a 64 core machine. And even if you’re a bit parallel, Amdahl’s law smacks you pretty hard here.
- jfkebwjsbx 7y agoThe L3 cache is shared and has a mindblowing size, let's not forget that. Plus you get way more thermal room. Plus you can disable SMT. And there is always more than 1 process running if you take into account kernel threads. So I would say way quite a lot more than 2%. More like 10%.
- hinkley 7y ago> a lot more than 2%. More like 10%. That's still essentially single digit percentages. And if is really in the low double digits, you're back to single digits when you go to 80 cores. On the positive side, I don't think multi-user systems would be where they are today if we hadn't been staring at a future full of hardware that can only be fully utilized by embarrassingly parallel tasks.
- pilif 7y agoalso great commit message: https://github.com/spacejam/sled/commit/b7a5d14399540daa433a1c9352359e3664e41967 https://github.com/spacejam/sled/commit/b7a5d14399540daa433a... the limit to 16 threads under windows was added in a commit named "refactor imports". If somebody in the future tries to find out why the threads are limited to 16 under windows, they will eventually end up at that commit and if the committer isn't around any more (or doesn't remember), nobody will ever know the reasoning. For projects that have even the slightest chance of staying around, be mindful about your commit messages and explain why you are doing something, not what you are doing - that's what the code itself is for.
- hnarn 7y ago> be mindful about your commit messages and explain why you are doing something, not what you are doing It's sad to see how many experienced programmers do not understand this. Of course, it takes a little more effort, but without this type of thinking you might as well skip the commit message completely. I am a very junior programmer and it was one of the first thing I was corrected for not doing, and rightly so.
- andy_ppp 7y agoCode for your future self in 6 months time...
- cesarb 7y agoI've seen this kind of thing many times before: the developer wants to commit something (in this case, an import refactor), and accidentally commits other changes which were being worked on. This is why I don't recommend getting into the habit of using "git commit -a".
- pilif 7y agoabsolutely. I'm telling people to always use `git add -p` to help them produce self-contained commits. And if you screw something up, there's always `git commit --amend` or the sledgehammer of `git rebase -i`. Note: I'm not advocating doing this to public history, but I'm saying: Use these tools in order to create self-contained changes on your end before committing them to public history. When I'm doing archeology trying to find out why Foo is doing Bar instead of Baz, I don't want to know that back then, 10 years ago, you fixed a typo or added a "forgotten file".
- krenoten 7y agoonly narcs use windows for database workloads :P
- executesorder66 7y agoIf performance was something that was important to you, why would you even choose windows as your OS?
- jiggawatts 7y agoIgnoring the obvious that not everybody gets to choose the operating system their enterprise uses, there's real reasons to use Windows for performance. For one, SQL Server spanks most other databases for complex ad-hoc queries. Don't get me wrong, PostgreSQL is getting a lot better these days, and I'm sure that for a lot of basic workloads its very competitive, but the query planner in SQL Server is basically black magic. There are very few (any?) database engines out there that can, in a single query, combine in-memory tables, remote tables, functions, materalised views, columnar data, and recursive joins then run that query in parallel and efficiently. I think Oracle has most of those checkbox features, and perhaps a couple of others do too (DB2 perhaps?), but as far as I know MS SQL Server still outperforms them for this type of thing. Mind you, Oracle scales to bigger clusters, and DB2 runs on bigger iron, but that's not my use-case. I have one Intel box.
- executesorder66 7y ago> Ignoring the obvious that not everybody gets to choose the operating system their enterprise uses For sure. If you can't choose, then I don't blame you. > there's real reasons to use Windows for performance I can't really think of any reason though. You gave the example of SQLServer, but that also runs on Linux. Does SQL Server perform better on Windows than Linux? Edit: found a benchmark here: https://www.dbbest.com/blog/running-sql-server-on-linux/ https://www.dbbest.com/blog/running-sql-server-on-linux/ Turns out you get better performance for SQL Server on Linux compared to windows. I am not surprised by this at all.
- cal97g 7y agoThreads are more expensive in Windows; which may partially explain this.