8 ms·
7-Zip for Windows can now use more than 64 CPU threads for compression
- avidiax 1y agoWhy was there a limitation on Windows? I can't find any such limit for Linux.
- lofties 1y agoWindows has a concept of processor groups, that can have up to 64 (hardware) threads. I assume they updated 7zip to support multiple processor groups.
- okanat 1y agoEvery operating system, that's relevant in 2025, needs that concept. It is called NUMA. At some point you cannot model the system ignoring the memory affinity / closeness to the cores. Modern AMD CPUs are literally consist of core groups on chiplets. It is better for an OS to make decisions / expose APIs for cores that are physically so far away from each other that moving data back-and-forth over the RAM, system bus or interconnect has significant time penalties.
- lmm 1y agoSeems like this is a general Windows thing per https://learn.microsoft.com/en-us/windows/win32/procthread/processor-groups https://learn.microsoft.com/en-us/windows/win32/procthread/p... - applications that want to run on more than 64 CPUs need to be written with dedicated support for doing so.
- deleted 1y ago[deleted]
- dwattttt 1y agoThe linked Processor Group documentation also says: > Applications that do not call any functions that use processor affinity masks or processor numbers will operate correctly on all systems, regardless of the number of processors. I suspect the limitation 7zip encountered was in how it checked how many logical processors a system has, to determine how many threads to spawn. GetActiveProcessorCount can tell you how many logical processors are on the system if you pass ALL_PROCESSOR_GROUPS, but that API was only added in Windows 7 (that said, that was more than 15 years ago, they probably could've found a moment to add and test a conditional call to it).
- dspillett 1y agoIt isn't just detecting the extra logical processors, you have to do work to utilise them. From the linked text: "If there are more than one processor group in Windows (on systems with more than 64 cpu threads), 7-Zip distributes running CPU threads across different processor groups." The OS does not do that for you under Windows. Other OSs handle that many cores differently. > more than 15 years ago, they probably could've found a moment to add and test a conditional call to it I suspect it hasn't been an issue much at all until recently. Any single block of data worth spinning up that many threads for compressing is going to be very large, you don't want to split something into too small chunks for compression or you lose some benefit of the dynamic compression dictionary (sharing that between threads would add a lot of inter-thread coordination work, killing any performance gain even if the threads are running local enough on the CPU to share cache). Compression is not an inherently parallelizable task, at least not “embarrassingly” so like some processes. Even when you do have something to compress that would benefit for more than 64 separate tasks in theory, unless it is all in RAM (or on an incredibly quick & low latency drive/array) the process is likely to be IO starved long before it is compute starved, when you have that much compute resource to hand. Recent improvements in storage options & CPUs (and the bandwidth between them) have presumably pushed the occurrences of this being worthwhile (outside of artificial tests) from “practically zero” to “near zero, but it happens”, hence the change has been made. Note that two or more 7-zip instances working on different data could always use more than 64 threads between them, if enough cores to make that useful were available.
- dwattttt 1y agoAre you sure that if you don't attempt to set any affinities, Windows won't schedule 64+ threads over other processor groups? I don't have any system handy that'll produce more than 64 logical processors to test this, but I'd be surprised if Windows' scheduler won't distribute a process's threads over other processor groups if you exceed the number of cores in the group it launches into. The referenced text suggests applications will "work", but that isn't really explicit.
- Dylan16807 1y ago
- silon42 1y agoMaybe WaitForMultipleObjects limit of 64 (MAXIMUM_WAIT_OBJECTS) applies? An ugly limitation on an API that initially looks superior to Linux equivalents.
- monocasa 1y agoA lot of synchronization primitives in the NT kernel are based on a register width bit mask of a CPU set, so each collection of 64 hardware threads on 64 bit systems kind of runs in its own instance of the scheduler. It's also unfortunately part of the driver ABI since these ops were implemented as macros and inline functions. Because of that, transitioning a software thread to another processor group is a manual process that has to be managed by user space.
- zik 1y agoWow. That's surprisingly lame.
- Const-me 1y agoThe NT kernel dates back to 1993. Computers didn’t exceed 64 logical processors per system until around 2014. And doing it back then required a ridiculously expensive server with 8 Intel CPUs. The technical decision Microsoft made initially worked well for over two decades. I don’t think it was lame; I believe it was a solid choice back then.
- monocasa 1y agoI mean, x86 didn't, but other systems had been exceeding 64 cores since the late 90s. And x86 arguably didn't ship >64 hardware thread systems until then because NT didn't support it.
- Const-me 1y ago> other systems had been exceeding 64 cores since the late 90s. Windows didn’t run on these other systems, why would Microsoft care about them? > x86 arguably didn't ship >64 hardware thread systems until then because NT didn't support it For publicly accessible web servers, Linux overtook Windows around 2005. Then in 2006 Amazon launched EC2, and the industry started that massive transition to the clouds. Linux is better suited for clouds, due to OS licensing and other reasons.
- whalesalad 1y agoWindows is a terrible operating system.
- xxs 1y agoWaitForMultipleObjects is limited to 64... since forever.
- d33 1y agoI worry that 7-Zip is going to lose relevance because lack of zstd support. zlib's performance is intolerable for large files and zlib-ng's SIMD implementation only helps here a bit. Which is a shame, because 7-Zip is a pretty amazing container format, especially with its encryption and file splitting capabilities.
- sammy2255 1y agohttps://github.com/mcmilk/7-Zip-zstd https://github.com/mcmilk/7-Zip-zstd
- abhinavk 1y agohttps://github.com/M2Team/NanaZip https://github.com/M2Team/NanaZip It includes the above patches as well as few QoL features.
- birksherty 1y agoTried it, compression is worse compared to 7-Zip-zstd by mcmilk for zstd, for same speed. The removal of text from toolbar icons is enough for me to never use it again. 7zip can change file associations directly with it and very easy. Feels like NanaZip is worse in QoL than 7zip.
- d33 1y agoThanks! Any ideas why it didn't get merged? Clearly 7-Zip has some development activity going on and so does this fork...
- Beretta_Vexee 1y agoWorking with Igor Pavlov, the creator of 7-zip, does not seem very straightforward (understatement).
- Tuldok 1y ago7-zip's development is very cathedral. Igor Pavlov doesn't look like he accepts contributions from the public.
- marcodiego 1y agohttps://xkcd.com/619/ https://xkcd.com/619/
- aquir 1y ago7-zip is one of the software that I miss since I’ve moved to macOS
- DeepSeaTortoise 1y agoHow about PeaZip?
- aquir 1y agoI've used PeaZip in the past but only on Windows, I was not aware that a MacOS version exists! I'll give it a try. Cheers
- MYEUHD 1y agoIf you're talking about the program you use in the terminal, you can install it via homebrew
- immibis 1y agoNo, the GUI. 7-zip integrates well with the shell: select a group of files, right click -> make zip file, and so on. Or right-click a zip file and select extract. If you're accustomed to Linux you might not know what they're talking about. TortoiseGit (and TortoiseSVN) are similarly convenient. Right click a folder with an SVN repo checked out, and select "SVN update". Right-click an empty space, and select "SVN checkout". SVN was the main distribution method for some modding communities before things like Steam Workshop and Github, specifically because TortoiseSVN made it so convenient. Checkout into your addons folder, and periodically update. What could be simpler?
- portaltonowhere 1y agoKeka is also really nice! https://www.keka.io/ https://www.keka.io/
- aquir 1y agoNever heard of it, I'll give it a try!
- ltbarcly3 1y agoWow, a program that doesn't matter anymore has been very very minimally enhanced on a platform that doesn't matter anymore, benefitting the 7 users that have more than 64 real cores with Windoes and are regularly compressing archives so large that it doesn't drastically reduce the compression ratio to split it into more thsn 64 sections. Posting this link to hn has consumed more human potential than the thing it is describing will save up to the end of time.
- tobinc 1y agoA 1% speed improvement for 1% of 7zip users is several times more productive than your comment.
- starkrights 1y ago> a program that doesn’t matter anymore The rest of this comment has, though gratuitously snarky, a point, but I don’t think claiming that 7zip is irrelevant as an independent statement is even remotely coherent.
- ltbarcly3 1y agotar.zstd is superior in basically every way, open, and portable
- michaelcampbell 1y agoIt's fine that we run in different circles, but I have yet to see one of these in the wild. Betamax was better, too.
- lihaciudaniel 1y ago7zip has been the greatest usage for limbo x86 on mobile. You just termux qemu-utils convert your qcow2 partitions to IMG and 7zip can read IMG file Try yourself to see you can extract from your emulated windows
- leecarraher 1y agoI've used pbzip2 which takes the same parallel blocked compression approach 7zip seems to be taking (using AI's analysis of the changes). Theoretically the compression is less efficient, but i haven't noticed a difference in practice.
- izzydata 1y agoThis may or may not be a relevant question, but does the terminology of "zip" have the same origin as the zip disk drive?
- malfist 1y agoNo. Zip format significantly predates the zip disk.
- vpShane 1y agoThe ZIP file format was created in 1989 by Phil Katz for his PKZIP software. The name 'ZIP' was chosen to convey the idea of compressing files into a smaller size, making them 'zip' together for easier storage and transfer. The term suggests speed and efficiency in compressing and uncompressing files. Like zipping something up in a bag. You unzip the bag and have all of its contents, zip it up and all of its contents are in one bag (file). 7zip has always been the defacto-best, I'm sure there's other decent compression algorithms but 7z has always taken the cake for me.
- ninjis 1y agoI had initially migrated to NanaZip, but with Windows natively supporting the 7z format now, I'm not sure it's needing anymore.
- fabiensanglard 1y agoHow does that work? You cannot write to disk before you know the compressed size. Or if you do you can use a data descriptor but you cannot write concurrently. I guess they buffer the compressed stream to RAM before writing to zip. If they want to keep their zip stable (always the same output given the same input), they also need to keep it a bit longer than necessary in RAM.
- 1W6MIC49CYX9GAP 1y agoI think you get different compressed files depending on how many threads you use to compress
- deleted 1y ago[deleted]
- amelius 1y agoMaybe Windows allows a file to be constructed quickly from parts that are not necessarily multiples of the blocksize. Maybe they have a fast api for writing multiple files and then turning it into a single file. POSIX doesn't allow that, but it is quite old.
- deleted 1y ago[deleted]