15 ms·
Windows file system compression had to be dumbed down
- olavgg 10y agoI wonder how effective LZ4 would be as default compression algorithm for NTFS. It is extremely effective with ZFS.
- zdw 10y agoPort it to Alpha and let's see. I'll loan you my 21164 over the weekend.
- dfox 10y agoThe article mentions that alpha was weak on bit-twiddling, which is something that almost certainly got fixed by introduction of BWX instructions on 21164.
- dchest 10y agoInterestingly, LZ4 works mostly on nibble and byte level without much bit-twiddling.
- honkhonkpants 10y agoWas axp even the weakest of the four windows architectures for this task? PPC and MIPS can leave the bit twiddler scratching their heads, too.
- dfox 10y agoProblem with original Alpha is that accessing memory as anything other than aligned 32b or 64b words (yes, including bytes) counts as bit-twiddling because only load/store instructions that it has operate on these two word sizes. I can't remember any other non-niche architecture with 8*2^n word size that shares this (mis-)feature. (I suspect that Cray 1 and it's derivates also share this, but that probably counts as niche architecture)
- EvanAnderson 10y agoThanks for mentioning that. You sent me down a brief rabbit-hole of nostalgia looking for the EV4/EV5 differences. If anybody else wants to go down that rabbit-hole here's where I ended-up: http://alasir.com/articles/alpha_history/alpha_21164_21164pc.html http://alasir.com/articles/alpha_history/alpha_21164_21164pc...
- krylon 10y agoAlso, when was the last Alpha built? About ten years ago? It's fine for OpenBSD or NetBSD to support it, but Microsoft has not supported it in a long time.
- toast0 10y agoSince the last released build of Windows for Alpha was Windows 2000 RC1 (although it looks like there's an RC2 build floating around), I don't think we need to support that processor for a new NTFS extension.
- pavlov 10y agoFor the algorithm that was ultimately chosen, the smallest unit of encoding in the compressed stream was the nibble Feels to me like nobody talks of nibbles anymore, maybe because we have ample memory and storage and can usually afford to waste some bits for the convenience of byte alignment. (It's half a byte, or 4 bits.) Disk compression is interesting because Microsoft originally included it already in MS-DOS but lost a lawsuit brought by the company behind a popular utility called Stacker: http://articles.latimes.com/1994-02-24/business/fi-26671_1_software-patent http://articles.latimes.com/1994-02-24/business/fi-26671_1_s... That was the first time Microsoft got in hot water for bundling features into DOS/Windows (the web browser would be the straw that broke the camel's back).
- IntelMiner 10y ago"Software patents remain controversial; critics have contended that the U.S. Patent Office does not understand the industry and issues patents that are too broad. Patent attorneys said Wednesday's decision will encourage small software firms to use patents as leverage against the industry's big players" Oh if only they knew what would happen
- pjc50 10y ago"Patent attorneys said Wednesday's decision will encourage small software firms to use patents as leverage against the industry's big players" The patent industry always wheels out the small inventor as PR when defending the system. As soon as noone's looking they use the same system to keep small inventors out.
- billyhoffman 10y agoto be clear, Stacker sued MS over intellectual property concerns, and not over monopolistic actions like bundling a web browser into their OS like they encountered with IE/Windows 98
- reacweb 10y agoAs far as I remember, Stacker was a marvelous software that compressed disks in a transparent manner. Everybody was using it (often pirated). Then Microsoft has provided DoubleSpace for free with its new release of DOS. It was slightly inferior (more bugs) but had basically the same functionalities. This was the murder of Stacker by the infamous Microsoft.
- pyreal 10y agoInteresting timing. I just used Windows' built in compression to free up 25 GB on a customer's full C: drive. First time I've used the feature in years. I noted that it didn't seem to be very good compression level.
- rexicus 10y agoIt's never been worth using due to the slowdown and fragmentation it causes.
- anonymfus 10y agoSlowdown in one set of tasks and speed-up in another. For example, boot time is usually shorter when system folders are compressed (with compact /CompactOS:always).
- Alupis 10y ago> For example, boot time is usually shorter when system folders are compressed Reads, in general, will be "boosted" when the filesystem is compressed. You get an (almost) free boost by reading and extracting compressed data on the fly into memory/cpu. ie. read 4mb's off the disk but it expands to 8mb's (or more!) in memory, so read performance is elevated. Writing of course, is slowed. For some server loads, disk compression still makes a lot of sense - making the claim "We live in a post-file-system-compression world" a little dubious.
- Dylan16807 10y ago> We live in a post-file-system-compression world. You'd think so, but allow me to point at my steamapps folder saving 20% disk space even with the not-very-good compression that NTFS offers. If it could use a better algorithm and a block size of 2MB instead of 64KB, it could nearly double that. Hard drives have always been growing in size. We have always been in a 'post-file-system-compression' world. But people want to do things like fit on an SSD, so compression continues to be useful. I just wish it didn't ultra-fragment files on purpose.
- danieldk 10y agoWe have always been in a 'post-file-system-compression' world. But people want to do things like fit on an SSD, so compression continues to be useful. Note though that some controllers do compression to reduce write amplification (e.g. the SandForce controllers used to do this). So, by using filesystem compression, you might be increasing write amplification and shortening the SSD lifetime.
- rictic 10y agoIs that increasing write amplification to a meaningful degree? Compressing already compressed data is generally only slightly less efficient than just compressing the data once, isn't it?
- danieldk 10y agoThat's an interesting question. As far as I understand, yes. If you buy, say an 256GB SSD backed by roughly the same amount of storage in physical cells. Such an SSD stores 256 GB regardless whether it is compressed or not. Now suppose that we have a compressed filesystem and an uncompressed filesystem and that both the filesystem and controller compression rates are 2:1. Now let's compare the two scenarios: 1. With filesystem compression: you could store twice the amount of data (~512GB). However, since the controller is not able to compress the data, all cells are completely full. 2. Without filesystem compression: you can store 256GB data. However, since the controller can compress the data, only half of the actual cells are used. Suppose now that in both setups, the SSD is nearly full and we rewrite or perhaps extend some data. In the former case, the controller has to scavenge for partially-used cells to combine and erase. In the latter case, the SSD controller still has 128GB of pristine cells to write the changed data immediately. Of course, compression was always somewhat of a cheapskate solution (and one of the reasons the SandForce controllers are not liked much). Higher-end SSDs would just have more storage space than logically addressable to have some leeway when the SSD is almost full.
- revelation 10y agoBut of course all of that old code is still there and maintained. We live in a post-file-system-compression world with ubiquitous file-system-compression.
- jxy 10y agoIn other words, if you have control over both software and hardware, you can deliver a much better user experience.
- rasz_pl 10y ago>We live in a post-file-system-compression world. says a guy working for a company shipping system with >6GB (often 10GB in 100K individual small files!) of redundant NEVER EVER touched data inside WinSxS. Data that cant be moved off the main drive without serious hacks (hardlinking). This fits in the general pattern of indifference to user hardware. Another one (my fav) is non movable hiberfil.sys that MUST be on primary drive, cant even move it with hardlinks. There goes 16GB of your SSD for a file that gets used ONCE per day. This is what happens when you hire straight out of college programmers and put them on top of the line workstations.
- yuhong 10y agoTo be more precise, CBS is to blame for this.
- leeter 10y agotbf... almost all of WinSXS is hardlinks, there is very little actual duplication. It's just that explorer doesn't understand that and reports it as duplication.
- rasz_pl 10y agoeasily disproved: C:\WINDOWS\system32>dism /Online /Cleanup-Image /AnalyzeComponentStore Windows Explorer Reported Size of Component Store : 8.18 GB Actual Size of Component Store : 7.78 GB
- acqq 10y agoThe next line of the output from the command you specified but which you didn't quote and that gives almost 5 GB on the machine in the example it is named "Shared with Windows" https://technet.microsoft.com/en-us/library/dn251566.aspx https://technet.microsoft.com/en-us/library/dn251566.aspx "This value provides the size of files that are hard linked so that they appear both in the component store and in other locations (for the normal operation of Windows). This is included in the actual size, but shouldn’t be considered part of the component store overhead."
- jonstewart 10y agoInterestingly, NTFS in Windows 10 introduces a new codec for compressed files. See http://www.swiftforensics.com/2016/10/wofcompressed-streams-in-windows-10.html http://www.swiftforensics.com/2016/10/wofcompressed-streams-...
- my123 10y agoDeduplication works well these days.
- CharlesMerriam2 10y agoSo many comments being harsh to Microsoft for bad engineering by ignoring the real world. It's Microsoft. Would you critical of a pigeon for defecating in flight?
- kstrauser 10y agoWhile I get the point he makes, and I'm certain that the team aren't idiots, I vehemently disagree. Amiga had a popular third-party library called XPK that let you install system-wide codecs on your system, and then any XPK-aware app installed could use any of those codecs to compress and decompress data on the fly. There were also patches on the filesystem so that the OS itself could detect a file's compression algorithm and decompress it transparently for apps that didn't know about XPK. In short, in the early 90s Amiga had configurable per-file compression algorithms. There were CPU-optimized versions of almost all of those codecs, so someone using an ancient 68000 could interact with files compressed by a PPC. I could pull a drive out of my fast system and give it to a buddy with an old, slow CPU, and he could either 1) live with the reading speed penalty, 2) decompress each file one time and then use the unpacked versions, or 3) recompress each file with an algorithm more friendly to his system. I don't think the Windows OS team is dumb by any stretch. I do think they might have been hampered by NIH syndrome, and weren't aware of (and likely couldn't care less about) how these problems were solved on other OSes.
- Grishnakh 10y ago>I don't think the Windows OS team is dumb by any stretch. I do think they might have been hampered by NIH syndrome I think NIH qualifies as a form of stupidity.
- lucb1e 10y agoThis article makes a lot of assumptions and has some weird thoughts. > Well, okay, you can compress differently depending on the system, but every system has to be able to decompress every compression algorithm. But that's not how this works. You don't design an entirely new algorithm for every possible performance, you create one or two algorithms and tweak them. Then make one or two decompressors that can handle any compression setting. Simple example: lz77 (used in deflate/gzip) whose decompressor is extremely fast regardless of your compression settings. > Now, Windows dropped support for the Alpha AXP quite a long time ago Why does this sound like "... and they changed nothing"? > you can buy 5TB hard drive from [brand] for just $120. Sure but we also have more data to store. If you wanted 100 games in the 80s you'd need what, 1MB storage? I'm just guessing, I'm too young to know that. Now that'd be what, 1TB? It might be cheaper but I'm just saying, storage prices going down does not change that compression is a good idea. > many (most?) popular file formats are already compressed Before they were all about data accessibility and recovery on different/damaged systems but compression doesn't help that. Now this is a good thing? Additionally, binaries and libraries are not compressed; many of my documents are just text files; and databases still benefit from this a lot. (But who has a database on their computer? Anyone who uses a web browser and email program.) > We live in a post-file-system-compression world. I still think it's a fine idea. > Tags: History Lol
- hobarrera 10y ago> However, we also live in a world where you can buy 5TB hard drive from Newegg for just $120 It's a lot more expensive to install that disk it into an ultrabook/mba.
- yyhhsj0521 10y agoWhy can't we just use different compression levels? Or it wouldn't be hard to build into Windows multiple compress-algorithms. So that fast machines use high compression level or CPU-demanding algorithms, and slow machines use the contrary. Therefore slow machines could still decompress files from faster machines efficiently, because in decompression HDD I/O is the bottleneck [1]. [1] http://superuser.com/questions/135594/what-is-more-important-when-extracting-rars-cpu-or-hdd http://superuser.com/questions/135594/what-is-more-important...
- wmf 10y agoWhen this stuff was being designed in the 1990s the hard disk wasn't the bottleneck.
- rbanffy 10y agoI think there would be a strong case for the decompression logic being able to read whatever you throw at it while the compression logic would need to decide how hard it should think to meet the performance requirements depending on what the present machine can/should do considering its speed and/or present load. Keeping the decompressor robust would easily solve the drive portability concerns, making the data readable (even if not at maximum speed) across machines and architectures.