8 ms·

Apparently, snaps are compressed to save disk space, which is why they take so long to start: - https://www.reddit.com/r/Ubuntu/comments/9scoif/snap_packages_a

Apparently, snaps are compressed to save disk space, which is why they take so long to start: - https://www.reddit.com/r/Ubuntu/comments/9scoif/snap_packages_are_very_slow_to_load_in_ubuntu/ https://www.reddit.com/r/Ubuntu/comments/9scoif/snap_package... Saving disk space is certainly useful for rarely used apps, however, your web browser (and any other frequencely used apps), shouldn't be compressed, especially if there is ample disk space.

kleiba 6y ago
Saving disk space is certainly useful for rarely used apps Is it really? I can't recall the last time I ran into disk space issues, must have been in the 1990s.
- saurik 6y ago
  Yeah... and when something is truly large it generally doesn't compress well anyway as the large assets are embedded media files; I don't understand the point of compressing stuff like this :/.
- jcelerier 6y ago
  I run on disk space issues almost monthly, and I have a fair amount of terabytes
- Matt3o12_ 6y ago
  Really? I constantly run into disk space issues. Apple still ships their flagship 13" macbook pro 128gb storage and they charge $200 for another 128gb. While other manufacturer's laptops charge a lot less for storage these days, most still only come with 256 which is not enough these days for development IMO. Even on my desktop, I managed to fill 750GB with various VMs and android development tools (the SDKs, etc). While I am not sure how much compression could have saved me, it could still be worth it (especially since I only use certain VMs or SDK version once a month).
  - ssivark 6y ago
    Yeah, then stop burying yourself with Apple devices. Anyways, it makes sense to maintain something like an LRU cache, and compress only the least used things.
    - danieldk 6y ago
      Why would you? lz4 decompresses at ~5GB/s on a modern CPU [1] with good compression ratios, that's still more than most SSDs can push nowadays. Most applications are small fraction of that size on-disk. The problems arise when you start using xz-compressed squashfs images. LZMA2 is optimized for compression ratio and typically decompresses several times slower than even zlib deflate (which is already ~10 slower than lz4). [1] https://github.com/lz4/lz4 https://github.com/lz4/lz4
- amaccuish 6y ago
  I still don't understand though. I run on btrfs with compression turned on and have noticed 0 performance issues. Only with snaps as mentioned.
tjoff 6y ago
Whoa, it is a weird focus in this day and age... Especially since this has been a solved problem for ages without any real performance penalty. NTFS have had this since 1995 it seems and zfs probably since its inception.
dmw_ng 6y ago
The bottom line is they optimize installation time by amortizing it out over the runtime life of the package, or in other words, optimizing a one time 15 second process to be a 14 second process, in return for making a many-times 1 second process a 30 second process. It makes absolutely no sense. They do this using a filesystem originally designed for embedded devices, using a driver hacked to disable threading support because the sheer number of filesystems snapd mounts would otherwise consume a huge amount of memory in per-cpu buffers used for decompression. In other words, they broke squashfs for everyone in the process of trying to make it work for snap. On-demand decompression like this has made very little sense on desktops since the mid 90s, and even if it did, snapd's manifestation of it is particularly terrible.
- jquast 6y ago
  > On-demand decompression like this has made very little sense on desktops since the mid 90s Ok, maybe not desktops? But ZFS on-disk compression is a sysadmin's frickin dream -- just one example that you can access logfiles with plaintext tools like grep while benefiting from the space savings with neglible cost, LZ4 has basically no overhead at all, https://www.servethehome.com/the-case-for-using-zfs-compression/ https://www.servethehome.com/the-case-for-using-zfs-compress... I really hope you will try on-disk compression, encryption, deduplication, and that sort of thing sometime, you will see it is so much better than gzip-compressed, gpg-encrypted files
  - zrm 6y ago
    Filesystem compression is a completely different animal than this. It has to deal with your ability to modify the file at any time. It doesn't compress the whole file together, it does it in blocks. When you launch a binary (and the system mmaps it) it doesn't have to decompress the entire file before you can start using it, only the first compression block. Compression also typically makes it faster to launch applications from spinning rust, because the bottleneck is the drive and reading 50MB and decompressing it is faster than reading 100MB uncompressed. This would be true of SSDs as well except that most of them already do this internally. But snap isn't reading e.g. 64kB and then giving you 128kB on demand (and then prefetching the next block) like the filesystem does, it has to read and decompress the entire 100+MB package before you can even open it. That is very silly and adds a perceptible amount of latency.
chrisseaton 6y ago
Isn't the point of compression to save transfer time from the disk to memory, not space on the disk? That's why the kernel is compressed.
- lonelappde 6y ago
  Yes. It's a great idea for spinning rust, meh for SATA SSD, and bad for NVMe SSD.
  - deleted 6y ago
    [deleted]
ken 6y ago
Apple has silently compressed files (including executables) since Snow Leopard [1] -- to increase speed. Did Ubuntu pick the wrong compression algorithm? [1]: https://arstechnica.com/gadgets/2009/08/mac-os-x-10-6/3/ https://arstechnica.com/gadgets/2009/08/mac-os-x-10-6/3/
- thaumasiotes 6y ago
  I see two mentions of "increased speed" in that article: 1. Increased installation speed. This one's obvious; less data takes less time to install. This is mentioned in your parent comment. 2. "But compression isn't just about saving disk space. It's also a classic example of trading CPU cycles for decreased I/O latency and bandwidth. Over the past few decades, CPU performance has gotten better (and computing resources more plentiful—more on that later) at a much faster rate than disk performance has increased. Modern hard disk seek times and rotational delays are still measured in milliseconds. In one millisecond, a 2 GHz CPU goes through two million cycles. And then, of course, there's still the actual data transfer time to consider. [...] Given the almost comical glut of CPU resources on a modern multi-core Mac under normal use, the total time needed to transfer a compressed payload from the disk and use the CPU to decompress its contents into memory will still usually be far less than the time it'd take to transfer the data in uncompressed form." It's an interesting point, but seek times and rotational delays don't apply to SSDs. This is kind of an uneasy comparison to draw with "I hate that Chromium’s snap takes more than 10 seconds to load on cold boot on a freaking SSD".
  - danieldk 6y ago
    It's an interesting point, but seek times and rotational delays don't apply to SSDs. There is another reason to do compression on SSDs: you have more storage free and thus less write amplification and your SSDs will last longer. In fact some SSD controllers (e.g. SandForce controllers used to do this) compress data to reduce write amplification. https://en.wikipedia.org/wiki/SandForce#Technology https://en.wikipedia.org/wiki/SandForce#Technology The trick that they applied is that say, if you had a 500GB SSD and you stored 400GB uncompressed which the controller compressed to 200GB, the drive would still report only having 100GB free, giving it an ample 300GB of free blocks, thus greatly reducing write amplification. (Of course, the benefit of controller-level compression is gone with full-disk encryption. But I guess FDE was less popular when SandForce SSDs became popular.)
Florin_Andrei 6y ago
> snaps are compressed to save disk space The list of dumb decisions that Ubuntu has been making recently just keeps increasing.

kleiba 6y ago

Saving disk space is certainly useful for rarely used apps Is it really? I can't recall the last time I ran into disk space issues, must have been in the 1990s.

saurik 6y ago
Yeah... and when something is truly large it generally doesn't compress well anyway as the large assets are embedded media files; I don't understand the point of compressing stuff like this :/.
jcelerier 6y ago
I run on disk space issues almost monthly, and I have a fair amount of terabytes
Matt3o12_ 6y ago
Really? I constantly run into disk space issues. Apple still ships their flagship 13" macbook pro 128gb storage and they charge $200 for another 128gb. While other manufacturer's laptops charge a lot less for storage these days, most still only come with 256 which is not enough these days for development IMO. Even on my desktop, I managed to fill 750GB with various VMs and android development tools (the SDKs, etc). While I am not sure how much compression could have saved me, it could still be worth it (especially since I only use certain VMs or SDK version once a month).
- ssivark 6y ago
  Yeah, then stop burying yourself with Apple devices. Anyways, it makes sense to maintain something like an LRU cache, and compress only the least used things.
  - danieldk 6y ago
    Why would you? lz4 decompresses at ~5GB/s on a modern CPU [1] with good compression ratios, that's still more than most SSDs can push nowadays. Most applications are small fraction of that size on-disk. The problems arise when you start using xz-compressed squashfs images. LZMA2 is optimized for compression ratio and typically decompresses several times slower than even zlib deflate (which is already ~10 slower than lz4). [1] https://github.com/lz4/lz4 https://github.com/lz4/lz4
amaccuish 6y ago
I still don't understand though. I run on btrfs with compression turned on and have noticed 0 performance issues. Only with snaps as mentioned.

tjoff 6y ago

Whoa, it is a weird focus in this day and age... Especially since this has been a solved problem for ages without any real performance penalty. NTFS have had this since 1995 it seems and zfs probably since its inception.

dmw_ng 6y ago

The bottom line is they optimize installation time by amortizing it out over the runtime life of the package, or in other words, optimizing a one time 15 second process to be a 14 second process, in return for making a many-times 1 second process a 30 second process. It makes absolutely no sense. They do this using a filesystem originally designed for embedded devices, using a driver hacked to disable threading support because the sheer number of filesystems snapd mounts would otherwise consume a huge amount of memory in per-cpu buffers used for decompression. In other words, they broke squashfs for everyone in the process of trying to make it work for snap. On-demand decompression like this has made very little sense on desktops since the mid 90s, and even if it did, snapd's manifestation of it is particularly terrible.

jquast 6y ago
> On-demand decompression like this has made very little sense on desktops since the mid 90s Ok, maybe not desktops? But ZFS on-disk compression is a sysadmin's frickin dream -- just one example that you can access logfiles with plaintext tools like grep while benefiting from the space savings with neglible cost, LZ4 has basically no overhead at all, https://www.servethehome.com/the-case-for-using-zfs-compression/ https://www.servethehome.com/the-case-for-using-zfs-compress... I really hope you will try on-disk compression, encryption, deduplication, and that sort of thing sometime, you will see it is so much better than gzip-compressed, gpg-encrypted files
- zrm 6y ago
  Filesystem compression is a completely different animal than this. It has to deal with your ability to modify the file at any time. It doesn't compress the whole file together, it does it in blocks. When you launch a binary (and the system mmaps it) it doesn't have to decompress the entire file before you can start using it, only the first compression block. Compression also typically makes it faster to launch applications from spinning rust, because the bottleneck is the drive and reading 50MB and decompressing it is faster than reading 100MB uncompressed. This would be true of SSDs as well except that most of them already do this internally. But snap isn't reading e.g. 64kB and then giving you 128kB on demand (and then prefetching the next block) like the filesystem does, it has to read and decompress the entire 100+MB package before you can even open it. That is very silly and adds a perceptible amount of latency.

chrisseaton 6y ago

Isn't the point of compression to save transfer time from the disk to memory, not space on the disk? That's why the kernel is compressed.

lonelappde 6y ago
Yes. It's a great idea for spinning rust, meh for SATA SSD, and bad for NVMe SSD.
- deleted 6y ago
  [deleted]

ken 6y ago

Apple has silently compressed files (including executables) since Snow Leopard [1] -- to increase speed. Did Ubuntu pick the wrong compression algorithm? [1]: https://arstechnica.com/gadgets/2009/08/mac-os-x-10-6/3/ https://arstechnica.com/gadgets/2009/08/mac-os-x-10-6/3/

thaumasiotes 6y ago
I see two mentions of "increased speed" in that article: 1. Increased installation speed. This one's obvious; less data takes less time to install. This is mentioned in your parent comment. 2. "But compression isn't just about saving disk space. It's also a classic example of trading CPU cycles for decreased I/O latency and bandwidth. Over the past few decades, CPU performance has gotten better (and computing resources more plentiful—more on that later) at a much faster rate than disk performance has increased. Modern hard disk seek times and rotational delays are still measured in milliseconds. In one millisecond, a 2 GHz CPU goes through two million cycles. And then, of course, there's still the actual data transfer time to consider. [...] Given the almost comical glut of CPU resources on a modern multi-core Mac under normal use, the total time needed to transfer a compressed payload from the disk and use the CPU to decompress its contents into memory will still usually be far less than the time it'd take to transfer the data in uncompressed form." It's an interesting point, but seek times and rotational delays don't apply to SSDs. This is kind of an uneasy comparison to draw with "I hate that Chromium’s snap takes more than 10 seconds to load on cold boot on a freaking SSD".
- danieldk 6y ago
  It's an interesting point, but seek times and rotational delays don't apply to SSDs. There is another reason to do compression on SSDs: you have more storage free and thus less write amplification and your SSDs will last longer. In fact some SSD controllers (e.g. SandForce controllers used to do this) compress data to reduce write amplification. https://en.wikipedia.org/wiki/SandForce#Technology https://en.wikipedia.org/wiki/SandForce#Technology The trick that they applied is that say, if you had a 500GB SSD and you stored 400GB uncompressed which the controller compressed to 200GB, the drive would still report only having 100GB free, giving it an ample 300GB of free blocks, thus greatly reducing write amplification. (Of course, the benefit of controller-level compression is gone with full-disk encryption. But I guess FDE was less popular when SandForce SSDs became popular.)

Florin_Andrei 6y ago

> snaps are compressed to save disk space The list of dumb decisions that Ubuntu has been making recently just keeps increasing.