7 ms·

Compression speed is around 2 KB/sec. The figure is given in nncp_v2.1.pdf. This is 1000x slower than xz, which in turn is 1000x slower than gzip. I have alwa

by eps 5y ago

Compression speed is around 2 KB/sec. The figure is given in nncp_v2.1.pdf. This is 1000x slower than xz, which in turn is 1000x slower than gzip. I have always wondered if a compression service could be a viable commercial offering and with NNCP rates and its compression speed, it looks like it could! That's for cases when you need to really compress the heck out of something, e.g. prior to distributing a large static piece of data to a very large number of recipients. I think there was a service like this, used by AAA game developers for distributing patches, but it didn't take because it had a very slow decompression... and, now reading further through the paper, it looks like an issue with NNCP too - The NNCP v2 decompression speed is similar to its compression speed. So it's a no-go for my brilliant and lazy startup idea, but fascinating stuff nonetheless.

xxs 5y ago
Yup, it isn't a dictionary based compression. Nonetheless that's a fascinating take
- thesz 5y ago
  With preprocessing it is a dictionary based compression. It uses a fixed dictionary to improve on compression.
raverbashing 5y ago
For real I know shipping speeds are important, but it seems game developers could try harder with delta compressing their updates, because it's hard to believe a tens-of-GBs download is the best they could do when you already have a BluRay disk or something (unless it's new content, of course) I wonder how much their bandwidth costs are and how quickly a reduction would pay itself.
- xxs 5y ago
  What would pay for itself is p2p, e.g. bittorrent... alas.
  - lxgr 5y ago
    Blizzard used to use Bittorrent in some of their installers/updaters a while ago. I'm not sure how viable this is today, or ever was: Many residential connections are heavily asymmetric. Players presumably aren't too excited either by the idea of continuing to seed while they already play online, with shitty ISP-provided routers suffering hundreds of milliseconds of bufferbloat whenever the uplink is saturated.
    - xxs 5y ago
      >Blizzard used to use Bittorrent in some of their installers/updaters a while ago. Yes, WoW used to have bit-torrent based update. >Many residential connections are heavily asymmetric. this is a US centric-view, though. For instance I have 300Mb up/down. At least for most of the Europe (and I suppose Asia), it's pretty common not to be restricted in any way. I could leave a bit-torrent client non-stop and won't feel it. Like I've said: alas. There is some stigma associated with p2p/torrents.
      - dividuum 5y ago
        Not sure where you are in Europe, but in Germany you have to search a bit to find a residential ISP with symmetric bandwidth. Mine is 50/10.
      - paavohtl 5y ago
        I have very rarely seen symmetric broadband plans available in Finland. In the past I've had 10/1, 24/1, 100/10 and now I have 1000/100. With ADSL connections being asymmetric was basically necessary because the bandwidth is so limited - I think most consumers would prefer 24/1 over 12/12. With fiber I think it's more about market segmentation: ISPs want businesses to buy significantly more expensive symmetric plans.
      - cinntaile 5y ago
        This is not the standard at all in Europe, maybe that's the standard in your specific country?
- cinntaile 5y ago
  Would it even pay for itself? It seems to me like bandwidth is pretty cheap compared to computation costs so why bother trying to squeeze everything out. Just do some basic compression and you're good to go. Buy a bigger SSD if necessary, with technologies like reBAR compression just slows down a game anyway.
  - literallycancer 5y ago
    As far as cloud providers go, bandwidth is where the lock in is. It's priced to make it too expensive to switch to a competitor.
- tomalpha 5y ago
  Does DRM contribute to this perhaps? Given the layers of encryption and obfuscation, and leaving aside the merits or otherwise of DRM in the first place, I can imagine it could be hard to do delta patching. At least, without making it easier to crack the DRM - e.g. you’d presumably have to decrypt the asset that needed patching, apply the delta, and re-encrypt/re-sign it all client side.
m463 5y ago
I think maybe matching the compression speed with the network speed is what you want to do. In a place with ubiquitous fast broadband, a compromise with regard to decompression speed might be ok. If you're sending something to mars... 2 kb/sec might make sense.
- xxs 5y ago
  >If you're sending something to mars... 2 kb/sec might make sense. Keep in mind that 2KB/s is on something like 3090... Massively power hungry even on the decompression.
- loeg 5y ago
  > I think maybe matching the compression speed with the network speed is what you want to do. You might be interested: Zstandard has an adaptive mode that attempts to do this.
kitd 5y ago
Compression speed is around 2 KB/sec. Apologies if this is a noob question, but could you improve it with training?
- esquire_900 5y ago
  > Apologies if this is a noob question, but could you improve it with training? Considering all other things like GPU, code quality etc. equal, the speed here mainly depends on the size of the network. Larger networks result in better performance (logarithmic), but also slower speeds (linear). There are some tricks like pruning to create significantly smaller networks with similar-ish performance, but good models are often large. Unrelated to your question, what nobody mentions is that the decompression side also needs to complete network. With 57M and 187M parameters, those files are going to be quite large, > 100MB for sure. That completely annihilates the performance wins for 1 time transfers.
  - 317070 5y ago
    No, as I understand it, the network does not need to be transmitted. The decoder learns on the same stream as the encoder, so you do not need to transmit network parameters.
    - ironSkillet 5y ago
      If I compress a large file at location A using this algorithm, and want to send it to location B, how does location B know how to decompress it?
      - clavigne 5y ago
        see the (current) top comment https://news.ycombinator.com/item?id=27244810 https://news.ycombinator.com/item?id=27244810 it's a very clever use of symmetry.
        vidarh 5y ago
agumonkey 5y ago
What about decompression speed ?
AtlasBarfed 5y ago
Hm, there are likely multiple large files that exist that are very common (DLLs, versions of shared libs, etc etc etc) that might be frequently compressed (SCP, stored in SSDs, etc) since disk space isn't free yet last I checked. Each file is really its own special theoretical multivariate space to find the best theoretical compression ratio. Usually finding the optimal or near-optimal compression is the hardest computationally. And it's basically a perfectly cacheable computation if you have some sort of database mapping algorith + parameters to apply to a given file. Decompression usually isn't as big a deal for performance. Anyway an online database for a service that has actively tried to take a known common file (match on filesize, hash, etc) down to its best compressed representation so 1) we don't have to do it every time. Cloud providers would presumably love these, since S3 probably has millions copies of many files and would 1) get to compress the file and 2) get to share its representation across all instances.
wolf550e 5y ago
This is what the games industry uses in practice. It is optimized for fast decompression. http://www.radgametools.com/oodle.htm http://www.radgametools.com/oodle.htm
- eps 5y ago
  Reads like a snake oil blurb to be honest. No comparision against any modern compressors, eg lz4, no specs, no pricing. Just "trust us, it's awesome." Looks really strange.
  - wolf550e 5y ago
    Of course there is comparison with LZ4: http://www.radgametools.com/oodlemermaidselkie.htm http://www.radgametools.com/oodlemermaidselkie.htm LZ4HC is LZ4 compressed with the slow compressor to achieve better compression. It is decompressed using regular LZ4 decompressor. They also compare with zstd and lzma, which are the open source competition. I don't know what "specs" you think are missing. They do say which architectures are supported, what the memory use is, whether multi-threading is supported, etc. No public price, true. You can read the blogs of the people who work on this code to get a feel for the caliber: https://cbloomrants.blogspot.com/ https://cbloomrants.blogspot.com/ https://fgiesen.wordpress.com/ https://fgiesen.wordpress.com/ These are known people with known reputation. The product itself is well known in the industry.

by eps 5y ago

xxs 5y ago

Yup, it isn't a dictionary based compression. Nonetheless that's a fascinating take

thesz 5y ago
With preprocessing it is a dictionary based compression. It uses a fixed dictionary to improve on compression.

raverbashing 5y ago

For real I know shipping speeds are important, but it seems game developers could try harder with delta compressing their updates, because it's hard to believe a tens-of-GBs download is the best they could do when you already have a BluRay disk or something (unless it's new content, of course) I wonder how much their bandwidth costs are and how quickly a reduction would pay itself.

xxs 5y ago
What would pay for itself is p2p, e.g. bittorrent... alas.
- lxgr 5y ago
  Blizzard used to use Bittorrent in some of their installers/updaters a while ago. I'm not sure how viable this is today, or ever was: Many residential connections are heavily asymmetric. Players presumably aren't too excited either by the idea of continuing to seed while they already play online, with shitty ISP-provided routers suffering hundreds of milliseconds of bufferbloat whenever the uplink is saturated.
  - xxs 5y ago
    >Blizzard used to use Bittorrent in some of their installers/updaters a while ago. Yes, WoW used to have bit-torrent based update. >Many residential connections are heavily asymmetric. this is a US centric-view, though. For instance I have 300Mb up/down. At least for most of the Europe (and I suppose Asia), it's pretty common not to be restricted in any way. I could leave a bit-torrent client non-stop and won't feel it. Like I've said: alas. There is some stigma associated with p2p/torrents.
    - dividuum 5y ago
      Not sure where you are in Europe, but in Germany you have to search a bit to find a residential ISP with symmetric bandwidth. Mine is 50/10.
    - paavohtl 5y ago
      I have very rarely seen symmetric broadband plans available in Finland. In the past I've had 10/1, 24/1, 100/10 and now I have 1000/100. With ADSL connections being asymmetric was basically necessary because the bandwidth is so limited - I think most consumers would prefer 24/1 over 12/12. With fiber I think it's more about market segmentation: ISPs want businesses to buy significantly more expensive symmetric plans.
    - cinntaile 5y ago
      This is not the standard at all in Europe, maybe that's the standard in your specific country?
cinntaile 5y ago
Would it even pay for itself? It seems to me like bandwidth is pretty cheap compared to computation costs so why bother trying to squeeze everything out. Just do some basic compression and you're good to go. Buy a bigger SSD if necessary, with technologies like reBAR compression just slows down a game anyway.
- literallycancer 5y ago
  As far as cloud providers go, bandwidth is where the lock in is. It's priced to make it too expensive to switch to a competitor.
tomalpha 5y ago
Does DRM contribute to this perhaps? Given the layers of encryption and obfuscation, and leaving aside the merits or otherwise of DRM in the first place, I can imagine it could be hard to do delta patching. At least, without making it easier to crack the DRM - e.g. you’d presumably have to decrypt the asset that needed patching, apply the delta, and re-encrypt/re-sign it all client side.

m463 5y ago

I think maybe matching the compression speed with the network speed is what you want to do. In a place with ubiquitous fast broadband, a compromise with regard to decompression speed might be ok. If you're sending something to mars... 2 kb/sec might make sense.

xxs 5y ago
>If you're sending something to mars... 2 kb/sec might make sense. Keep in mind that 2KB/s is on something like 3090... Massively power hungry even on the decompression.
loeg 5y ago
> I think maybe matching the compression speed with the network speed is what you want to do. You might be interested: Zstandard has an adaptive mode that attempts to do this.

kitd 5y ago

Compression speed is around 2 KB/sec. Apologies if this is a noob question, but could you improve it with training?

esquire_900 5y ago
> Apologies if this is a noob question, but could you improve it with training? Considering all other things like GPU, code quality etc. equal, the speed here mainly depends on the size of the network. Larger networks result in better performance (logarithmic), but also slower speeds (linear). There are some tricks like pruning to create significantly smaller networks with similar-ish performance, but good models are often large. Unrelated to your question, what nobody mentions is that the decompression side also needs to complete network. With 57M and 187M parameters, those files are going to be quite large, > 100MB for sure. That completely annihilates the performance wins for 1 time transfers.
- 317070 5y ago
  No, as I understand it, the network does not need to be transmitted. The decoder learns on the same stream as the encoder, so you do not need to transmit network parameters.
  - ironSkillet 5y ago
    If I compress a large file at location A using this algorithm, and want to send it to location B, how does location B know how to decompress it?
    - clavigne 5y ago
      see the (current) top comment https://news.ycombinator.com/item?id=27244810 https://news.ycombinator.com/item?id=27244810 it's a very clever use of symmetry.
      - vidarh 5y ago

agumonkey 5y ago

What about decompression speed ?

AtlasBarfed 5y ago

Hm, there are likely multiple large files that exist that are very common (DLLs, versions of shared libs, etc etc etc) that might be frequently compressed (SCP, stored in SSDs, etc) since disk space isn't free yet last I checked. Each file is really its own special theoretical multivariate space to find the best theoretical compression ratio. Usually finding the optimal or near-optimal compression is the hardest computationally. And it's basically a perfectly cacheable computation if you have some sort of database mapping algorith + parameters to apply to a given file. Decompression usually isn't as big a deal for performance. Anyway an online database for a service that has actively tried to take a known common file (match on filesize, hash, etc) down to its best compressed representation so 1) we don't have to do it every time. Cloud providers would presumably love these, since S3 probably has millions copies of many files and would 1) get to compress the file and 2) get to share its representation across all instances.

wolf550e 5y ago

This is what the games industry uses in practice. It is optimized for fast decompression. http://www.radgametools.com/oodle.htm http://www.radgametools.com/oodle.htm

eps 5y ago
Reads like a snake oil blurb to be honest. No comparision against any modern compressors, eg lz4, no specs, no pricing. Just "trust us, it's awesome." Looks really strange.
- wolf550e 5y ago
  Of course there is comparison with LZ4: http://www.radgametools.com/oodlemermaidselkie.htm http://www.radgametools.com/oodlemermaidselkie.htm LZ4HC is LZ4 compressed with the slow compressor to achieve better compression. It is decompressed using regular LZ4 decompressor. They also compare with zstd and lzma, which are the open source competition. I don't know what "specs" you think are missing. They do say which architectures are supported, what the memory use is, whether multi-threading is supported, etc. No public price, true. You can read the blogs of the people who work on this code to get a feel for the caliber: https://cbloomrants.blogspot.com/ https://cbloomrants.blogspot.com/ https://fgiesen.wordpress.com/ https://fgiesen.wordpress.com/ These are known people with known reputation. The product itself is well known in the industry.