5 ms·

So we distribute our game client over HTTP and it's something like 4GB compressed using gzip. When I saw zopfli, my immediate reaction was that it should save

So we distribute our game client over HTTP and it's something like 4GB compressed using gzip. When I saw zopfli, my immediate reaction was that it should save a non-trivial amount of money for us while leading to a slightly faster download for our users. Hearing a 100 times increase in compression time doesn't sound like much compared to the number of times the files are downloaded right? The last incremental patch we did to the game took 79 seconds in our build system to compress with the usual gzip. This is only compressing the files that were changed since the last patch, not the entire game client, so larger patches would take longer. If we changed to zopfli, it would take 2.2 hours. That would put a serious dent in our agility. When we are trying to deploy a new patch, adding 2.2 hours (or more) somewhere in the middle of it really isn't viable. (It's already maddening when the rest of the process takes ~40 minutes with all the automated tests that we run). So then lets look at the savings that this would buy us. We distributed about 30TB yesterday. Bandwidth costs 2c per gig. This would save us about 30 dollars a day. So much for that!

newman314 14y ago
Did you get noticeable compression from zopfli though?
shard 14y ago
What about the possibility of simply sending the gzip version until the zopfli version is ready, then switching over? Seems like you'd get the best of both worlds.
moonboots 14y ago
Have you considered a newer compression program like xz [1]? The backwards compatibility of Zopfli is only necessary for clients that are relatively hard to update, e.g. browsers. If you are already distributing a custom game client, you could bundle an arbitrary compression library that has a better compression ratio, faster compression time, etc. than gzip. [1] http://en.wikipedia.org/wiki/Xz http://en.wikipedia.org/wiki/Xz
- jmspring 14y ago
  I suspect compatibility with deflate is the main factor. There are a number of LZ related compression algorithms with different properties - speed, compression, suitable to embed, etc. It wholly depends on what the main goal is, in this case it seems better at rest storage for data that doesn't change regularly is key.
- jmspring 14y ago
  I suspect compatibility with deflate is the main factor. There are a number of LZ related compression algorithms with different properties - speed, compression, suitable to embed, etc. It wholly depends on what the main goal is, in this case it seems better at rest storage for data that doesn't change regularly is key.
cheald 14y ago
I'd just like to say that your team's patch delivery mechanism/speed is ridiculously impressive. I've been playing since early closed beta and the way you guys runs things makes me just plain happy. Kudos!
csense 14y ago
4GB is really pushing the boundaries of what you should be using HTTP for. It's a nontrivial download for users with slower connection speeds, so you should really have better failure recovery than a simple HTTP download provides. You should distribute it in pieces, or use bittorrent or rsync or some other protocol which allows correcting a bad block in transit. Users should download a small downloader program which speaks whatever protocol, then the downloader downloads the actual game. (Think about the way Blizzard does WoW updates.) Checking blocks should theoretically possible using HTTP transfers with byte ranges, if you have a separate index file with checksums for each block. That way you get failure recovery but serve your content from Really Efficient Webservers like nginx or Amazon S3. Anyone know of any projects that do this? You should also consider using a more modern compression algorithm like bz2 or lzma, if you have control over the application running on both ends of the connection. I'm guessing you'll see these results with Zopfli: gzip -c9 > zopfli -i1000 > bzip2 > bzip2 -c9 > xz # compressed size gzip -c9 < bzip2 < bzip2 -c9 < xz, zopfli -i1000 # compression time Not really sure about xz vs. zopfli for compression speed, but I'm sure that xz's LZMA algorithm gets way better compression than gzip for most data, due to fundamental limitations in the design of gzip's format. The only real reason to use Zopfli is when the client only knows gzip compression and you can't change it (for example, many, many people have static webpages, CSS, JS, etc., which need to support several different web browsers as clients).
- zobzu 14y ago
  Sounds like we'd be better off using lzma altogether.
- loeg 14y ago
  > Checking blocks should theoretically possible using HTTP transfers with byte ranges, if you have a separate index file with checksums for each block. That way you get failure recovery but serve your content from Really Efficient Webservers like nginx or Amazon S3. Anyone know of any projects that do this? Sure, one example is bittorrent (WebSeeding), among others[0]. Turns out bittorrent is a pretty good format for describing chunks of a file (full SHA1 per chunk, variable sized, if you care), and any httpd with byte-range support can fetch arbitrary bittorrent chunks of files. [0]: https://en.wikipedia.org/wiki/BitTorrent#Web_seeding https://en.wikipedia.org/wiki/BitTorrent#Web_seeding
endtime 14y ago
You could do a gzip patch, and then replace it with the Zopfli patch 2 hours later...no?
ck2 14y ago
Try 7zip's gzip output in "ultra" profile (mx9) - it's within 1% of zopfli and a fraction of the time to compress.
cperciva 14y ago
The last incremental patch we did to the game took 79 seconds in our build system to compress with the usual gzip. This is only compressing the files that were changed since the last patch, not the entire game client, so larger patches would take longer. If we changed to zopfli, it would take 2.2 hours. Why are you not using bsdiff? This seems like a bug.
- pja 14y ago
  Game assets don't change in the ways that compiled binaries do, so bsdiff doesn't sound like it would be appropriate.
rorrr 14y ago
You should send compressed DIFFs only, and not whole changed files.
thret 14y ago
How much time would it save your users, collectively?
kokey 14y ago
It will be nice for your customers to have a smaller download. To speed this sort of thing up you can probably spin up an elastic cluster to do the compression work, and shut down when it's done.
Tichy 14y ago
Apart from what others said, perhaps faster downloads also would lead to better user retention.
- cheald 14y ago
  Speaking as a customer of theirs, it's already impressively fast. :)
qznc 14y ago
Why not used BitTorrent? For example, Facebook uses it to deploy their server code. http://torrentfreak.com/facebook-uses-bittorrent-and-they-love-it-100625/ http://torrentfreak.com/facebook-uses-bittorrent-and-they-lo...

newman314 14y ago

Did you get noticeable compression from zopfli though?

shard 14y ago

What about the possibility of simply sending the gzip version until the zopfli version is ready, then switching over? Seems like you'd get the best of both worlds.

moonboots 14y ago

Have you considered a newer compression program like xz [1]? The backwards compatibility of Zopfli is only necessary for clients that are relatively hard to update, e.g. browsers. If you are already distributing a custom game client, you could bundle an arbitrary compression library that has a better compression ratio, faster compression time, etc. than gzip. [1] http://en.wikipedia.org/wiki/Xz http://en.wikipedia.org/wiki/Xz

jmspring 14y ago
I suspect compatibility with deflate is the main factor. There are a number of LZ related compression algorithms with different properties - speed, compression, suitable to embed, etc. It wholly depends on what the main goal is, in this case it seems better at rest storage for data that doesn't change regularly is key.
jmspring 14y ago
I suspect compatibility with deflate is the main factor. There are a number of LZ related compression algorithms with different properties - speed, compression, suitable to embed, etc. It wholly depends on what the main goal is, in this case it seems better at rest storage for data that doesn't change regularly is key.

cheald 14y ago

I'd just like to say that your team's patch delivery mechanism/speed is ridiculously impressive. I've been playing since early closed beta and the way you guys runs things makes me just plain happy. Kudos!

csense 14y ago

4GB is really pushing the boundaries of what you should be using HTTP for. It's a nontrivial download for users with slower connection speeds, so you should really have better failure recovery than a simple HTTP download provides. You should distribute it in pieces, or use bittorrent or rsync or some other protocol which allows correcting a bad block in transit. Users should download a small downloader program which speaks whatever protocol, then the downloader downloads the actual game. (Think about the way Blizzard does WoW updates.) Checking blocks should theoretically possible using HTTP transfers with byte ranges, if you have a separate index file with checksums for each block. That way you get failure recovery but serve your content from Really Efficient Webservers like nginx or Amazon S3. Anyone know of any projects that do this? You should also consider using a more modern compression algorithm like bz2 or lzma, if you have control over the application running on both ends of the connection. I'm guessing you'll see these results with Zopfli: gzip -c9 > zopfli -i1000 > bzip2 > bzip2 -c9 > xz # compressed size gzip -c9 < bzip2 < bzip2 -c9 < xz, zopfli -i1000 # compression time Not really sure about xz vs. zopfli for compression speed, but I'm sure that xz's LZMA algorithm gets way better compression than gzip for most data, due to fundamental limitations in the design of gzip's format. The only real reason to use Zopfli is when the client only knows gzip compression and you can't change it (for example, many, many people have static webpages, CSS, JS, etc., which need to support several different web browsers as clients).

zobzu 14y ago
Sounds like we'd be better off using lzma altogether.
loeg 14y ago
> Checking blocks should theoretically possible using HTTP transfers with byte ranges, if you have a separate index file with checksums for each block. That way you get failure recovery but serve your content from Really Efficient Webservers like nginx or Amazon S3. Anyone know of any projects that do this? Sure, one example is bittorrent (WebSeeding), among others[0]. Turns out bittorrent is a pretty good format for describing chunks of a file (full SHA1 per chunk, variable sized, if you care), and any httpd with byte-range support can fetch arbitrary bittorrent chunks of files. [0]: https://en.wikipedia.org/wiki/BitTorrent#Web_seeding https://en.wikipedia.org/wiki/BitTorrent#Web_seeding

endtime 14y ago

You could do a gzip patch, and then replace it with the Zopfli patch 2 hours later...no?

ck2 14y ago

Try 7zip's gzip output in "ultra" profile (mx9) - it's within 1% of zopfli and a fraction of the time to compress.

cperciva 14y ago

The last incremental patch we did to the game took 79 seconds in our build system to compress with the usual gzip. This is only compressing the files that were changed since the last patch, not the entire game client, so larger patches would take longer. If we changed to zopfli, it would take 2.2 hours. Why are you not using bsdiff? This seems like a bug.

pja 14y ago
Game assets don't change in the ways that compiled binaries do, so bsdiff doesn't sound like it would be appropriate.

rorrr 14y ago

You should send compressed DIFFs only, and not whole changed files.

thret 14y ago

How much time would it save your users, collectively?

kokey 14y ago

It will be nice for your customers to have a smaller download. To speed this sort of thing up you can probably spin up an elastic cluster to do the compression work, and shut down when it's done.

Tichy 14y ago

Apart from what others said, perhaps faster downloads also would lead to better user retention.

cheald 14y ago
Speaking as a customer of theirs, it's already impressively fast. :)

qznc 14y ago

Why not used BitTorrent? For example, Facebook uses it to deploy their server code. http://torrentfreak.com/facebook-uses-bittorrent-and-they-love-it-100625/ http://torrentfreak.com/facebook-uses-bittorrent-and-they-lo...