5 ms·

4GB is really pushing the boundaries of what you should be using HTTP for. It's a nontrivial download for users with slower connection speeds, so you should re

by csense 14y ago

4GB is really pushing the boundaries of what you should be using HTTP for. It's a nontrivial download for users with slower connection speeds, so you should really have better failure recovery than a simple HTTP download provides. You should distribute it in pieces, or use bittorrent or rsync or some other protocol which allows correcting a bad block in transit. Users should download a small downloader program which speaks whatever protocol, then the downloader downloads the actual game. (Think about the way Blizzard does WoW updates.) Checking blocks should theoretically possible using HTTP transfers with byte ranges, if you have a separate index file with checksums for each block. That way you get failure recovery but serve your content from Really Efficient Webservers like nginx or Amazon S3. Anyone know of any projects that do this? You should also consider using a more modern compression algorithm like bz2 or lzma, if you have control over the application running on both ends of the connection. I'm guessing you'll see these results with Zopfli: gzip -c9 > zopfli -i1000 > bzip2 > bzip2 -c9 > xz # compressed size gzip -c9 < bzip2 < bzip2 -c9 < xz, zopfli -i1000 # compression time Not really sure about xz vs. zopfli for compression speed, but I'm sure that xz's LZMA algorithm gets way better compression than gzip for most data, due to fundamental limitations in the design of gzip's format. The only real reason to use Zopfli is when the client only knows gzip compression and you can't change it (for example, many, many people have static webpages, CSS, JS, etc., which need to support several different web browsers as clients).

zobzu 14y ago
Sounds like we'd be better off using lzma altogether.
loeg 14y ago
> Checking blocks should theoretically possible using HTTP transfers with byte ranges, if you have a separate index file with checksums for each block. That way you get failure recovery but serve your content from Really Efficient Webservers like nginx or Amazon S3. Anyone know of any projects that do this? Sure, one example is bittorrent (WebSeeding), among others[0]. Turns out bittorrent is a pretty good format for describing chunks of a file (full SHA1 per chunk, variable sized, if you care), and any httpd with byte-range support can fetch arbitrary bittorrent chunks of files. [0]: https://en.wikipedia.org/wiki/BitTorrent#Web_seeding https://en.wikipedia.org/wiki/BitTorrent#Web_seeding
Negitivefrags 14y ago
I absolutely agree that improvements could be made to our patching system, so here are a few more details as to why we do things how we do them. 1) It is actually working right now. This is the biggest deal at a startup. It's good enough. We have lots of other features to work on that will give us higher bang for our buck. 2) We wanted to be able to use an "off the shelf" CDN for distribution. It's nice to be able to farm out parts of your infrastructure you don't want to deal with. 3) We don't want to have to deal with versions in the patching system. Our patching systems first use is right in our office. The same deployment infrastructure that deploys the game to production runs in our office with every commit from a programmer or artist. When someone runs the game client (or one of the in house tools) the patching system syncs up the files to the latest version. For real production clients, there can be several versions between the last time someone ran the game client and the next time. It's easier conceptually to deal with. 4) The client is over 100,000 files. Each file is generally quite small which limits the damage of not using an actual diffing solution. The larger files like textures and audio don't diff very well anyway when artists make changes due to the nature of the data formats. 5) It was really fast to develop. It's just libcurl. Once again, we are a small startup so we have lots of things we need to be focusing on that are not our patching system. All the files are checked against hashes after download so we can correct errors. This is especially important due to the surprisingly large number of buggy caching proxies that some users seem to have in front of them. One day I would love to make the best possible patching system. Unfortunately it's just not as high on the list as so many of the other things we need to be doing!
- glomph 14y ago
  From a user perspective I wish you guys would switch to something better like bit torrent (at least offer an alternative torrent download). I almost gave up before playing the game when an interrupted download meant that I had to start again, and it is pretty frustrating not even maxing out my slow internet connection at peak times for patches. The fact that you don't have any kind of diffing solution seems like absoulte madness to me.
  - Negitivefrags 14y ago
csense 14y ago
Nitpicking my own comment: Actually bzip2 and bzip2 -c9 are the same thing, since -9 is the default for bzip2.

by csense 14y ago

zobzu 14y ago

Sounds like we'd be better off using lzma altogether.

loeg 14y ago

> Checking blocks should theoretically possible using HTTP transfers with byte ranges, if you have a separate index file with checksums for each block. That way you get failure recovery but serve your content from Really Efficient Webservers like nginx or Amazon S3. Anyone know of any projects that do this? Sure, one example is bittorrent (WebSeeding), among others[0]. Turns out bittorrent is a pretty good format for describing chunks of a file (full SHA1 per chunk, variable sized, if you care), and any httpd with byte-range support can fetch arbitrary bittorrent chunks of files. [0]: https://en.wikipedia.org/wiki/BitTorrent#Web_seeding https://en.wikipedia.org/wiki/BitTorrent#Web_seeding

Negitivefrags 14y ago

I absolutely agree that improvements could be made to our patching system, so here are a few more details as to why we do things how we do them. 1) It is actually working right now. This is the biggest deal at a startup. It's good enough. We have lots of other features to work on that will give us higher bang for our buck. 2) We wanted to be able to use an "off the shelf" CDN for distribution. It's nice to be able to farm out parts of your infrastructure you don't want to deal with. 3) We don't want to have to deal with versions in the patching system. Our patching systems first use is right in our office. The same deployment infrastructure that deploys the game to production runs in our office with every commit from a programmer or artist. When someone runs the game client (or one of the in house tools) the patching system syncs up the files to the latest version. For real production clients, there can be several versions between the last time someone ran the game client and the next time. It's easier conceptually to deal with. 4) The client is over 100,000 files. Each file is generally quite small which limits the damage of not using an actual diffing solution. The larger files like textures and audio don't diff very well anyway when artists make changes due to the nature of the data formats. 5) It was really fast to develop. It's just libcurl. Once again, we are a small startup so we have lots of things we need to be focusing on that are not our patching system. All the files are checked against hashes after download so we can correct errors. This is especially important due to the surprisingly large number of buggy caching proxies that some users seem to have in front of them. One day I would love to make the best possible patching system. Unfortunately it's just not as high on the list as so many of the other things we need to be doing!

glomph 14y ago
From a user perspective I wish you guys would switch to something better like bit torrent (at least offer an alternative torrent download). I almost gave up before playing the game when an interrupted download meant that I had to start again, and it is pretty frustrating not even maxing out my slow internet connection at peak times for patches. The fact that you don't have any kind of diffing solution seems like absoulte madness to me.
- Negitivefrags 14y ago

csense 14y ago

Nitpicking my own comment: Actually bzip2 and bzip2 -c9 are the same thing, since -9 is the default for bzip2.