5 ms·
Go is used, sure, but the cool part about this is the binary Railgun protocol. Really smart. Send only file hashes and binary diffs back and forth, do a little
by calinet6 14y ago
Go is used, sure, but the cool part about this is the binary Railgun protocol. Really smart. Send only file hashes and binary diffs back and forth, do a little extra computation to figure out the changes, but only send the absolute minimum data you need to the CDN. That's just smart, and frankly, I hope other CDNs have been doing this already, because at any high volume it seems to be an obvious solution.
So that brings up the question—is this just something CloudFlare is announcing for the PR, or is it actually innovative?
- 0x0 14y agoIt sounds like they reinvented rsync to me?
- jgrahamc 14y agoNo, because we have more information than rsync does. We own both ends of the connection and can keep versions synchronized.
- 0x0 14y agoThat sounds interesting, could you elaborate on how it is different from rsync though? "Keep versions synchronized" is a bit vague
- jgrahamc 14y agoThe piece in the CloudFlare network and the piece in the customer network are able to keep track of which page versions they each have and so the part in the CloudFlare network sends a request saying "Please do GET /foo and compress it against version X". That means that at request time there's no back-and-forth between the components deciding what compression dictionary to use.
- 0x0 14y agoA bit like rsync's --fuzzy or --compare-dest then?
- jgrahamc 14y agoWell, fuzzy tries to find something to use as a 'destination' file so it can send across some hashes. Railgun has more complete information because it is keeping synchronized and thus the part making a request can specify the dictionary to compress with in a single hash.
- 0x0 14y agoThanks for the explanation, that does sound useful! :)
- IheartApplesDix 14y agoDon't you understand? We need you to accept that this is a new technology and a ground-breaking algorithm and a new innovative (and valuable, non-obvious) technology. CloudFlare was established in 2007 with the goal to develop a faster, safter, better internet. CloudFlare, the web performance and security company, set records this month hitting more than 100 million daily active users and more than 50 billion monthly page views!
- DannyBee 14y agoWell, no good binary delta algorithm uses compression dictionaries anyway (since they are binary deltas, not compression algorithms :P), except to compress the newly added strings, which you can't avoid. Note of course, that relying on the data not being corrupt on the client (which you must if you assume the compression dictionaries are sane) is dangerous. I assume you guys must store some checksum that you compare once to make sure when someone says "i have version 5, delta against this", that they really have a good copy of version 5? SVN used to what you are suggesting, btw. We only send clients deltas against the versions they already have, and precompute them in some cases :)
- jgrahamc 14y agoI assume you guys must store some checksum that you compare once to make sure when someone says "i have version 5, delta against this", that they really have a good copy of version 5? Yes.
- huhtenberg 14y agoFor what it's worth, this is fairly standard binary patching approach as used in software updates. I am aware of at least two mainstream titles that do this, and I'd be surprised if Firefox, for example, doesn't push updates this way. (edit) That's an awesome name by the way. Railgun.
- jgrahamc 14y agoCan't claim credit for the name. I wanted to call it Rocket Sled.
- eastdakota 14y agoI'm glad we didn't call it Rocket Sled.
- abcd_f 14y ago> we have more information than rsync does. To what end? Rsync too works off both copies.
- DannyBee 14y agorsync is going to perform checksums on blocks to see if the blocks are the same. It transmits these checksums, and where the checksums differ, it deltas the blocks. Note that insertion/deletion in a file can push block boundaries off between two files, causing a problem known as "stream alignment", which can cause your binary delta to be much larger because it doesn't realize the block really shifted 16384 bytes over (or whatever), and so it thinks the client really doesn't have any of the bytes of that block. In any case, if you know the files are related, you 1. Don't need to do any of this. You can simply send the binary delta that is is usually copy/add instructions (IE copy offset 16384, length 500 to offset 32768) 2. Can precompute the deltas. You can actually precompute in any case, it just makes no sense unless you know you will be diffed against something else.
- huhtenberg 14y agoI always thought rsync detected block moves and that's what made it a worthy PhD thesis.
- DannyBee 14y agoYes, I simplified and I shouldn't have. It does detect them, but it does have a minimum size of block move it can detect due to the signature matching method.
- zobzu 14y agoI thought that too. It'd be interesting to see a comparison of the two software designs with actual difference in resource usage (cpu, io, bandwidth). That would be really cool in fact.
- dubya 14y agoI think it's more like rsync + git. You have copies of previous versions, and just ask for their hash to figure out which previous version to diff against the current version, then send the diff.
- calinet6 14y agoI think that's a good analogy; it's rsync with versioning. Very cool.
- judofyr 14y agoNot sure why you're downvoted. This is basically what rsync is: a smart algorithm for doing rolling checksums and only sending diffs.
- coldtea 14y agoBecause every diff algorithm "reinvents rsync", right?