5 ms·

Eliminate HTTP requests using JavaScript diffs and LocalStorage

78 points by joshfraser 13y ago

gruseom 13y ago
After applying a diff, how does it load the new code? Eval?
- aaronem 13y ago
  I wondered that myself, but the answer is no; it appends a script element, whose content is the updated code, to the head element. [1] [1] https://github.com/diurnalist/dynosrc/blob/master/dynoSrc.js#L119 https://github.com/diurnalist/dynosrc/blob/master/dynoSrc.js...
mikegioia 13y ago
The problem here is that this could remove your ability to use localstorage in your application for anything but your JS source. We were hitting issues with some browser's 2-3MB limit when storing data for offline use.
- eli 13y ago
  I believe it's at least 5MB on all major browsers (that support it).... but I think text is stored UTF-16, so that gets used up more quickly than you might expect.
  - Fishkins 13y ago
    I've been using LZString for data I put in LocalStorage. Obviously you can still hit the limit, but it takes quite a bit longer. http://pieroxy.net/blog/pages/lz-string/index.html http://pieroxy.net/blog/pages/lz-string/index.html
    - diurnalist 13y ago
      Nice! Thanks for the link. I've opened an issue for us to track adding compression in and out of the cache here: https://github.com/diurnalist/dynosrc/issues/4 https://github.com/diurnalist/dynosrc/issues/4
    - recuter 13y ago
      Quite a bit being basically twice as long. So you're up to localstorage being usable with 6-10MB now which is nothing to sneeze at.
      - Fishkins 13y ago
        For the data I was compressing (JSON) it was getting it to about 17% of the original size. Obviously that will vary depending on your data.
- diurnalist 13y ago
  Hmm, good point. Given that we conceptualized and implemented the first iteration of this over a hackathon, I'm not surprised we missed some important things ;) I've added an issue to track allowing configuring how much cache to allow the JS storage to eat up: https://github.com/diurnalist/dynosrc/issues/5 https://github.com/diurnalist/dynosrc/issues/5
- recuter 13y ago
  I think people would appreciate if you elaborate on which browsers were problematic. I assume the old Android 'Browser' and iOS<6 as well as oldIE?
vezzy-fnord 13y ago
This appears to be a rudimentary implementation of a delta encoding scheme? I am unsure as to whether this will contribute to a significant performance gain, or if most websites even require one like this in the first place.
iLoch 13y ago
Stopped reading after I saw the meme. :(
deleted 13y ago
[deleted]
JetSpiegel 13y ago
"a lot of people have 4G" Welcome to the real world. What do they mean by a lot?
- hayksaakian 13y ago
  a lot of people (who won't close your site because it's slow) have 4G
adnrw 13y ago
This is a neat idea, but a problem I didn't see discussed in the article is that on iOS (at least), LocalStorage is disabled when running in private mode. There are ways to detect it and work around it obviously but considering this is pitched as a solution for mobile users, it's probably something to consider.
- diurnalist 13y ago
  Good point. I believe if the cache is disabled the fallback behavior will be that all the JS assets just get inlined in full on the page, which is still arguably better than requiring an extra HTTP request. I haven't tested this yet, though, so it may not work out of the box yet.
frio80 13y ago
I remember seeing something similar to this a while back called Diffable http://googlediffable.blogspot.com/ http://googlediffable.blogspot.com/. Looks like the same thing but maybe I'm wrong... Still cool though.
ricardobeat 13y ago
Google Chrome has had native support for SDHC (http://en.wikipedia.org/wiki/Shared_Dictionary_Compression_Over_HTTP http://en.wikipedia.org/wiki/Shared_Dictionary_Compression_O...) for a long time now, but it has had near zero adoption. CloudFlare also does something similar within their own network, they call it Railgun: http://blog.cloudflare.com/railgun-in-the-real-world http://blog.cloudflare.com/railgun-in-the-real-world
- diurnalist 13y ago
  One of the authors of the dynoSRC hackathon project here. Thanks for the link! I wasn't aware of this white paper. Would be great to integrate this as an automatic feature in the dynoSRC express middleware. I've added an issue for us to look into (and implement if feasible) SDCH here: https://github.com/diurnalist/dynosrc/issues/3 https://github.com/diurnalist/dynosrc/issues/3
- est 13y ago
  Sorry to nitpick SDHC - SD card high-capacity SDCH - Shared Dictionary Compression over HTTP
riskable 13y ago
Gate One already does this (though it sends whole files not just the diffs). Once a client has an asset (JS, CSS) there's no reason for it to be re-downloaded unless there's a change. Another improvement in Gate One's implementation is that it uses IndexedDB instead of localStorage. This is significantly more performant--especially as you add more/larger assets (localStorage blocks--always--whenever it is used). It also downloads all assets over the WebSocket connection which has significantly improved performance over multiple GET requests. It's quite fast.
- ricardobeat 13y ago
  Caching with localStorage is not new, the whole point is sending diffs for changes instead of re-downloading everything. This improves performance even for very dynamic data that is constantly refreshed.
ape4 13y ago
premature optimization, anyone
- voltagex_ 13y ago
  When web "apps" are over 1 megabyte initial load, no, I don't think this is premature. I think it's long overdue.
ddebernardy 13y ago
Doesn't this end up eliminating URIs and suffer from the same problems as hashbangs? (Or perhaps those problems have been solved and fixed since?)
nornagon 13y ago
> computes diffs by forking a git process on the server sounds expensive. are the diffs cached?
cromwellian 13y ago
If you have any kind of JS minification in your pipeline, then just applying git diff is likely to produce large patches for small changes as renaming and compiler optimizations on Javascript can vary wildly. Google has had diff tools like this for awhile, as such, the Closure Compiler contains a feature called stable name maps to prevent a small change from causing all of the symbols globally to be minified differently. It's outputs two files of variable and property names, and subsequent compilers can attempt to minimize diff-cheese. They also help SDCH. https://groups.google.com/forum/#!topic/closure-compiler-discuss/MrENQdmovSE https://groups.google.com/forum/#!topic/closure-compiler-dis...
- diurnalist 13y ago
  DynoSRC will actually insert white space into minified files before doing the diff so it can still do line diffs. We were and are concerned about the overhead of a diff on the minfication use-case, but it's promising that some minifiers like Closure have settings you can use to ensure that two versions of a file will match up relatively well when minified. Thanks for the link!

78 points by joshfraser 13y ago

gruseom 13y ago

After applying a diff, how does it load the new code? Eval?

aaronem 13y ago
I wondered that myself, but the answer is no; it appends a script element, whose content is the updated code, to the head element. [1] [1] https://github.com/diurnalist/dynosrc/blob/master/dynoSrc.js#L119 https://github.com/diurnalist/dynosrc/blob/master/dynoSrc.js...

mikegioia 13y ago

The problem here is that this could remove your ability to use localstorage in your application for anything but your JS source. We were hitting issues with some browser's 2-3MB limit when storing data for offline use.

eli 13y ago
I believe it's at least 5MB on all major browsers (that support it).... but I think text is stored UTF-16, so that gets used up more quickly than you might expect.
- Fishkins 13y ago
  I've been using LZString for data I put in LocalStorage. Obviously you can still hit the limit, but it takes quite a bit longer. http://pieroxy.net/blog/pages/lz-string/index.html http://pieroxy.net/blog/pages/lz-string/index.html
  - diurnalist 13y ago
    Nice! Thanks for the link. I've opened an issue for us to track adding compression in and out of the cache here: https://github.com/diurnalist/dynosrc/issues/4 https://github.com/diurnalist/dynosrc/issues/4
  - recuter 13y ago
    Quite a bit being basically twice as long. So you're up to localstorage being usable with 6-10MB now which is nothing to sneeze at.
    - Fishkins 13y ago
      For the data I was compressing (JSON) it was getting it to about 17% of the original size. Obviously that will vary depending on your data.
diurnalist 13y ago
Hmm, good point. Given that we conceptualized and implemented the first iteration of this over a hackathon, I'm not surprised we missed some important things ;) I've added an issue to track allowing configuring how much cache to allow the JS storage to eat up: https://github.com/diurnalist/dynosrc/issues/5 https://github.com/diurnalist/dynosrc/issues/5
recuter 13y ago
I think people would appreciate if you elaborate on which browsers were problematic. I assume the old Android 'Browser' and iOS<6 as well as oldIE?

vezzy-fnord 13y ago

This appears to be a rudimentary implementation of a delta encoding scheme? I am unsure as to whether this will contribute to a significant performance gain, or if most websites even require one like this in the first place.

iLoch 13y ago

Stopped reading after I saw the meme. :(

deleted 13y ago

[deleted]

JetSpiegel 13y ago

"a lot of people have 4G" Welcome to the real world. What do they mean by a lot?

hayksaakian 13y ago
a lot of people (who won't close your site because it's slow) have 4G

adnrw 13y ago

This is a neat idea, but a problem I didn't see discussed in the article is that on iOS (at least), LocalStorage is disabled when running in private mode. There are ways to detect it and work around it obviously but considering this is pitched as a solution for mobile users, it's probably something to consider.

diurnalist 13y ago
Good point. I believe if the cache is disabled the fallback behavior will be that all the JS assets just get inlined in full on the page, which is still arguably better than requiring an extra HTTP request. I haven't tested this yet, though, so it may not work out of the box yet.

frio80 13y ago

I remember seeing something similar to this a while back called Diffable http://googlediffable.blogspot.com/ http://googlediffable.blogspot.com/. Looks like the same thing but maybe I'm wrong... Still cool though.

ricardobeat 13y ago

Google Chrome has had native support for SDHC (http://en.wikipedia.org/wiki/Shared_Dictionary_Compression_Over_HTTP http://en.wikipedia.org/wiki/Shared_Dictionary_Compression_O...) for a long time now, but it has had near zero adoption. CloudFlare also does something similar within their own network, they call it Railgun: http://blog.cloudflare.com/railgun-in-the-real-world http://blog.cloudflare.com/railgun-in-the-real-world

diurnalist 13y ago
One of the authors of the dynoSRC hackathon project here. Thanks for the link! I wasn't aware of this white paper. Would be great to integrate this as an automatic feature in the dynoSRC express middleware. I've added an issue for us to look into (and implement if feasible) SDCH here: https://github.com/diurnalist/dynosrc/issues/3 https://github.com/diurnalist/dynosrc/issues/3
est 13y ago
Sorry to nitpick SDHC - SD card high-capacity SDCH - Shared Dictionary Compression over HTTP

riskable 13y ago

Gate One already does this (though it sends whole files not just the diffs). Once a client has an asset (JS, CSS) there's no reason for it to be re-downloaded unless there's a change. Another improvement in Gate One's implementation is that it uses IndexedDB instead of localStorage. This is significantly more performant--especially as you add more/larger assets (localStorage blocks--always--whenever it is used). It also downloads all assets over the WebSocket connection which has significantly improved performance over multiple GET requests. It's quite fast.

ricardobeat 13y ago
Caching with localStorage is not new, the whole point is sending diffs for changes instead of re-downloading everything. This improves performance even for very dynamic data that is constantly refreshed.

ape4 13y ago

premature optimization, anyone

voltagex_ 13y ago
When web "apps" are over 1 megabyte initial load, no, I don't think this is premature. I think it's long overdue.

ddebernardy 13y ago

Doesn't this end up eliminating URIs and suffer from the same problems as hashbangs? (Or perhaps those problems have been solved and fixed since?)

nornagon 13y ago

> computes diffs by forking a git process on the server sounds expensive. are the diffs cached?

cromwellian 13y ago

If you have any kind of JS minification in your pipeline, then just applying git diff is likely to produce large patches for small changes as renaming and compiler optimizations on Javascript can vary wildly. Google has had diff tools like this for awhile, as such, the Closure Compiler contains a feature called stable name maps to prevent a small change from causing all of the symbols globally to be minified differently. It's outputs two files of variable and property names, and subsequent compilers can attempt to minimize diff-cheese. They also help SDCH. https://groups.google.com/forum/#!topic/closure-compiler-discuss/MrENQdmovSE https://groups.google.com/forum/#!topic/closure-compiler-dis...

diurnalist 13y ago
DynoSRC will actually insert white space into minified files before doing the diff so it can still do line diffs. We were and are concerned about the overhead of a diff on the minfication use-case, but it's promising that some minifiers like Closure have settings you can use to ensure that two versions of a file will match up relatively well when minified. Thanks for the link!