4 ms·

I'm curious about the viability of the following approach. Can a generic static dictionary be made to be held on the client side such that said compressor/deco

I'm curious about the viability of the following approach. Can a generic static dictionary be made to be held on the client side such that said compressor/decompressor can utilize efficiently? This would prevent the need to send along the dictionary with the compressed package every time. Even at an 80/20 - 90/10 success rate (of course we'd have a fall back to worse case), wouldn't this be a great advancement and reduce massive network load? With modern hd sizes, many could spare a few gigabites which could be dialed up/down as necessary depending on how optimized you'd want it. I would think we could identify particular binary patterns based on the users xfer habits (eg downloading text vs application vs audio vs video) and have different dictionaries optimized to their usage (eg roku would just store dictionaries maximized for video usage)

bryanlarsen 11y ago
Yup, that's called SDCH (shared dictionary compression for HTTP, colloquially "sandwich"), which was first proposed in 2009 by Google. Brotli supports SDCH, but it's an additional component. 1: https://engineering.linkedin.com/shared-dictionary-compression-http-linkedin https://engineering.linkedin.com/shared-dictionary-compressi...
cbr 11y ago
That sounds a lot like sdch, which you can actually use today if you're serving to in Chrome or Opera. (Look for "Accept-Encoding: sdch".)
twotwotwo 11y ago
Brotli has a static dictionary, an appendix of the draft: https://tools.ietf.org/html/draft-alakuijala-brotli-01 https://tools.ietf.org/html/draft-alakuijala-brotli-01. It comes up a few times in the comments here: https://news.ycombinator.com/item?id=7894299 https://news.ycombinator.com/item?id=7894299 . It has words and phrases in various languages, bits of HTML and JavaScript, and miscellaneous wacky stuff--looks like the kind of thing you could generate from a Web crawl, say.
- mrits 11y ago
  Is there anything out that let you have "pluggable" dictionaries? I have a client-server communication that often sends 90% of the same words. I'd like to build to build this dictionary on the server and ship it down to the clients. I prototyped something myself and SHDC for the browser was the closet I could find.

bryanlarsen 11y ago

Yup, that's called SDCH (shared dictionary compression for HTTP, colloquially "sandwich"), which was first proposed in 2009 by Google. Brotli supports SDCH, but it's an additional component. 1: https://engineering.linkedin.com/shared-dictionary-compression-http-linkedin https://engineering.linkedin.com/shared-dictionary-compressi...

cbr 11y ago

That sounds a lot like sdch, which you can actually use today if you're serving to in Chrome or Opera. (Look for "Accept-Encoding: sdch".)

twotwotwo 11y ago

Brotli has a static dictionary, an appendix of the draft: https://tools.ietf.org/html/draft-alakuijala-brotli-01 https://tools.ietf.org/html/draft-alakuijala-brotli-01. It comes up a few times in the comments here: https://news.ycombinator.com/item?id=7894299 https://news.ycombinator.com/item?id=7894299 . It has words and phrases in various languages, bits of HTML and JavaScript, and miscellaneous wacky stuff--looks like the kind of thing you could generate from a Web crawl, say.

mrits 11y ago
Is there anything out that let you have "pluggable" dictionaries? I have a client-server communication that often sends 90% of the same words. I'd like to build to build this dictionary on the server and ship it down to the clients. I prototyped something myself and SHDC for the browser was the closet I could find.