4 ms·

Last November I had covid and during that time I wrote a little pastebin that runs in the browser, compresses the paste data with Brotli, and puts it in the URL

Last November I had covid and during that time I wrote a little pastebin that runs in the browser, compresses the paste data with Brotli, and puts it in the URL. It has line numbers, file naming, and syntax highlighting so it's feature complete for my own use. Here's a demo, but beware, the URLs are loooong: https://nicd.gitlab.io/t/#NhfW?Qm3F?6-&22c&CQZXuP+Aej5OXzXk7WPSpyN1MjAdr+PwAZltsgU&3R8c8cCSUJBybp+~U7~IOktUPAS/rXc+60kTVsWT21IwCw$r2o+Em3N5zyYc?qhE7ECJWHoSjQ8NemN8BZk_2UQ~Nf-gUr4Y/j854zD4RK=/QKB$qJy&uNSzZiGpk3dCj0alu?CaHH9Uaqpk9kh~1so0DqHLIdmIYJ8C2gpA625pD-VDal-rkL/egUCa/P$2f5kdOdgipOVc?ikrRv8bT5$ADVQR7PRt$kS4TpNO9&P74.xsv-ZsZVEHW6dTExkj8u1Pr3JZ=HJ7+bIx8aS6ounfMpRg=fUHqG5PYiLD9FVChk7aIb0nCMmL8K3A4VPuUDwWlO~zAta8YyiU8SGjromJ~I.edi9WoVMbp-lt1j.kydxkvLyYYRmM?N0bqbjL8P39cL~HUyi_+RW6~dsNW4c9n.sYlNN7h=aUs4jT+Mi_&EtTDdlA9Qtbj_Ln1jxFGS.OsSuHYNubts_eGJ_x https://nicd.gitlab.io/t/#NhfW?Qm3F?6-&22c&CQZXuP+Aej5OXzXk7... I find it interesting to sometimes play around with the pasted data just to see how it compresses. Sometimes adding a character or two may cut several characters from the URL length!

didericis 4y ago
Compression is one of the closest things to real life magic I’m aware of. Especially now that people have basically compressed the entire internet into a couple gig AI models. And then when I think about biology/genetics and what kind of unfathomably weird and convoluted compression is going on there, and then think about whatever it is our brain is doing… and then when I think about how like all of our perception is probably some kind of weird inescapable compressed representation of true reality… It’s all magic. Real life magic.
jansan 4y ago
Is the Brotli compression done in the browser? If so, can you give some insight how this is done?
- Ndymium 4y ago
  Yep, it is! You can find the sources on GitLab[0]. The compression is done using brotli-wasm[1]. The entry point to the compression is the syncCompress[2] function, which converts the data to UTF-8, adds the needed header bytes, and then compresses using brotli-wasm. Decompression is done in a streaming way in streamDecompress[3]. This is to avoid zip bomb attacks, where even a short URL could decompress to gigabytes of data, locking the browser. Thankfully brotli-wasm had streaming decompression builtin, I just had to write the wrapper code to update the text content and the status bar in the UI. You can find the brotli-wasm code I'm using in the vendor folder[4], there is a JS wrapper and then the WASM code (both are straight from brotli-wasm). [0] https://gitlab.com/Nicd/t https://gitlab.com/Nicd/t [1] https://github.com/httptoolkit/brotli-wasm https://github.com/httptoolkit/brotli-wasm [2] https://gitlab.com/Nicd/t/-/blob/05e587b0183ff80b1c6e050b5d30f375330572ec/tahnaroskakori.js#L196 https://gitlab.com/Nicd/t/-/blob/05e587b0183ff80b1c6e050b5d3... [3] https://gitlab.com/Nicd/t/-/blob/05e587b0183ff80b1c6e050b5d30f375330572ec/tahnaroskakori.js#L223 https://gitlab.com/Nicd/t/-/blob/05e587b0183ff80b1c6e050b5d3... [4] https://gitlab.com/Nicd/t/-/tree/05e587b0183ff80b1c6e050b5d30f375330572ec/vendor https://gitlab.com/Nicd/t/-/tree/05e587b0183ff80b1c6e050b5d3...
calebegg 4y ago
> The maximum URL size is 2048 FYI, this is almost entirely untrue. Maybe for some old versions of IE, but modern browsers can handle extremely long URIs
- Ndymium 4y ago
  Yep, I rechecked it and seems Chrome/Fx/Safari all support at least 32k. I replied in another comment I'm considering a checkbox to allow even longer URLs, though, I don't know who would want to paste them for others to read. :D

didericis 4y ago

Compression is one of the closest things to real life magic I’m aware of. Especially now that people have basically compressed the entire internet into a couple gig AI models. And then when I think about biology/genetics and what kind of unfathomably weird and convoluted compression is going on there, and then think about whatever it is our brain is doing… and then when I think about how like all of our perception is probably some kind of weird inescapable compressed representation of true reality… It’s all magic. Real life magic.

jansan 4y ago

Is the Brotli compression done in the browser? If so, can you give some insight how this is done?

Ndymium 4y ago
Yep, it is! You can find the sources on GitLab[0]. The compression is done using brotli-wasm[1]. The entry point to the compression is the syncCompress[2] function, which converts the data to UTF-8, adds the needed header bytes, and then compresses using brotli-wasm. Decompression is done in a streaming way in streamDecompress[3]. This is to avoid zip bomb attacks, where even a short URL could decompress to gigabytes of data, locking the browser. Thankfully brotli-wasm had streaming decompression builtin, I just had to write the wrapper code to update the text content and the status bar in the UI. You can find the brotli-wasm code I'm using in the vendor folder[4], there is a JS wrapper and then the WASM code (both are straight from brotli-wasm). [0] https://gitlab.com/Nicd/t https://gitlab.com/Nicd/t [1] https://github.com/httptoolkit/brotli-wasm https://github.com/httptoolkit/brotli-wasm [2] https://gitlab.com/Nicd/t/-/blob/05e587b0183ff80b1c6e050b5d30f375330572ec/tahnaroskakori.js#L196 https://gitlab.com/Nicd/t/-/blob/05e587b0183ff80b1c6e050b5d3... [3] https://gitlab.com/Nicd/t/-/blob/05e587b0183ff80b1c6e050b5d30f375330572ec/tahnaroskakori.js#L223 https://gitlab.com/Nicd/t/-/blob/05e587b0183ff80b1c6e050b5d3... [4] https://gitlab.com/Nicd/t/-/tree/05e587b0183ff80b1c6e050b5d30f375330572ec/vendor https://gitlab.com/Nicd/t/-/tree/05e587b0183ff80b1c6e050b5d3...

calebegg 4y ago

> The maximum URL size is 2048 FYI, this is almost entirely untrue. Maybe for some old versions of IE, but modern browsers can handle extremely long URIs

Ndymium 4y ago
Yep, I rechecked it and seems Chrome/Fx/Safari all support at least 32k. I replied in another comment I'm considering a checkbox to allow even longer URLs, though, I don't know who would want to paste them for others to read. :D