3 ms·

There are plenty of usecases for very slow and very good text compression. Imagine you run a big email server for millions of users. Any email that hasn't bee

There are plenty of usecases for very slow and very good text compression. Imagine you run a big email server for millions of users. Any email that hasn't been accessed for a month is probably a good candidate for compressing with this method. In the unlikely event the user wishes to read an old email again, I'm sure they won't mind waiting 0.1 seconds for 10 kilobytes of email (ie. a screenfull) to decompress.

otabdeveloper4 3y ago
Old email needs to be indexed for search, and those indexes might be as large as the original text.
CharlesW 3y ago
> Any email that hasn't been accessed for a month is probably a good candidate for compressing with this method. You don't think "the same exact GPU model and program versions must be used for compression and decompression" rules it out as more than an interesting experiment, especially given that storage is cheap and continues to get cheaper?
- londons_explore 3y ago
  I think they mean the exact same model data (for the GPU). I don't think it requires the exact same model of GPU.
  - CharlesW 3y ago
    I’m also reading it as "the same exact GPU [LLM] and program versions must be used for compression and decompression".
eklitzke 3y ago
As soon as someone uses a POP or IMAP client to fetch all of their mail your server is going to catch on fire. Even if you had POP/IMAP disabled, the computation costs to decode data are enormous with this method, which is precisely why people don't use the most maximally compressing codecs for anything other than true archival applications (e.g. database backups which, unlike old email, really are expected to be accessed almost never).
- londons_explore 3y ago
  Sounds like a good reason to just ship the compressed data to the customer and say "decompress it yourself - this bellard guy makes a decompressor!"
  - etiam 3y ago
    That sounds like recipe for really poor relations with the sizeable fraction of customers who are more concerned with getting their data back than delighted to be introduced to a cutting-edge text compression method...
- londons_explore 3y ago
  A full IMAP sync of a 20 gigabyte inbox from gmail takes ~12 hours. Thats 400 kilobytes per second. And much of that is image attachments and stuff - perhaps only 10 kilobytes/second of text. Maybe Google already does this?

otabdeveloper4 3y ago

Old email needs to be indexed for search, and those indexes might be as large as the original text.

CharlesW 3y ago

> Any email that hasn't been accessed for a month is probably a good candidate for compressing with this method. You don't think "the same exact GPU model and program versions must be used for compression and decompression" rules it out as more than an interesting experiment, especially given that storage is cheap and continues to get cheaper?

londons_explore 3y ago
I think they mean the exact same model data (for the GPU). I don't think it requires the exact same model of GPU.
- CharlesW 3y ago
  I’m also reading it as "the same exact GPU [LLM] and program versions must be used for compression and decompression".

eklitzke 3y ago

As soon as someone uses a POP or IMAP client to fetch all of their mail your server is going to catch on fire. Even if you had POP/IMAP disabled, the computation costs to decode data are enormous with this method, which is precisely why people don't use the most maximally compressing codecs for anything other than true archival applications (e.g. database backups which, unlike old email, really are expected to be accessed almost never).

londons_explore 3y ago
Sounds like a good reason to just ship the compressed data to the customer and say "decompress it yourself - this bellard guy makes a decompressor!"
- etiam 3y ago
  That sounds like recipe for really poor relations with the sizeable fraction of customers who are more concerned with getting their data back than delighted to be introduced to a cutting-edge text compression method...
londons_explore 3y ago
A full IMAP sync of a 20 gigabyte inbox from gmail takes ~12 hours. Thats 400 kilobytes per second. And much of that is image attachments and stuff - perhaps only 10 kilobytes/second of text. Maybe Google already does this?