4 ms·
AFAIK, files uploaded to Usenet still need to be ASCII encoded using something like uuencode or base64, which increases file sizes to something like a third lar
by Cyberdog 2y ago
AFAIK, files uploaded to Usenet still need to be ASCII encoded using something like uuencode or base64, which increases file sizes to something like a third larger than their original sizes. So you can decrease the amount of data you need to store quite significantly by only storing the unencoded versions.
After that, deduplication can probably bring you down at least another 50% - how many of these files are just the same things being posted over and over? Probably quite many of them. Store a file once with a database tracking its SHA1 hash, and whenever you see a file with the same hash come in, throw it away and instead store a reference to the first file.
- xhrpost 2y agoMost posts are yEnc encoded these days https://en.m.wikipedia.org/wiki/YEnc https://en.m.wikipedia.org/wiki/YEnc