3 ms·
There are several reasons a compressed archive isn't a good data format outside trusted backups. 1. inconsistent binary formats, float precision edge condition
by Joel_Mckay 3y ago
There are several reasons a compressed archive isn't a good data format outside trusted backups.
1. inconsistent binary formats, float precision edge conditions, and unknown Endianness. Thus, assuming the Marshalling of some document is reliable is risky/fragile, so pick some standard your partners also support... try XML/XSLT, BSON, JSON, SOAP, AMQP+why, or even EDIFACT.
2. "dump and load" is usually inefficient, with an exception when the entire dataset is going to change every time (NOAA weather maps etc.)
3. Anyone wise to the 42TiB bzip 18kiB file joke is acutely aware of what compressed files can do to server scripts.
4. Tuning firewall traffic-shaping for a web-server is different from a server designed to handle large files. Too tolerant rules causes persistent DDoS exposure issues, and too strict causes connections to become unreliable when busy/slow.
5. Anyone that has to deal with CSV files knows how many proprietary interpretations of a simple document format emerge.
Best of luck, and remember to have fun =)
- Bost 3y agoZIP Bomb is 42.zip compressed archive whose size is 42 kilobytes but expands to 4.5 petabytes of uncompressed data. https://blog.fileformat.com/compression/zip-bombs-exploding-your-storage/ https://blog.fileformat.com/compression/zip-bombs-exploding-...
- isoprophlex 3y agoAbout your Point 5, let me tell you about the ESV file format: Eggplant Separated Values! Who needs escaping if you have the eggplant emoji!
- Joel_Mckay 3y agoMy favorite was console control-characters accidentally embedded in plain text by cross-platform document code-page encoding errors. One learns over the years that every program stage/interface must sanitize input, sanity check formats, and isolate data on a per account log. Did the user intend "P=0/O" as a joke... we'll never know. =)