3 ms·

I love SQLite and thanks for sharing it but there should be a "(2018)" at the end in the title: > As of this writing (2018-05-29) the only other recommended st

I love SQLite and thanks for sharing it but there should be a "(2018)" at the end in the title: > As of this writing (2018-05-29) the only other recommended storage formats for datasets are XML, JSON, and CSV.

maxloh 5mo ago
FYI, they added a lot more formats to the list after that. Preferred 1. Platform-independent, character-based formats are preferred over native or binary formats as long as data is complete, and retains full detail and precision. Preferred formats include well-developed, widely adopted, de facto marketplace standards, e.g. a. Formats using well known schemas with public validation tool available b. Line-oriented, e.g. TSV, CSV, fixed-width c. Platform-independent open formats, e.g. .db, .db3, .sqlite, .sqlite3 2. Any proprietary format that is a de facto standard for a profession or supported by multiple tools (e.g. Excel .xls or .xlsx, Shapefile) 3. Character Encoding, in descending order of preference: a. UTF-8, UTF-16 (with BOM), b. US-ASCII or ISO 8859-1 c. Other named encoding --- Acceptable For data (in order of preference): 1. Non-proprietary, publicly documented formats endorsed as standards by a professional community or government agency, e.g. CDF, HDF 2. Text-based data formats with available schema For aggregation or transfer: 1. ZIP, RAR, tar, 7z with no encryption, password or other protection mechanisms. https://www.loc.gov/preservation/resources/rfs/data.html https://www.loc.gov/preservation/resources/rfs/data.html
- xxs 5mo ago
  .7z being there just discredits the entire process. The underlying compression algorithm is a free-hand one and can be anything[0], or contain bugs and exploits[1]. Personally I use only zstd with .7z which is 'non-standard' by the official (Russian) release. [0]: https://7-zip.org/7z.html https://7-zip.org/7z.html [1]: CVE-2025-0411
  - tnelsond4 5mo ago
    I love using zstd, it's so fast to decompress. I especially like that the JavaScript decoder is 8kb and still really fast. Though the 25kb wasm decoders are about twice as fast. What are the advantages or reasons to use zstd in a 7z container versus just .zst?
    - xxs 5mo ago
      I love zstd as the next guy and I do use zstd solo for the most part. I had a talk on it few years back too (incl. using the lib directly from Java, massively decreasing log storage, and so on). Why use it w/ 7-zip though. 7-zip archives multiple files/directories and supports encryption. It has the UI too.. On Windows there is NanaZip that's available in the microsoft store which has been graced by corporate for user-install (unlike zstd that effectively needs WSL), and most folks won't be able to use the command line tool. Of course using tar with zstd is always an option if you are on linux.
  - deleted 5mo ago
    [deleted]

maxloh 5mo ago

FYI, they added a lot more formats to the list after that. Preferred 1. Platform-independent, character-based formats are preferred over native or binary formats as long as data is complete, and retains full detail and precision. Preferred formats include well-developed, widely adopted, de facto marketplace standards, e.g. a. Formats using well known schemas with public validation tool available b. Line-oriented, e.g. TSV, CSV, fixed-width c. Platform-independent open formats, e.g. .db, .db3, .sqlite, .sqlite3 2. Any proprietary format that is a de facto standard for a profession or supported by multiple tools (e.g. Excel .xls or .xlsx, Shapefile) 3. Character Encoding, in descending order of preference: a. UTF-8, UTF-16 (with BOM), b. US-ASCII or ISO 8859-1 c. Other named encoding --- Acceptable For data (in order of preference): 1. Non-proprietary, publicly documented formats endorsed as standards by a professional community or government agency, e.g. CDF, HDF 2. Text-based data formats with available schema For aggregation or transfer: 1. ZIP, RAR, tar, 7z with no encryption, password or other protection mechanisms. https://www.loc.gov/preservation/resources/rfs/data.html https://www.loc.gov/preservation/resources/rfs/data.html

xxs 5mo ago
.7z being there just discredits the entire process. The underlying compression algorithm is a free-hand one and can be anything[0], or contain bugs and exploits[1]. Personally I use only zstd with .7z which is 'non-standard' by the official (Russian) release. [0]: https://7-zip.org/7z.html https://7-zip.org/7z.html [1]: CVE-2025-0411
- tnelsond4 5mo ago
  I love using zstd, it's so fast to decompress. I especially like that the JavaScript decoder is 8kb and still really fast. Though the 25kb wasm decoders are about twice as fast. What are the advantages or reasons to use zstd in a 7z container versus just .zst?
  - xxs 5mo ago
    I love zstd as the next guy and I do use zstd solo for the most part. I had a talk on it few years back too (incl. using the lib directly from Java, massively decreasing log storage, and so on). Why use it w/ 7-zip though. 7-zip archives multiple files/directories and supports encryption. It has the UI too.. On Windows there is NanaZip that's available in the microsoft store which has been graced by corporate for user-install (unlike zstd that effectively needs WSL), and most folks won't be able to use the command line tool. Of course using tar with zstd is always an option if you are on linux.
- deleted 5mo ago
  [deleted]