3 ms·

I think this issue can be bypassed if a better format such as Arrow or Parquet can be used in Excel.

• masklinn 5y ago
  Ideally excel would just speak sqlite. Not import/export or connect via odbc, straight be an sqlite client.
• cryptonym 5y ago
  That doesn't qualify as "good enough to be compiled by humans". We need something dumb and easy you can author from scratch with any text editor, and read/understand with just `cat`.
• pdonis 5y ago
  CSV isn't supported in Excel because Microsoft prefers it. MS prefers its own Excel format. Excel supports CSV because a large enough majority of Excel users want it, which means MS includes support for it. For MS to remove CSV support in favor of Arrow or Parquet support would not benefit MS at all, and would make many of its customers push back, so I don't see it happening.
  • mr_toad 5y ago
    Even if they did support Parquet they’d use the same embrace and extend mechanisms they do with CSV files.
• xtracto 5y ago
  Interesting that the two main "alternatives" mentioned here are columnar data storages. The problem I see with both Arrow and Parquet are that they are binary-based files. Sure, binary files can be VERY efficient. But the "magic" behind files like CSV, JSON or YAML is that they are both "sufficiently" both human and machine readable. Maybe a CSV killer would be a human readable columnar based file format. Nevertheless, the article basically discusses the issues encountered with the manual "editability" of CSV files, not so much with its performance. It also mentions parquet or arrow and concedes that they require specialized format to read/write. If we are looking to that, then there are a lot of options such as sqlite format, BerkleDB (used by some cryptocurrency projects) among plenty of others.