5 ms·

I've spent a lot of time thinking about a better format that is close enough to CSV to be practical, but has more precisely defined semantics and structure, als

I've spent a lot of time thinking about a better format that is close enough to CSV to be practical, but has more precisely defined semantics and structure, also to support better usability (decreasing the need for manual integrity checks after parsing). I wanted at least a defined encoding and defined schema (table definitions with fixed number of typed columns). Optionally Unique Keys and Foreign Keys, but that quickly leads to a situation where there are more possible features with diminishing returns to consider. I ended up with this [1] and a python implementation [2], and it turned out not too bad. I've also done a more pragmatic C implementation (couple hundred LOC) in a toy project [3] (wsl.c and *.wsl files), and it turned out quite usable. I think what prevents adoption of such a thing is that it's very hard to standardize on primitive types and integrity features. [1] http://jstimpfle.de/projects/wsl/main.html http://jstimpfle.de/projects/wsl/main.html [2] https://github.com/jstimpfle/python-wsl/ https://github.com/jstimpfle/python-wsl/ [3] https://github.com/jstimpfle/learn-opengl https://github.com/jstimpfle/learn-opengl

jacquesm 9y ago
It's the age old trap of trying to fix a format or a protocol, only to end up with yet another format or protocol.
- jstimpfle 9y ago
  I don't think you can fix CSV. You could define a reasonable-by-today's-standards subset - I'd say, UTF-8, 0x20 delimiters and per-lexem choice of C-style identifiers or string literals. But I need more structure. My format supports an arbitrary number of tables in a single file, some automated integrity checking, and direct conversion of text from/to runtime structures (e.g. C structs, possibly with automatically allocated backing buffers for variable-width datatypes) with only little boilerplate. So yes, that's a different format.
- herrvogel- 9y ago
  Relevant xkcd: https://xkcd.com/927/ https://xkcd.com/927/
baldfat 9y ago
I went the same route but ended up adopting feather from the makers of Python's Pandas and R's tidyverse, Wes McKinney and Hadley Wickham. https://blog.rstudio.com/2016/03/29/feather/ https://blog.rstudio.com/2016/03/29/feather/ > SUMMARY: Feather's good performance is a side effect of its design, but the primary goal of the project is to have a common memory layout (Apache Arrow) and metadata (type information) for use in multiple programming languages. http://wesmckinney.com/blog/feather-its-the-metadata/ http://wesmckinney.com/blog/feather-its-the-metadata/ The approach is data frames on disk storage. > data frames are lists of named, equal-length columns, which can be numeric, boolean, and date-and-time, categorical (_factors), or _string. Every column can have missing values. This fits the needs of all of my CSV usage.
- jstimpfle 9y ago
  Is there also a canonical and pretty (i.e. editor/VCS-friendly) text representation for feather data?
  - baldfat 9y ago
    Good question: I use RStudio but that might be a good project for someone.
rwmj 9y ago
You could come up with the most wonderful format in the world, but unless it's transparently readable and writable by Excel then it will never replace CSV. How does Excel handle your whitespace-separated files?
- jstimpfle 9y ago
  Many of them would actually be handled just fine as CSV files (as fine as any random actual CSV file), but I use a different string syntax ([like this] instead of "like this"), so that would pose some problems. It would not be hard to write an Excel plugin to read my format, while restricting to a fixed set of basic datatypes, and getting some usability gains there. The problem I see is that full-featured interoperability is hard to achieve because the set of supported features is hard to agree on.
tomp 9y ago
This seems to be just .csv with " " as delimiters, "[]" used for quoting and line-comments starting with "%".
- jstimpfle 9y ago
  You can use any syntax for the lexems if you are willing to write the code. "[]" is not really "quoting". It's just the lexical syntax for the string datatype I use, and for consistency it's required even for strings that have no spaces or special characters. And lines starting with "%" hold enforced (even required) schema structure. Yes, I decided to require a single " " as delimiter, and that is not so easy to change with the current approach.

jacquesm 9y ago

It's the age old trap of trying to fix a format or a protocol, only to end up with yet another format or protocol.

jstimpfle 9y ago
I don't think you can fix CSV. You could define a reasonable-by-today's-standards subset - I'd say, UTF-8, 0x20 delimiters and per-lexem choice of C-style identifiers or string literals. But I need more structure. My format supports an arbitrary number of tables in a single file, some automated integrity checking, and direct conversion of text from/to runtime structures (e.g. C structs, possibly with automatically allocated backing buffers for variable-width datatypes) with only little boilerplate. So yes, that's a different format.
herrvogel- 9y ago
Relevant xkcd: https://xkcd.com/927/ https://xkcd.com/927/

baldfat 9y ago

I went the same route but ended up adopting feather from the makers of Python's Pandas and R's tidyverse, Wes McKinney and Hadley Wickham. https://blog.rstudio.com/2016/03/29/feather/ https://blog.rstudio.com/2016/03/29/feather/ > SUMMARY: Feather's good performance is a side effect of its design, but the primary goal of the project is to have a common memory layout (Apache Arrow) and metadata (type information) for use in multiple programming languages. http://wesmckinney.com/blog/feather-its-the-metadata/ http://wesmckinney.com/blog/feather-its-the-metadata/ The approach is data frames on disk storage. > data frames are lists of named, equal-length columns, which can be numeric, boolean, and date-and-time, categorical (_factors), or _string. Every column can have missing values. This fits the needs of all of my CSV usage.

jstimpfle 9y ago
Is there also a canonical and pretty (i.e. editor/VCS-friendly) text representation for feather data?
- baldfat 9y ago
  Good question: I use RStudio but that might be a good project for someone.

rwmj 9y ago

You could come up with the most wonderful format in the world, but unless it's transparently readable and writable by Excel then it will never replace CSV. How does Excel handle your whitespace-separated files?

jstimpfle 9y ago
Many of them would actually be handled just fine as CSV files (as fine as any random actual CSV file), but I use a different string syntax ([like this] instead of "like this"), so that would pose some problems. It would not be hard to write an Excel plugin to read my format, while restricting to a fixed set of basic datatypes, and getting some usability gains there. The problem I see is that full-featured interoperability is hard to achieve because the set of supported features is hard to agree on.

tomp 9y ago

This seems to be just .csv with " " as delimiters, "[]" used for quoting and line-comments starting with "%".

jstimpfle 9y ago
You can use any syntax for the lexems if you are willing to write the code. "[]" is not really "quoting". It's just the lexical syntax for the string datatype I use, and for consistency it's required even for strings that have no spaces or special characters. And lines starting with "%" hold enforced (even required) schema structure. Yes, I decided to require a single " " as delimiter, and that is not so easy to change with the current approach.