10 ms·

Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the diff

Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relational data, somewhat ubiquitous, types defined but not enforced parquet / hdf5 / apache feather / etc -- Designed for scientific use cases, robust, efficient, less ubiquitous capn proto, prototype buffers, avro, thrift -- Has specific features for data communication between systems xml -- Useful if you are programming in early 2000s GDBM, Kyoto Cabinet, etc -- Useful if you are programming in late 1990s Pick your poison I guess. Engineering is about trade offs.

zzo38computer 4y ago
The latest version of SQLite has a STRICT command to enforce the data types. This option is set per table, but even in a STRICT table you can specify the type of some columns as ANY if you want to allow any type of data in that column (this is not the same meaning of ANY in non-strict tables).
- briHass 4y ago
  You are limited to the basic types (int, floating point, string, and blob), however. I can somewhat get behind the opinionated argument for not needing more specific types like most common language types, but not the lack of a date type.
  - leeoniya 4y ago
    > but not the lack of a date type. i've also found this to be truly bizarre. even more bizarre than not actually respecting (via coercing or error) to the specified type...why even have types, then?
    - zoomablemind 4y ago
      What's so special about having a named type for datetime? User will still need to call functions to manipulate the dates. If only for the default display and import?
      - GekkePrutser 4y ago
        Having a standardised format for data from different sources. If there's no standard people will use their OS specific format. That makes it harder to compare datasets from different sources
      - PeterisP 4y ago
        For dates, having a specific date type instead of a text field is required to have proper behavior when sorting and efficient storage/data transfer. It's also important to have date-related functions on the DB server side, so that you can use them in filtering data before it gets sent over to the user code running on the client, to avoid unnecessary data transfer and allow proper use of indexes in optimizing it. Also, it is nice if a DB engine can perform the equivalent of `WHERE year(date)=2021` without actually running that function on every date, but rather automatically optimize it to an index lookup of `WHERE date between '2021-01-01' and '2021-12-31'`.
        zoomablemind 4y ago
        > ...`WHERE year(date)=2021` without actually running that function on every date, but rather automatically optimize it to an index lookup of `WHERE date between '2021-01-01' and '2021-12-31'` Sure this would be handy. Are there engines that implement such optimization? I can also see how the 'dated' WHERE clause could be used directly in SQLite to leverage the index. Of course, using year() is more expressive. It may also make sense in such a case to simply add a year column and have it indexed.
      - pmontra 4y ago
        I've seen a sqlite database with datetimes in three different formats in the same field, because different parts of the application I inherited had different ideas of how to write a datetime and sqlite accepts everything. It's only a string after all. That's a mistake that the same bad developer couldn't have done with a PostgreSQL or a MySQL.
jayd16 4y ago
None of them seem all that conducive to source control or merging. Any good format for that?
- tantalor 4y ago
  Don't put data in source control; use a database.
  - nvader 4y ago
    Laughs in Lisp Code is Data would like to have a word
    - civilized 4y ago
      Code may be data but data isn't code.
      - Banana699 4y ago
        Data is code that prints/evaluates to itself.
        zzo38computer 4y ago
        In some programming languages (such as PostScript, where evaluating as itself vs being executed, is a flag separate from the value's type, and can be changed by the cvlit and cvx operators), it is.
        civilized 4y ago
        The higher-ups might be unhappy if you use that excuse after committing a gigabyte of customer data.
    - deleted 4y ago
      [deleted]
    - uuyi 4y ago
bezospen15 4y ago
Ther are tons of billion dollar companies that have entire systems utilizing csv and xlsx tubular data for mission critical processes lol
- deleted 4y ago
  [deleted]
clarkevans 4y ago
The DBT (DBase III) format was common in the 80s and 90s. It is a typed, fixed-width format that was directly supported by Excel, Visual Basic grid widgets, among many other tools. For example, Norton Commander supported it directly, letting you preview database tables without loading another program.
- zzo38computer 4y ago
  Do you have the file format documentation?
  - pmontra 4y ago
    http://www.dbase.com/Knowledgebase/INT/db7_file_fmt.htm http://www.dbase.com/Knowledgebase/INT/db7_file_fmt.htm
- vram22 4y ago
  >The DBT (DBase III) format was common in the 80s and 90s. That should actually be DBF, the format for the main database tables. DBT was an ancillary format for the memo fields, which were used to store longer pieces of text in one column of a DBF record. Overall, that generic format and associated software is called XBASE. People still use it in production. And data entry (CRUD) using it with plain text or TUI DOS-style apps blows Web and even GUI apps out of the water in speed of use.
Banana699 4y ago
The author doesn't like any of those tradeoffs and wishes to make another one, what's the problem with that ? You don't believe the design space is exhaustively explored by the designs and protocols you mentioned, do you? there is always another local optimum to be found.
- jonas21 4y ago
  The problem is that the author states that none of these existing formats are "decent" and falls back on shallow dismissals like "Don’t even get me started on Excel’s proprietary, ghastly binary format."
lumost 4y ago
I've often wondered what would happen if there was a standard text editor plugin for dealing with parquet and co. It seems like these formats are disliked, as they are difficult to inspect - but there really isn't any reason UTF-8 bytes arranged in a large sequence (aka CSV) should be any easier to read except for editor support. Sure writes would be slower, but I'd expect most users wouldn't care on modern hardware.
- sanderjd 4y ago
  Yeah I don't really understand the downside to a format like parquet here. "Less ubiquitous" seems to be the only one in the parent comment's list?
jillesvangurp 4y ago
ndjson is actually a really pragmatic choice here that should not be overlooked. Tabular formats break down when the data stops being tabular. This comes up a lot. People love spread sheets as editing tools but they then end up doing things like putting comma separated values in a cell. I've also seen business people use empty cells to indicate hierarchical 'inheritance". An alternate interpretation of that is that that data has some kind of hierarchy and isn't really row based. People just shoehorn all sorts of stuff into spreadsheets because they are there. With ndjson, every line is a json object. Every cell is a named field. If you need multiple values, you can use arrays for the fields. Json has actual types (int, float, strings, boolean). So you can have both hierarchical and multivalued data in a row. The case where all the fields are simple primitives is just the simple case. It has an actual specification too: https://github.com/ndjson/ndjson-spec https://github.com/ndjson/ndjson-spec. I like it because I can stream process it and represent arbitrarily complex objects/documents instead of having to flatten it into columns. The parsing overhead makes it more expensive to use than tsv though. The file size is fine if you use e.g. gzip compression. It compresses really well generally. But I also use tab separated values quite often for simpler data. I mainly like it because google spread sheets provides that as an export option and is actually a great editor for tabular data that I can just give to non technical people. Both file formats can be easily manipulated with command line tools (jq, csvkit, sed, etc.). Both can be processed using mature parsers in a wide range of languages. If you really want, you can edit them with simple text editors, though you probably should be careful with that. Tools like bat know how to format and highlight these files as well. Etc. Tools like that are important because you can use them and script them together rather than reinventing wheels. Formats like parquet are cumbersome mainly because none of the tools I mention support it. No editors. Not a lot of command line tools. No formatting tools. If you want to inspect the data, you pretty much have to write a program to do it. I guess this would be fixable but people seem to be not really interested in doing that work. Parquet becomes nice when you need to process data at scale and in any case use a lot of specialized tooling and infrastructure. Not for everyone in other words. Character encoding is not an issue with either tsv or ndjson if you simply use UTF-8, always. I see no good technical reason why you should use anything else. Anything else should be treated as a bug or legacy. Of course a lot of data has encoding issues regardless. Shit in, shit out basically. Fix it at the source, if you can. The last point is actually key because all of the issues with e.g. csv usually start with people just using really crappy tools to produce source data. Switching to a different file format won't fix these issues since you still deal with the same crappy tools that of course do not support this file format. Anything else you could just fix to not suck to begin with. And if you do, it stops being an issue. The problem is when you can't. Nothing wrong with tsv if you use UTF-8 and a some nice framework that generates properly escaped values and does all the right things. The worst you can say about it is that there are a bit too many choices here and people tend to improvise their own crappy data generation tools with escaping bugs and other issues. Most of the pain is self inflicted. The reason csv/tsv are popular is that you don't need a lot of frameworks / tools. But of course the flipside is that DYI leads to people introducing all sorts of unnecessary issues. Try not to do that.
SkyPuncher 4y ago
Most csv utilities support an alternative delimiter. If I need to edit a file by hand, I'll typically pick an uncommon character for the delimiter (pipe "|" works well since it's uncommon). For me, that pretty much entirely eliminates any of the pain with CSV.
- riffraff 4y ago
  tab-separated-value has never betrayed me! I think it's the default postgresql file export too.
  - atoav 4y ago
    As long as the fields are quoted and escaped : )
- fiddlerwoaroof 4y ago
  More tools should use the ASCII unit and field separator characters intended for this purpose: https://ronaldduncan.wordpress.com/2009/10/31/text-file-formats-ascii-delimited-text-not-csv-or-tab-delimited-text/ https://ronaldduncan.wordpress.com/2009/10/31/text-file-form...
  - mro_name 4y ago
    wow, the solution hides in clear sight right before our eyes. Once again.
  - vim-guru 4y ago
    This is gold
  - cylinder714 4y ago
    See also Control Character Separated Values: https://www.ccsv.io/ https://www.ccsv.io/
PeterisP 4y ago
I've seen ubiquitous use of tab-separated value files instead of csv, as as simpler format without quoting support and a restriction that your data fields can't contain tabs or newlines, which (unlike commas) is okay for many scenarios.
- wodenokoto 4y ago
  But that’s sort of the problem with csv. You never really know which rules your csv files has. Many .csv files are indeed tab separated.
  - lordnacho 4y ago
    Gotta wonder why the format isn't just a column separator char, a row separator char, and then all the data guaranteed not to have those two chars. Then you could save the thing by finding any two chars that aren't used in the data. I guess this is why we have a zillion formats.
    - m_eiman 4y ago
      Then you can't edit or view it in a normal text editor, which is part of the appeal of CSV.
      - ant6n 4y ago
        Maybe normal text editors should support the separators.
      - alpaca128 4y ago
        That and the fact that the comma (or semicolon, as I prefer it due to the rare usage) is on every keyboard. A text editor can be made to handle those separators, but editing or processing the file can be more cumbersome than csv and tsv.
    - atoav 4y ago
      I guess if you use an strange character nobody has on their keyboard, you could also just make it a binary format for efficiency
zzo38computer 4y ago
I suppose that another possible format is PostScript format, which has some of its own advantages and disadvantages. It has both text and binary format, and the binary object sequence format can easily be read by a C code (I wrote a set of macros to do so) without needing to parse the entire structure into memory at first; just read the element that you need, alone, one at a time, in any order. (I have used this format in some of my own software.) JSON also has its own advantages and disadvantages. Of course, both of these formats are more structured than CSV, but XML is also a more structured format. (And, like I and others have mentioned too, the format that they dsecribed isn't actually unique; it just doesn't seem to be that common, but there are enough people who had done the same thing independently, and a few programs which support it, that you could use it if you want to do, and it should work OK.)
bryanrasmussen 4y ago
>xml -- Useful if you are programming in early 2000s I guess nobody knows about document formats anymore.
- zo1 4y ago
  They are re learning those lessons slowly. I.e. OpenAPI and json schema are pretty much poor re implementations of SOAP and XSD but for json. I don't want to be that get off my lawn guy but it's laughable how equivalent they are for 99% of daily use cases.
  - petepete 4y ago
    Every time I hear someone talking about validating JSON I just think about how, despite its flaws, XSD is actually pretty decent despite being 20 years old.
    - _abox 4y ago
      XML and its associated formats were just so complex. I remember considering getting a book on XML and it was 4 inches thick. Just for a text-based data storage format... This is just prohibitively complex. Formats like JSON and YAML thrive because they don't have the complexity of trying to fit every possible scenario ever. The KISS principle still works.
      - bryanrasmussen 4y ago
        Several things though - 1. tech books tend to be too big 2. These XML books tended to have section on XML and well formedness, namespaces, UTF-8, examples of designing a format - generally a book or address format - all this stuff probably came in to approximately 80-115 pages. Which was what you needed to understand the basis of XML. 3. Then would come the secondary stuff to understand, XPath and XSLT. I would say this would be another 100 - 150 pages, so a query language and a programming DSL to manipulating the data/document format. All this together 265 pages. 4. Then validation and entities in DTDs noting that this was old stuff from SGML days and you didn't need it and there was going to be some other way to validate really soon. Another 60 pages? (and then when that validation language came it sucked, as I noted elsewhere) 5. Then because tech books need to be thick and a 300 page book is not big enough a bunch of stuff that never amounted to anything, like Xlink or some breathless stuff about some XML formats, maybe a talk about SVG and VML, XSL-FO blah blah blah. Another 300 pages of unnecessary stuff.
omegalulw 4y ago
One thing of note is that there isn't a single format that's optimal for storing tabular data - what's optimal depends on your use case. And if performance doesn't matter much, just use CSV. As a simple example, column based formats can significantly speed up queries that don't access the full set of columns. They can come in handy for analysis type SQL queries against a big lump of exported data - where different users query the subset they are interested it.
mftb 4y ago
As best I can tell no one has mentioned Recutils[0]? It is a little bizarre that csv has never really been nailed down, but yea it's all about trade-offs. [0]https://www.gnu.org/software/recutils/ https://www.gnu.org/software/recutils/
- mftb 4y ago
  Replying to myself, because after reviewing other comments I realized I didn't read the linked post thoroughly enough. Near the end he proposes a format called "usv", basically using the control codes (unit separator \u001f and record separator \u001e) built into ASCII and now Unicode for their intended purpose, which is actually a really good idea! Apparently this had been noted earlier as a format called "adt"[0]. [0]https://ronaldduncan.wordpress.com/2009/10/31/text-file-formats-ascii-delimited-text-not-csv-or-tab-delimited-text/ https://ronaldduncan.wordpress.com/2009/10/31/text-file-form...
dukoid 4y ago
Surprised that nobody has mentioned ARFF: https://www.cs.waikato.ac.nz/ml/weka/arff.html https://www.cs.waikato.ac.nz/ml/weka/arff.html
hdjjhhvvhga 4y ago
> GDBM, Kyoto Cabinet, etc -- Useful if you are programming in late 1990s Hold your horses: https://charlesleifer.com/blog/completely-un-scientific-benchmarks-of-some-embedded-databases-with-python/ https://charlesleifer.com/blog/completely-un-scientific-benc... While I would never consider GDBM for any new project, I wouldn't dismiss its performance for small/locally accessed data stores.

zzo38computer 4y ago

The latest version of SQLite has a STRICT command to enforce the data types. This option is set per table, but even in a STRICT table you can specify the type of some columns as ANY if you want to allow any type of data in that column (this is not the same meaning of ANY in non-strict tables).

briHass 4y ago
You are limited to the basic types (int, floating point, string, and blob), however. I can somewhat get behind the opinionated argument for not needing more specific types like most common language types, but not the lack of a date type.
- leeoniya 4y ago
  > but not the lack of a date type. i've also found this to be truly bizarre. even more bizarre than not actually respecting (via coercing or error) to the specified type...why even have types, then?
  - zoomablemind 4y ago
    What's so special about having a named type for datetime? User will still need to call functions to manipulate the dates. If only for the default display and import?
    - GekkePrutser 4y ago
      Having a standardised format for data from different sources. If there's no standard people will use their OS specific format. That makes it harder to compare datasets from different sources
    - PeterisP 4y ago
      For dates, having a specific date type instead of a text field is required to have proper behavior when sorting and efficient storage/data transfer. It's also important to have date-related functions on the DB server side, so that you can use them in filtering data before it gets sent over to the user code running on the client, to avoid unnecessary data transfer and allow proper use of indexes in optimizing it. Also, it is nice if a DB engine can perform the equivalent of `WHERE year(date)=2021` without actually running that function on every date, but rather automatically optimize it to an index lookup of `WHERE date between '2021-01-01' and '2021-12-31'`.
      - zoomablemind 4y ago
        > ...`WHERE year(date)=2021` without actually running that function on every date, but rather automatically optimize it to an index lookup of `WHERE date between '2021-01-01' and '2021-12-31'` Sure this would be handy. Are there engines that implement such optimization? I can also see how the 'dated' WHERE clause could be used directly in SQLite to leverage the index. Of course, using year() is more expressive. It may also make sense in such a case to simply add a year column and have it indexed.
    - pmontra 4y ago
      I've seen a sqlite database with datetimes in three different formats in the same field, because different parts of the application I inherited had different ideas of how to write a datetime and sqlite accepts everything. It's only a string after all. That's a mistake that the same bad developer couldn't have done with a PostgreSQL or a MySQL.

jayd16 4y ago

None of them seem all that conducive to source control or merging. Any good format for that?

tantalor 4y ago
Don't put data in source control; use a database.
- nvader 4y ago
  Laughs in Lisp Code is Data would like to have a word
  - civilized 4y ago
    Code may be data but data isn't code.
    - Banana699 4y ago
      Data is code that prints/evaluates to itself.
      - zzo38computer 4y ago
        In some programming languages (such as PostScript, where evaluating as itself vs being executed, is a flag separate from the value's type, and can be changed by the cvlit and cvx operators), it is.
      - civilized 4y ago
        The higher-ups might be unhappy if you use that excuse after committing a gigabyte of customer data.
  - deleted 4y ago
    [deleted]
  - uuyi 4y ago

bezospen15 4y ago

Ther are tons of billion dollar companies that have entire systems utilizing csv and xlsx tubular data for mission critical processes lol

deleted 4y ago
[deleted]

clarkevans 4y ago

The DBT (DBase III) format was common in the 80s and 90s. It is a typed, fixed-width format that was directly supported by Excel, Visual Basic grid widgets, among many other tools. For example, Norton Commander supported it directly, letting you preview database tables without loading another program.

zzo38computer 4y ago
Do you have the file format documentation?
- pmontra 4y ago
  http://www.dbase.com/Knowledgebase/INT/db7_file_fmt.htm http://www.dbase.com/Knowledgebase/INT/db7_file_fmt.htm
vram22 4y ago
>The DBT (DBase III) format was common in the 80s and 90s. That should actually be DBF, the format for the main database tables. DBT was an ancillary format for the memo fields, which were used to store longer pieces of text in one column of a DBF record. Overall, that generic format and associated software is called XBASE. People still use it in production. And data entry (CRUD) using it with plain text or TUI DOS-style apps blows Web and even GUI apps out of the water in speed of use.

Banana699 4y ago

The author doesn't like any of those tradeoffs and wishes to make another one, what's the problem with that ? You don't believe the design space is exhaustively explored by the designs and protocols you mentioned, do you? there is always another local optimum to be found.

jonas21 4y ago
The problem is that the author states that none of these existing formats are "decent" and falls back on shallow dismissals like "Don’t even get me started on Excel’s proprietary, ghastly binary format."

lumost 4y ago

I've often wondered what would happen if there was a standard text editor plugin for dealing with parquet and co. It seems like these formats are disliked, as they are difficult to inspect - but there really isn't any reason UTF-8 bytes arranged in a large sequence (aka CSV) should be any easier to read except for editor support. Sure writes would be slower, but I'd expect most users wouldn't care on modern hardware.

sanderjd 4y ago
Yeah I don't really understand the downside to a format like parquet here. "Less ubiquitous" seems to be the only one in the parent comment's list?

jillesvangurp 4y ago

ndjson is actually a really pragmatic choice here that should not be overlooked. Tabular formats break down when the data stops being tabular. This comes up a lot. People love spread sheets as editing tools but they then end up doing things like putting comma separated values in a cell. I've also seen business people use empty cells to indicate hierarchical 'inheritance". An alternate interpretation of that is that that data has some kind of hierarchy and isn't really row based. People just shoehorn all sorts of stuff into spreadsheets because they are there. With ndjson, every line is a json object. Every cell is a named field. If you need multiple values, you can use arrays for the fields. Json has actual types (int, float, strings, boolean). So you can have both hierarchical and multivalued data in a row. The case where all the fields are simple primitives is just the simple case. It has an actual specification too: https://github.com/ndjson/ndjson-spec https://github.com/ndjson/ndjson-spec. I like it because I can stream process it and represent arbitrarily complex objects/documents instead of having to flatten it into columns. The parsing overhead makes it more expensive to use than tsv though. The file size is fine if you use e.g. gzip compression. It compresses really well generally. But I also use tab separated values quite often for simpler data. I mainly like it because google spread sheets provides that as an export option and is actually a great editor for tabular data that I can just give to non technical people. Both file formats can be easily manipulated with command line tools (jq, csvkit, sed, etc.). Both can be processed using mature parsers in a wide range of languages. If you really want, you can edit them with simple text editors, though you probably should be careful with that. Tools like bat know how to format and highlight these files as well. Etc. Tools like that are important because you can use them and script them together rather than reinventing wheels. Formats like parquet are cumbersome mainly because none of the tools I mention support it. No editors. Not a lot of command line tools. No formatting tools. If you want to inspect the data, you pretty much have to write a program to do it. I guess this would be fixable but people seem to be not really interested in doing that work. Parquet becomes nice when you need to process data at scale and in any case use a lot of specialized tooling and infrastructure. Not for everyone in other words. Character encoding is not an issue with either tsv or ndjson if you simply use UTF-8, always. I see no good technical reason why you should use anything else. Anything else should be treated as a bug or legacy. Of course a lot of data has encoding issues regardless. Shit in, shit out basically. Fix it at the source, if you can. The last point is actually key because all of the issues with e.g. csv usually start with people just using really crappy tools to produce source data. Switching to a different file format won't fix these issues since you still deal with the same crappy tools that of course do not support this file format. Anything else you could just fix to not suck to begin with. And if you do, it stops being an issue. The problem is when you can't. Nothing wrong with tsv if you use UTF-8 and a some nice framework that generates properly escaped values and does all the right things. The worst you can say about it is that there are a bit too many choices here and people tend to improvise their own crappy data generation tools with escaping bugs and other issues. Most of the pain is self inflicted. The reason csv/tsv are popular is that you don't need a lot of frameworks / tools. But of course the flipside is that DYI leads to people introducing all sorts of unnecessary issues. Try not to do that.

SkyPuncher 4y ago

Most csv utilities support an alternative delimiter. If I need to edit a file by hand, I'll typically pick an uncommon character for the delimiter (pipe "|" works well since it's uncommon). For me, that pretty much entirely eliminates any of the pain with CSV.

riffraff 4y ago
tab-separated-value has never betrayed me! I think it's the default postgresql file export too.
- atoav 4y ago
  As long as the fields are quoted and escaped : )
fiddlerwoaroof 4y ago
More tools should use the ASCII unit and field separator characters intended for this purpose: https://ronaldduncan.wordpress.com/2009/10/31/text-file-formats-ascii-delimited-text-not-csv-or-tab-delimited-text/ https://ronaldduncan.wordpress.com/2009/10/31/text-file-form...
- mro_name 4y ago
  wow, the solution hides in clear sight right before our eyes. Once again.
- vim-guru 4y ago
  This is gold
- cylinder714 4y ago
  See also Control Character Separated Values: https://www.ccsv.io/ https://www.ccsv.io/

PeterisP 4y ago

I've seen ubiquitous use of tab-separated value files instead of csv, as as simpler format without quoting support and a restriction that your data fields can't contain tabs or newlines, which (unlike commas) is okay for many scenarios.

wodenokoto 4y ago
But that’s sort of the problem with csv. You never really know which rules your csv files has. Many .csv files are indeed tab separated.
- lordnacho 4y ago
  Gotta wonder why the format isn't just a column separator char, a row separator char, and then all the data guaranteed not to have those two chars. Then you could save the thing by finding any two chars that aren't used in the data. I guess this is why we have a zillion formats.
  - m_eiman 4y ago
    Then you can't edit or view it in a normal text editor, which is part of the appeal of CSV.
    - ant6n 4y ago
      Maybe normal text editors should support the separators.
    - alpaca128 4y ago
      That and the fact that the comma (or semicolon, as I prefer it due to the rare usage) is on every keyboard. A text editor can be made to handle those separators, but editing or processing the file can be more cumbersome than csv and tsv.
  - atoav 4y ago
    I guess if you use an strange character nobody has on their keyboard, you could also just make it a binary format for efficiency

zzo38computer 4y ago

I suppose that another possible format is PostScript format, which has some of its own advantages and disadvantages. It has both text and binary format, and the binary object sequence format can easily be read by a C code (I wrote a set of macros to do so) without needing to parse the entire structure into memory at first; just read the element that you need, alone, one at a time, in any order. (I have used this format in some of my own software.) JSON also has its own advantages and disadvantages. Of course, both of these formats are more structured than CSV, but XML is also a more structured format. (And, like I and others have mentioned too, the format that they dsecribed isn't actually unique; it just doesn't seem to be that common, but there are enough people who had done the same thing independently, and a few programs which support it, that you could use it if you want to do, and it should work OK.)

bryanrasmussen 4y ago

>xml -- Useful if you are programming in early 2000s I guess nobody knows about document formats anymore.

zo1 4y ago
They are re learning those lessons slowly. I.e. OpenAPI and json schema are pretty much poor re implementations of SOAP and XSD but for json. I don't want to be that get off my lawn guy but it's laughable how equivalent they are for 99% of daily use cases.
- petepete 4y ago
  Every time I hear someone talking about validating JSON I just think about how, despite its flaws, XSD is actually pretty decent despite being 20 years old.
  - _abox 4y ago
    XML and its associated formats were just so complex. I remember considering getting a book on XML and it was 4 inches thick. Just for a text-based data storage format... This is just prohibitively complex. Formats like JSON and YAML thrive because they don't have the complexity of trying to fit every possible scenario ever. The KISS principle still works.
    - bryanrasmussen 4y ago
      Several things though - 1. tech books tend to be too big 2. These XML books tended to have section on XML and well formedness, namespaces, UTF-8, examples of designing a format - generally a book or address format - all this stuff probably came in to approximately 80-115 pages. Which was what you needed to understand the basis of XML. 3. Then would come the secondary stuff to understand, XPath and XSLT. I would say this would be another 100 - 150 pages, so a query language and a programming DSL to manipulating the data/document format. All this together 265 pages. 4. Then validation and entities in DTDs noting that this was old stuff from SGML days and you didn't need it and there was going to be some other way to validate really soon. Another 60 pages? (and then when that validation language came it sucked, as I noted elsewhere) 5. Then because tech books need to be thick and a 300 page book is not big enough a bunch of stuff that never amounted to anything, like Xlink or some breathless stuff about some XML formats, maybe a talk about SVG and VML, XSL-FO blah blah blah. Another 300 pages of unnecessary stuff.

omegalulw 4y ago

One thing of note is that there isn't a single format that's optimal for storing tabular data - what's optimal depends on your use case. And if performance doesn't matter much, just use CSV. As a simple example, column based formats can significantly speed up queries that don't access the full set of columns. They can come in handy for analysis type SQL queries against a big lump of exported data - where different users query the subset they are interested it.

mftb 4y ago

As best I can tell no one has mentioned Recutils[0]? It is a little bizarre that csv has never really been nailed down, but yea it's all about trade-offs. [0]https://www.gnu.org/software/recutils/ https://www.gnu.org/software/recutils/

mftb 4y ago
Replying to myself, because after reviewing other comments I realized I didn't read the linked post thoroughly enough. Near the end he proposes a format called "usv", basically using the control codes (unit separator \u001f and record separator \u001e) built into ASCII and now Unicode for their intended purpose, which is actually a really good idea! Apparently this had been noted earlier as a format called "adt"[0]. [0]https://ronaldduncan.wordpress.com/2009/10/31/text-file-formats-ascii-delimited-text-not-csv-or-tab-delimited-text/ https://ronaldduncan.wordpress.com/2009/10/31/text-file-form...

dukoid 4y ago

Surprised that nobody has mentioned ARFF: https://www.cs.waikato.ac.nz/ml/weka/arff.html https://www.cs.waikato.ac.nz/ml/weka/arff.html

hdjjhhvvhga 4y ago

> GDBM, Kyoto Cabinet, etc -- Useful if you are programming in late 1990s Hold your horses: https://charlesleifer.com/blog/completely-un-scientific-benchmarks-of-some-embedded-databases-with-python/ https://charlesleifer.com/blog/completely-un-scientific-benc... While I would never consider GDBM for any new project, I wouldn't dismiss its performance for small/locally accessed data stores.