3 ms·

"You give up human readable files, but what you gain in return is..." Stop right there. You lose more than you gain. Plus, taking the data out of [proprietary

"You give up human readable files, but what you gain in return is..." Stop right there. You lose more than you gain. Plus, taking the data out of [proprietary software app my client's data is in] in csv is usually easy. Taking the data out in Apache Parquet is...usually impossible, but if it is possible at all you'll need to write the code for it. Loading the data into [proprietary software app my client wants data put into] using a csv is usually already a feature it has. If it doesn't, I can manipulate csv to put it into their import format with any language's basic tools. And if it doesn't work, I can look at the csv myself, because it's human readable, to see what the problem is. 90% of real world coding is taking data from a source you don't control, and somehow getting it to a destination you don't control, possibly doing things with it along the way. Your choices are usually csv, xlsx, json, or [shudder] xml. Looking at the pros and cons of those is a reasonable discussion to have.

TimTheTinker 3y ago
I think his arguments apply more closely to SQLite databases. They're not directly human readable, but boy are there a lot of tools for working with them.
- askvictor 3y ago
  We have a use case where we effectively need to have a relational database, but in git. The database doesn't change much, but when it does, references between tables may need to be updated. But we need to easily be able to see diffs between different versions. We're trying an SQLite DB, with exports to CSV as part of CI - the CSV files are human-readable and diff'able. It's also worth noting that SQLite can ingest CSV files into memory and perform queries on them directly - if the files are not too large, it's possible to bypass the sqlite format entirely.
  - zachmu 3y ago
    Somebody already said this, but we built exactly this and it's called Dolt. https://github.com/dolthub/dolt https://github.com/dolthub/dolt Would love to hear how it addresses your use case or falls short.
  - john-shaffer 3y ago
    Have you considered https://github.com/dolthub/dolt https://github.com/dolthub/dolt for your use case?
  - thenbe 3y ago
    > we need to easily be able to see diffs between different versions Can git attributes help in this case? It allows you to teach git how to diff binary files using external tools. Here [0] is a demonstration for teaching git to produce an "image diff" for *.png files using exiftool. You can do something similar for *.sqlite files by adding these 3 lines [1] [2]. The sqlite3 cli needs to be installed. Alternatively, there's a tool that might also fit the bill called datafold/data-diff [3]. I'm pretty sure I originally heard of it on a HN thread so those comments may offer even more alternative solutions. [0]: https://youtu.be/Md44rcw13k4?t=540 https://youtu.be/Md44rcw13k4?t=540 [the relevant timestamp is @ 9:00] [1]: https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8ac1931c2859d70c956df2/modules/git/include/.gitconfig#L47-L48 https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8... [2]: https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8ac1931c2859d70c956df2/modules/git/include/.gitattributes#L35 https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8... [3]: https://github.com/datafold/data-diff https://github.com/datafold/data-diff
saulrh 3y ago
In my experience, human readable file formats are a mistake. As soon as people can read a single file they think that that's the entire format and that it's okay to write it by hand or write their own code for it. And when everyone writes code based on the just what they've personally seen about a format, everyone is sad. This is why not a single piece of software on earth uses the CSV RFC. This is why people hand you CSVs that don't quote string fields with commas in them. This is why you find software that can't handle non-comma delimiters. This is why you find software that assumes that any field made of digits is an integer and then crashes when it tries to do string operations on it. This is why you find software that can't be used unless you change your computer's locale because the front end understands locales and uses commas for numbers but the backend is running on a server and doesn't know what locales are and now everything is broken. This has happened for every single "human-readable" format in existence: html, markdown, CSV, rtf, json, everything. I consider human readability to be a death knell for a format's chances of broad, easy interoperability. There are certainly other things that can doom a format - parquet is almost too complex to implement and so only just barely works, for example - but I'll take a sqlite database over a csv every single time.
- asddubs 3y ago
  I think a takeaway could also be not to give people options when making a human-readable format. "you always need quotes, they're not optional" solves the comma problem. "the delimiter is always a comma" solves the delimiter problem. json has also fared better than csv, I'd say.
  - emj 3y ago
    That makes the delimiter "," which is ugly so someone is just going to use , instead and you are back to square one.
    - asddubs 3y ago
      this is only the case because this was allowed from day 1. If no one ever allowed this to begin with, that just wouldn't work. Of course this is speaking in hypotheticals, but my point is a more general one about specifications of human-readable formats. no one ever attempts to use strings without quotes in json, because then you would be incompatible with everything. There's compatibility issues, but they're far more subtle edge cases
ultrarunner 3y ago
Real world example of this that we just experienced: I work with a provider who offers CSV exports as the only way to access data. Recently, we found they were including unsanitized user input directly in fields. They weren't even quoting these fields. The platform "notified their quality assurance team ASAP" (like every other issue, we never heard back), but we had a deadline. This, of course, was a mess, but being able to quickly open the file and fix quotes was all it took. I shudder at the thought of trying to salvage a corrupted binary with zero help from the holder of the data.
- Too 3y ago
  This sounds like a problem that wouldn’t have existed in the first place if following a binary protocol with a standard format and using a proper serialization library. The issue comes from CSV files looking easy to generate by hand, when it in fact is not.
  - rossdavidh 3y ago
    ...but if it was a binary protocol that did have a problem, of any sort whatsoever, and you couldn't get the provider to address is (in time), then you're hosed if it's not human readable.
  - ultrarunner 3y ago
    This is a decent point, but practically the platform uses some library in their ancient ASP application. The issue is that these types of things can't be fixed because the original author is gone and the tech debt has become unmanageable. This is not the only issue we've had, unfortunately. Debugging this same issue in a binary format is far and away not going to happen in this scenario.

TimTheTinker 3y ago

I think his arguments apply more closely to SQLite databases. They're not directly human readable, but boy are there a lot of tools for working with them.

askvictor 3y ago
We have a use case where we effectively need to have a relational database, but in git. The database doesn't change much, but when it does, references between tables may need to be updated. But we need to easily be able to see diffs between different versions. We're trying an SQLite DB, with exports to CSV as part of CI - the CSV files are human-readable and diff'able. It's also worth noting that SQLite can ingest CSV files into memory and perform queries on them directly - if the files are not too large, it's possible to bypass the sqlite format entirely.
- zachmu 3y ago
  Somebody already said this, but we built exactly this and it's called Dolt. https://github.com/dolthub/dolt https://github.com/dolthub/dolt Would love to hear how it addresses your use case or falls short.
- john-shaffer 3y ago
  Have you considered https://github.com/dolthub/dolt https://github.com/dolthub/dolt for your use case?
- thenbe 3y ago
  > we need to easily be able to see diffs between different versions Can git attributes help in this case? It allows you to teach git how to diff binary files using external tools. Here [0] is a demonstration for teaching git to produce an "image diff" for *.png files using exiftool. You can do something similar for *.sqlite files by adding these 3 lines [1] [2]. The sqlite3 cli needs to be installed. Alternatively, there's a tool that might also fit the bill called datafold/data-diff [3]. I'm pretty sure I originally heard of it on a HN thread so those comments may offer even more alternative solutions. [0]: https://youtu.be/Md44rcw13k4?t=540 https://youtu.be/Md44rcw13k4?t=540 [the relevant timestamp is @ 9:00] [1]: https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8ac1931c2859d70c956df2/modules/git/include/.gitconfig#L47-L48 https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8... [2]: https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8ac1931c2859d70c956df2/modules/git/include/.gitattributes#L35 https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8... [3]: https://github.com/datafold/data-diff https://github.com/datafold/data-diff

saulrh 3y ago

In my experience, human readable file formats are a mistake. As soon as people can read a single file they think that that's the entire format and that it's okay to write it by hand or write their own code for it. And when everyone writes code based on the just what they've personally seen about a format, everyone is sad. This is why not a single piece of software on earth uses the CSV RFC. This is why people hand you CSVs that don't quote string fields with commas in them. This is why you find software that can't handle non-comma delimiters. This is why you find software that assumes that any field made of digits is an integer and then crashes when it tries to do string operations on it. This is why you find software that can't be used unless you change your computer's locale because the front end understands locales and uses commas for numbers but the backend is running on a server and doesn't know what locales are and now everything is broken. This has happened for every single "human-readable" format in existence: html, markdown, CSV, rtf, json, everything. I consider human readability to be a death knell for a format's chances of broad, easy interoperability. There are certainly other things that can doom a format - parquet is almost too complex to implement and so only just barely works, for example - but I'll take a sqlite database over a csv every single time.

asddubs 3y ago
I think a takeaway could also be not to give people options when making a human-readable format. "you always need quotes, they're not optional" solves the comma problem. "the delimiter is always a comma" solves the delimiter problem. json has also fared better than csv, I'd say.
- emj 3y ago
  That makes the delimiter "," which is ugly so someone is just going to use , instead and you are back to square one.
  - asddubs 3y ago
    this is only the case because this was allowed from day 1. If no one ever allowed this to begin with, that just wouldn't work. Of course this is speaking in hypotheticals, but my point is a more general one about specifications of human-readable formats. no one ever attempts to use strings without quotes in json, because then you would be incompatible with everything. There's compatibility issues, but they're far more subtle edge cases

ultrarunner 3y ago

Real world example of this that we just experienced: I work with a provider who offers CSV exports as the only way to access data. Recently, we found they were including unsanitized user input directly in fields. They weren't even quoting these fields. The platform "notified their quality assurance team ASAP" (like every other issue, we never heard back), but we had a deadline. This, of course, was a mess, but being able to quickly open the file and fix quotes was all it took. I shudder at the thought of trying to salvage a corrupted binary with zero help from the holder of the data.

Too 3y ago
This sounds like a problem that wouldn’t have existed in the first place if following a binary protocol with a standard format and using a proper serialization library. The issue comes from CSV files looking easy to generate by hand, when it in fact is not.
- rossdavidh 3y ago
  ...but if it was a binary protocol that did have a problem, of any sort whatsoever, and you couldn't get the provider to address is (in time), then you're hosed if it's not human readable.
- ultrarunner 3y ago
  This is a decent point, but practically the platform uses some library in their ancient ASP application. The issue is that these types of things can't be fixed because the original author is gone and the tech debt has become unmanageable. This is not the only issue we've had, unfortunately. Debugging this same issue in a binary format is far and away not going to happen in this scenario.