3 ms·

1. CSV is ubiquitous but has no singularly well-adopted standard. 2. software and data engineers struggle with CSVs as a result of #1. 3. tool is created to red

1. CSV is ubiquitous but has no singularly well-adopted standard. 2. software and data engineers struggle with CSVs as a result of #1. 3. tool is created to reduce pain and friction. 4. profit? a new market? a new standard? The "revolutionary new tool" to replace CSVs was XML in the late 1990s.

nerdponx 4y ago
Except not at all. XML is harder to enter by hand than CSV and not mess it up. CSV optimizes for the easiest cases and performs well on them. XML optimizes for the most complicated cases and therefore performs poorly on the easiest cases. JSON is somewhere in the middle. The main problems with CSV have to do with 1) MS Excel and 2) some kind of delusion among programmers that formatting or parsing arbitrary data is easy and you don't need a library for it, so you get hand-rolled generators and parsers that emit broken files. Otherwise, the problem with CSV has little to do with the CSV format as such and more to do with the fact that the data is stringly-typed. XML has the same problem. JSON interestingly does not. Everything has tradeoffs.
- yosai 4y ago
  @nerdponx You are spot on..
- aforwardslash 4y ago
  nitpick, I wouldn't place JSON in the middle (the lack of proper integers and precision problems is one of the issues). but other than that, spot on.
- refulgentis 4y ago
  That’s really interesting, I wonder if this simplifies down to “you want CSV with column typing and a typesafe CSV editor”, as you note, JSONs win is the lack of issues with typing, and CSV really isn’t complicated at all except for that property. JSON is just a row with keys that are column.
  - nerdponx 4y ago
    I definitely want that! Parquet is great for data interchange, but it's not easily hand-editable. I wonder if there's an open niche in the software world for an Excel-like data entry and manipulation tool, but with stronger/stricter typing of cells and columns, and with direct export to and import from SQLite and Parquet.
- Fnoord 4y ago
  To solve the XML issues you described we got schemas and syntax highlighting. I hate non-prettified JSON but its easy to prettify in any editor. So its a meh argument against JSON. But to solve the crap with the comma one needs a variant of JSON, and there's various of these... One other neat feature of CSV is it can be imported in a very popular and powerful IDE, called... Excel.
yosai 4y ago
@sgerenser Yes CSV is everywhere..YoBulk is smartly positioning itself for the data donor/provider or customer..It is the end customer or data provider who bite the bullet and do the time consuming data cleaning.The customer should know about the errors, duplicates,PII data,inconsitency in data.Customer has to be properly guided to clean the data in best possible manner.

nerdponx 4y ago

Except not at all. XML is harder to enter by hand than CSV and not mess it up. CSV optimizes for the easiest cases and performs well on them. XML optimizes for the most complicated cases and therefore performs poorly on the easiest cases. JSON is somewhere in the middle. The main problems with CSV have to do with 1) MS Excel and 2) some kind of delusion among programmers that formatting or parsing arbitrary data is easy and you don't need a library for it, so you get hand-rolled generators and parsers that emit broken files. Otherwise, the problem with CSV has little to do with the CSV format as such and more to do with the fact that the data is stringly-typed. XML has the same problem. JSON interestingly does not. Everything has tradeoffs.

yosai 4y ago
@nerdponx You are spot on..
aforwardslash 4y ago
nitpick, I wouldn't place JSON in the middle (the lack of proper integers and precision problems is one of the issues). but other than that, spot on.
refulgentis 4y ago
That’s really interesting, I wonder if this simplifies down to “you want CSV with column typing and a typesafe CSV editor”, as you note, JSONs win is the lack of issues with typing, and CSV really isn’t complicated at all except for that property. JSON is just a row with keys that are column.
- nerdponx 4y ago
  I definitely want that! Parquet is great for data interchange, but it's not easily hand-editable. I wonder if there's an open niche in the software world for an Excel-like data entry and manipulation tool, but with stronger/stricter typing of cells and columns, and with direct export to and import from SQLite and Parquet.
Fnoord 4y ago
To solve the XML issues you described we got schemas and syntax highlighting. I hate non-prettified JSON but its easy to prettify in any editor. So its a meh argument against JSON. But to solve the crap with the comma one needs a variant of JSON, and there's various of these... One other neat feature of CSV is it can be imported in a very popular and powerful IDE, called... Excel.

yosai 4y ago

@sgerenser Yes CSV is everywhere..YoBulk is smartly positioning itself for the data donor/provider or customer..It is the end customer or data provider who bite the bullet and do the time consuming data cleaning.The customer should know about the errors, duplicates,PII data,inconsitency in data.Customer has to be properly guided to clean the data in best possible manner.