4 ms·

It seems as though one could easily build a file format far more useful than CSV simply by utilizing these separators, and I'm sure it's been done countless tim

by g4zj 3y ago

It seems as though one could easily build a file format far more useful than CSV simply by utilizing these separators, and I'm sure it's been done countless times. Perhaps this would make an interesting personal project. Are you aware of any hurdles, missing key features, etc. that previous attempts at creating such a format have run into (other than adoption, obviously)?

hermitcrab 3y ago
The ASCII unit separator and record separator characters are not well supported by editors. That is why people stick to the (horrible and inconsistent) CSV format.
tambourine_man 3y ago
People don’t like invisible hard to type character. They prefer suffering quoting, escaping, escaping quotes and all that fun stuff
- t-3 3y ago
  Are people actually typing up *SV files by hand? It's trivial to support editing in an IDE and exporting from data-producing applications.
  - andyferris 3y ago
    Yes, sometimes, of course. It's a bit like JSON. Sometimes it's easiest to inject a small piece of hand-written data into a test or whatever. (That said every text editor since ever should have had a "table mode" that uses the ASCII field/record seperators (or whatever you choose), I was always confused why this isn't common. Maybe vim and emacs do?)
EvanAnderson 3y ago
I've done ETL work with systems that used the ASCII separators. It was very pleasant work. Not having to worry about escaping things (because the ASCII separators weren't permitted to be in valid source data to begin with) was very, very nice. I'm a Notepad++ person. When I needed to mock-up data typing the characters was easy-- just ALT and the ASCII code on the numeric pad. It took a bit to memorize the codes I needed to use. Their visual representation is just inverse text and initials.
dheera 3y ago
Unfortunately everyone has moved to "Parquet" (Packet? Parket? Pacquet?) already and we've sailed even further. I absolutely HATE this Parcquage.
- _kst_ 3y ago
  https://www.databricks.com/glossary/what-is-parquet https://www.databricks.com/glossary/what-is-parquet (I don't think everyone has moved to it. I had never heard of it myself.)
  - dheera 3y ago
    A lot of the machine learning world has started using it, it's annoying as hell, solves a problem that doesn't exist, has inadequate documentation, lacks a good GUI viewer, and lacks good command line converters to JSON, XML, CSV, and everything else.
- jll29 3y ago
  No binary format will ever kill CSV: plain-text based formats embody the UNIX philosophy of text files and text processing tool pipes to go with them, and nothing is more durable than keeping your data in text based exchange formats. You won't remember Parquet in 15 years, but you will have CSV files in 50 years.
  - nyokodo 3y ago
    > You won't remember Parquet in 15 years, but you will have CSV files in 50 years. You're probably right about CSV but probably not parquet. Parquet is already 11 years old, there are vast data warehouses that store parquet, it's first class in the spark ecosystem, and a key component of iceberg. Crucially, formats like parquet are "good enough" for a use case that doesn't appear to be going away. There is a high probability in my estimation that enough places are still using them in 15 years to be memorable even if it isn't as common or as visible.
  - dheera 3y ago
    CSV is actually a nice format if it weren't for literal newlines being allowed INSIDE values. That alone makes it much harder to parse correctly with simple code because you can't count on ASCII mode readline()-like functions to fetch 1 record in entirety. Considering it also separates records with newlines, they really should have replaced newlines with "\n" and require escaping "\" with "\\".
thayne 3y ago
I started writing up a spec and library for such a format, then my ADHD drew me to other projects before I finished it. Hopefully I'll get back to it someday. Edit: this is what I have so far: https://github.com/tmccombs/ssv https://github.com/tmccombs/ssv
- sumergomonstro 3y ago
  The "compact" file format (without the tab and newline) should be the SSV standard for data interchange and storage. The pretty-printed format should only be used locally to display/edit with non-compliant editors, then convert back as soon as you're done. In time, editors and file browsers should come to render separators visually in a logical way.
- g4zj 3y ago
  What you've got so far looks promising to me. Pretty much just what I was thinking of doing, in fact, albeit with some details worked out that I hadn't yet considered. Nice job. I hope you come back to finish the project eventually.

by g4zj 3y ago

hermitcrab 3y ago

The ASCII unit separator and record separator characters are not well supported by editors. That is why people stick to the (horrible and inconsistent) CSV format.

tambourine_man 3y ago

People don’t like invisible hard to type character. They prefer suffering quoting, escaping, escaping quotes and all that fun stuff

t-3 3y ago
Are people actually typing up *SV files by hand? It's trivial to support editing in an IDE and exporting from data-producing applications.
- andyferris 3y ago
  Yes, sometimes, of course. It's a bit like JSON. Sometimes it's easiest to inject a small piece of hand-written data into a test or whatever. (That said every text editor since ever should have had a "table mode" that uses the ASCII field/record seperators (or whatever you choose), I was always confused why this isn't common. Maybe vim and emacs do?)

EvanAnderson 3y ago

I've done ETL work with systems that used the ASCII separators. It was very pleasant work. Not having to worry about escaping things (because the ASCII separators weren't permitted to be in valid source data to begin with) was very, very nice. I'm a Notepad++ person. When I needed to mock-up data typing the characters was easy-- just ALT and the ASCII code on the numeric pad. It took a bit to memorize the codes I needed to use. Their visual representation is just inverse text and initials.

dheera 3y ago

Unfortunately everyone has moved to "Parquet" (Packet? Parket? Pacquet?) already and we've sailed even further. I absolutely HATE this Parcquage.

_kst_ 3y ago
https://www.databricks.com/glossary/what-is-parquet https://www.databricks.com/glossary/what-is-parquet (I don't think everyone has moved to it. I had never heard of it myself.)
- dheera 3y ago
  A lot of the machine learning world has started using it, it's annoying as hell, solves a problem that doesn't exist, has inadequate documentation, lacks a good GUI viewer, and lacks good command line converters to JSON, XML, CSV, and everything else.
jll29 3y ago
No binary format will ever kill CSV: plain-text based formats embody the UNIX philosophy of text files and text processing tool pipes to go with them, and nothing is more durable than keeping your data in text based exchange formats. You won't remember Parquet in 15 years, but you will have CSV files in 50 years.
- nyokodo 3y ago
  > You won't remember Parquet in 15 years, but you will have CSV files in 50 years. You're probably right about CSV but probably not parquet. Parquet is already 11 years old, there are vast data warehouses that store parquet, it's first class in the spark ecosystem, and a key component of iceberg. Crucially, formats like parquet are "good enough" for a use case that doesn't appear to be going away. There is a high probability in my estimation that enough places are still using them in 15 years to be memorable even if it isn't as common or as visible.
- dheera 3y ago
  CSV is actually a nice format if it weren't for literal newlines being allowed INSIDE values. That alone makes it much harder to parse correctly with simple code because you can't count on ASCII mode readline()-like functions to fetch 1 record in entirety. Considering it also separates records with newlines, they really should have replaced newlines with "\n" and require escaping "\" with "\\".

thayne 3y ago

I started writing up a spec and library for such a format, then my ADHD drew me to other projects before I finished it. Hopefully I'll get back to it someday. Edit: this is what I have so far: https://github.com/tmccombs/ssv https://github.com/tmccombs/ssv

sumergomonstro 3y ago
The "compact" file format (without the tab and newline) should be the SSV standard for data interchange and storage. The pretty-printed format should only be used locally to display/edit with non-compliant editors, then convert back as soon as you're done. In time, editors and file browsers should come to render separators visually in a logical way.
g4zj 3y ago
What you've got so far looks promising to me. Pretty much just what I was thinking of doing, in fact, albeit with some details worked out that I hadn't yet considered. Nice job. I hope you come back to finish the project eventually.