4 ms·

You always want CSV, don't you?

calpaterson 4y ago
Well, one of my bugbears is that most open data is released in weird formats. Take the boston housing dataset as an example: http://lib.stat.cmu.edu/datasets/boston http://lib.stat.cmu.edu/datasets/boston This is one of the most popular beginner datasets around. It's often used in tutorials and when experimenting. Does everyone who wants to use that have to write some custom parser? Why? https://csvbase.com/calpaterson/boston.csv https://csvbase.com/calpaterson/boston.csv is just so much easier
layer8 4y ago
Which CSV though? The format isn't well-defined in practice: https://donatstudios.com/Falsehoods-Programmers-Believe-About-CSVs https://donatstudios.com/Falsehoods-Programmers-Believe-Abou...
- pwdisswordfish9 4y ago
  https://www.w3.org/TR/tabular-data-primer/#dialects https://www.w3.org/TR/tabular-data-primer/#dialects
- swyx 4y ago
  > TSV isn't (not) CSV why? needs elaboration
  - Izkata 4y ago
    As stupid as it may sound, "csv" has become a generic term for any single-character delimited line-based file. We deal regularly with a "csv" that uses the pipe character instead of commas, for example.
    - chasd00 4y ago
      yeah that's pretty strange, does the file format documentation call it a CSV? In my experience those files are always referred to in their documentation as "pipe-delimited file"
      - 6510 4y ago
        I just export an excel sheet as scv in libreoffice. Opening the file in libreoffice with default settings gave me a pile of crap.
tinus_hn 4y ago
Except when you want to be sure you get what they intended. If it’s just some little throwaway project and you don’t care if the results don’t match up, use csv.
- bayesian_horse 4y ago
  That's actually when I'd rather use something like pickle or feather, maybe even json depending on the usecase, which is easier to parse back. I've encountered CSV mostly for data transfer among different programs. There sometimes are better options. Sometimes there isn't, maybe even just because CSV is easier and cheaper to implement on both (or multiple) ends.
  - chasd00 4y ago
    it's mainly because excel can read csv ootb (mostly). Also, many old UNIX programs read csv or fixed-width for batch record processing.
  - tinus_hn 4y ago
    Yeah as long as you don’t mind Excel changing a few values here and there and pretend translation and internationalization doesn’t exist. But hey, that’s ‘good enough’ so the developers don’t need to bother using a format developed in this century.
    - bayesian_horse 4y ago
      Excel is much like an Incel: it has a tendency to mistake something for a date that isn't.
  - RobinL 4y ago
    I would generally recommend parquet for the reasons set out here: https://www.robinlinacre.com/parquet_api/ https://www.robinlinacre.com/parquet_api/
bayesian_horse 4y ago
I don't know if I meant that in jest or for real. CSV is a great format, better than some, until it is not...

Well, one of my bugbears is that most open data is released in weird formats. Take the boston housing dataset as an example: http://lib.stat.cmu.edu/datasets/boston http://lib.stat.cmu.edu/datasets/boston This is one of the most popular beginner datasets around. It's often used in tutorials and when experimenting. Does everyone who wants to use that have to write some custom parser? Why? https://csvbase.com/calpaterson/boston.csv https://csvbase.com/calpaterson/boston.csv is just so much easier

layer8 4y ago

Which CSV though? The format isn't well-defined in practice: https://donatstudios.com/Falsehoods-Programmers-Believe-About-CSVs https://donatstudios.com/Falsehoods-Programmers-Believe-Abou...

pwdisswordfish9 4y ago
https://www.w3.org/TR/tabular-data-primer/#dialects https://www.w3.org/TR/tabular-data-primer/#dialects
swyx 4y ago
> TSV isn't (not) CSV why? needs elaboration
- Izkata 4y ago
  As stupid as it may sound, "csv" has become a generic term for any single-character delimited line-based file. We deal regularly with a "csv" that uses the pipe character instead of commas, for example.
  - chasd00 4y ago
    yeah that's pretty strange, does the file format documentation call it a CSV? In my experience those files are always referred to in their documentation as "pipe-delimited file"
    - 6510 4y ago
      I just export an excel sheet as scv in libreoffice. Opening the file in libreoffice with default settings gave me a pile of crap.

tinus_hn 4y ago

Except when you want to be sure you get what they intended. If it’s just some little throwaway project and you don’t care if the results don’t match up, use csv.

bayesian_horse 4y ago
That's actually when I'd rather use something like pickle or feather, maybe even json depending on the usecase, which is easier to parse back. I've encountered CSV mostly for data transfer among different programs. There sometimes are better options. Sometimes there isn't, maybe even just because CSV is easier and cheaper to implement on both (or multiple) ends.
- chasd00 4y ago
  it's mainly because excel can read csv ootb (mostly). Also, many old UNIX programs read csv or fixed-width for batch record processing.
- tinus_hn 4y ago
  Yeah as long as you don’t mind Excel changing a few values here and there and pretend translation and internationalization doesn’t exist. But hey, that’s ‘good enough’ so the developers don’t need to bother using a format developed in this century.
  - bayesian_horse 4y ago
    Excel is much like an Incel: it has a tendency to mistake something for a date that isn't.
- RobinL 4y ago
  I would generally recommend parquet for the reasons set out here: https://www.robinlinacre.com/parquet_api/ https://www.robinlinacre.com/parquet_api/

bayesian_horse 4y ago

I don't know if I meant that in jest or for real. CSV is a great format, better than some, until it is not...