7 ms·

Nope, Straightforward Way is like this: name:String, date:Date, value:Int "Miami", 2021-08-19 11:54:19.721376-05, 2 And making header mandatory. I th

by mamcx 5y ago

Nope, Straightforward Way is like this: name:String, date:Date, value:Int "Miami", 2021-08-19 11:54:19.721376-05, 2 And making header mandatory. I think forcing quoting of strings and forcing "," for separation and "\n" for lines. Dates are ISO, decimals use . That is all. P.D: This is similar how I done this for my mini-lang https://tablam.org/syntax https://tablam.org/syntax. Tabular data is very simple and the only problem with csv is that is too flexible and let everyone to change it, but making the "schema" outside from csv is against it purpose.

tmountain 5y ago
Simple, elegant, and backward compatible; albeit with degradation, with the existing "standard". I like this approach.
- teknopaul 5y ago
  IMHO the point about .csv is that writing a parser is trivial so none of this matters. Many writers bork on certain types of data. No biggie. Yes there are variants, no they are not complicated. Trying to support _all_ variants of something that is not well-defined is a mugs game. Most of the differences are by mistake. Some of the differences can't be fixed in the parser. If you need csv to work, pick any one interpretation that does work.
riedel 5y ago
This addresses part of the problem. Don't forget the character encoding, null encodings, separator, quote and escape chars,... No doubt, you can use csv in a well defined way if there is agreement and specs. The problem is there simply isn't any defacto standard and it won't be able to establish one in this domain... Btw. : The python intake package allows you to specify metadata.
- mamcx 5y ago
  But is like with json: We are not debating about that details because is a bit more well defined. For a improved CSV, that minutae must be defined from the start, but also must take in account how is CSV used, and keep it spirit. P.D: A example on something like this is https://commonmark.org https://commonmark.org, that codify the rules for markdown in a better defined speck.
- munk-a 5y ago
  Null encoding is entirely uncovered by the CSV format - if you want to communicate nulls you need to have an internal convention - some sort of reserved string. Quoting characters are pretty commonly agreed to be " (aka &quot) and escape characters are technically unnecessary. Deciding to support \\ and \n is a pretty nice thing to do since some producers and parsers will erroneously choke on mid-string newlines but the only character that actually needs to be escaped (the double quote) can just be doubled up as per RFC4180[1]. The indecision on escaping means that the literal character sequence "\n" is very hard to represent in CSV consistently this is an unfortunate malus of C programmers getting all up in the standard. RFC4180 is an after-the-fact RFC to try and codify common practices in the industry so some tools may refuse to conform to it - but it's really clearly written and like... super short. If you've never read it just go ahead and do so since it lays things out with exceeding care. 1. https://datatracker.ietf.org/doc/html/rfc4180#section-2 https://datatracker.ietf.org/doc/html/rfc4180#section-2
  - deleted 5y ago
    [deleted]
  - alecst 5y ago
    I work with CSVs often and we usually just use blanks like item1,item2,,item4,,,item7 to represent nulls. What do you think the pitfalls are here? (Sincere question, wondering how we can improve.)
    - mrobins 5y ago
      This breaks in systems where an empty string “” != null but I don’t have an elegant solution.
      - saltcured 5y ago
        A typical hack is to distinguish an empty field from a field with a quoted empty string. The former is null, the latter an empty string.
zzleeper 5y ago
Why do you prefer comma for delimiters? I found tabs are less likely to appear within the strings (so less need to quote the strings if a delimiter is found, and thus the files are smaller). And both can be typed with a single keystroke. The only advantage I would see for commas is if you are editing the file in a text editor that doesn't highlight strings (like notepad) so it's hard to distinguish them from spaces
- FroshKiller 5y ago
  If the delimiter isn't a comma, it isn't a CSV. I prefer tabs as well, but that's why I ask for a "tab-delimited text file" instead of a CSV.
  - sobriquet9 5y ago
    In locales where comma is used instead of decimal point, CSV fields are separated with semicolons.
  - fomine3 5y ago
    So it called "Character Separated Values".
- munk-a 5y ago
  The advantage you mentioned is the one I've seen that has the most benefit. If users are ever modifying the file raw they're going to bork the tab characters - nevermind some developer opening the file who has his editor configured to aggressively convert tabs to spaces. Tabs are trivial to actually type (so they're much off than, say, FormFeed) but they're difficult to visually distinguish. I also think it's generally a good habit to just force quoting of all fields in CSV to dodge any impending user errors. Human readable file formats have surrendered space efficiency for legibility and the main attribute they need to focus on, IMO, is defense against human errors.

by mamcx 5y ago

tmountain 5y ago

Simple, elegant, and backward compatible; albeit with degradation, with the existing "standard". I like this approach.

teknopaul 5y ago
IMHO the point about .csv is that writing a parser is trivial so none of this matters. Many writers bork on certain types of data. No biggie. Yes there are variants, no they are not complicated. Trying to support _all_ variants of something that is not well-defined is a mugs game. Most of the differences are by mistake. Some of the differences can't be fixed in the parser. If you need csv to work, pick any one interpretation that does work.

riedel 5y ago

This addresses part of the problem. Don't forget the character encoding, null encodings, separator, quote and escape chars,... No doubt, you can use csv in a well defined way if there is agreement and specs. The problem is there simply isn't any defacto standard and it won't be able to establish one in this domain... Btw. : The python intake package allows you to specify metadata.

mamcx 5y ago
But is like with json: We are not debating about that details because is a bit more well defined. For a improved CSV, that minutae must be defined from the start, but also must take in account how is CSV used, and keep it spirit. P.D: A example on something like this is https://commonmark.org https://commonmark.org, that codify the rules for markdown in a better defined speck.
munk-a 5y ago
Null encoding is entirely uncovered by the CSV format - if you want to communicate nulls you need to have an internal convention - some sort of reserved string. Quoting characters are pretty commonly agreed to be " (aka &quot) and escape characters are technically unnecessary. Deciding to support \\ and \n is a pretty nice thing to do since some producers and parsers will erroneously choke on mid-string newlines but the only character that actually needs to be escaped (the double quote) can just be doubled up as per RFC4180[1]. The indecision on escaping means that the literal character sequence "\n" is very hard to represent in CSV consistently this is an unfortunate malus of C programmers getting all up in the standard. RFC4180 is an after-the-fact RFC to try and codify common practices in the industry so some tools may refuse to conform to it - but it's really clearly written and like... super short. If you've never read it just go ahead and do so since it lays things out with exceeding care. 1. https://datatracker.ietf.org/doc/html/rfc4180#section-2 https://datatracker.ietf.org/doc/html/rfc4180#section-2
- deleted 5y ago
  [deleted]
- alecst 5y ago
  I work with CSVs often and we usually just use blanks like item1,item2,,item4,,,item7 to represent nulls. What do you think the pitfalls are here? (Sincere question, wondering how we can improve.)
  - mrobins 5y ago
    This breaks in systems where an empty string “” != null but I don’t have an elegant solution.
    - saltcured 5y ago
      A typical hack is to distinguish an empty field from a field with a quoted empty string. The former is null, the latter an empty string.

zzleeper 5y ago

Why do you prefer comma for delimiters? I found tabs are less likely to appear within the strings (so less need to quote the strings if a delimiter is found, and thus the files are smaller). And both can be typed with a single keystroke. The only advantage I would see for commas is if you are editing the file in a text editor that doesn't highlight strings (like notepad) so it's hard to distinguish them from spaces

FroshKiller 5y ago
If the delimiter isn't a comma, it isn't a CSV. I prefer tabs as well, but that's why I ask for a "tab-delimited text file" instead of a CSV.
- sobriquet9 5y ago
  In locales where comma is used instead of decimal point, CSV fields are separated with semicolons.
- fomine3 5y ago
  So it called "Character Separated Values".
munk-a 5y ago
The advantage you mentioned is the one I've seen that has the most benefit. If users are ever modifying the file raw they're going to bork the tab characters - nevermind some developer opening the file who has his editor configured to aggressively convert tabs to spaces. Tabs are trivial to actually type (so they're much off than, say, FormFeed) but they're difficult to visually distinguish. I also think it's generally a good habit to just force quoting of all fields in CSV to dodge any impending user errors. Human readable file formats have surrendered space efficiency for legibility and the main attribute they need to focus on, IMO, is defense against human errors.