5 ms·

So this would be another problem that could be avoided if "CSV" would actually friggin mean "comma separated values" and not "whatever Windows might consider co

by mhd 10y ago

So this would be another problem that could be avoided if "CSV" would actually friggin mean "comma separated values" and not "whatever Windows might consider comma-equivalent depending on locale and/or phase of the moon"?

stinos 10y ago
Pretty sure any locale-aware OS treats comma different depending on what locale has been used, no? Anyway commas are historically used as seperators in numbers in some countries, and in text, etc, so if you really want to shed your blame all over something, maybe it shouldn't be Windows here but rather the people who came up with comma as a record seperator in the first place. Probably it was ok for them in that time, at that place (US I'm guessing) and for the usage needed back then, but I'd really wish they chose a different seperator.
- Piskvorrr 10y ago
  Which "works", in the same way that national character sets "work". Until you start exchanging data across locales ("duh, easy: just don't talk to the Americans, solved!").
boulos 10y ago
No. From the detailed description: > A SYLK file is a text file that begins with "ID" or "ID_xxxx", where xxxx is a text string. The first record of a SYLK file is the ID_Number record. When Excel identifies this text at the beginning of a text file, it interprets the file as a SYLK file. Excel tries to convert the file from the SYLK format, but cannot do so because there are no valid SYLK codes after the "ID" characters. Because Excel cannot convert the file, you receive the error message. This is usually called "magic string" or "magic number" (https://en.m.wikipedia.org/wiki/Magic_number_(programming) https://en.m.wikipedia.org/wiki/Magic_number_(programming)). It has nothing to do with the comma, and everything to do with SYLK using a pretty risky magic string (ID) and Excel not having a "try SYLK, if that fails, try as csv". tl;dr: this is about guessing the input format and has nothing to do with the delimiter.
- mhd 10y ago
  It's not just ID, there's also mandatory fields after that, separated by a semicolon. Thus 'ID;P' would be a valid header for SYLK and e.g. for German "CSV"s. If CSV would stick to commas (and boohoo, just escape more fields when this causes locale issues), then even such overly simple heuristics would've sufficed, wouldn't it? Although you'd just run into other pathetic cases with such an informal format (CSV, not SYLK). But hey, I need to interchange formats with some IT guys and don't have a big IT department (ready) and they don't want to parse XLS(X), so I just use CSV. Now you have two problems.
  - philh 10y ago
    "ID;P" is valid at the start of CSV as well. Fields are allowed to contain semicolons.
    - Frank2312 10y ago
      Depending on locale. I believe in the Canadian French locale (and maybe many other locales), ";" is considered a separator (a "," equivalent).
      - philh 10y ago
        Sure, but mhd was saying that if we stuck to CSV being actually comma-separated, "ID;P" would distinguish CSV from SYLK. I'm saying that it wouldn't.
        Someone 10y ago
        mhd doesn't say that: Thus 'ID;P' would be a valid header for SYLK and e.g. for German "CSV"s.
        philh 10y ago
        "If CSV would stick to commas ... then even such overly simple heuristics would've sufficed, wouldn't it?"
taneq 10y ago
What's a comma if not a Unicode character that's comma-equivalent in the current locale? I suspect you're thinking of a comma as ASCII 44. If your source text isn't ASCII then it's just not this simple.
- derefr 10y ago
  I expect what the parent is truly imagining (but hasn't really thought through the usability implications of) is a format where ASCII newline + ASCII comma are used as delimiters, and then everything else is encoded further so as to basically become opaque octet-strings (whose encoding is such that it will never include '\n' or ','). In other words, if you take CSV and make it completely useless for hand-editing in a text-editor by taking it 90% of the way to being a length-prefixed binary encoding, you'll have what the programmer's mind intuitively jumps to as "what CSV should be."
  - supergreg 10y ago
    Why not just use ASCII 31/30 instead of comma/new line. Just give it a new name like ASV (ASCII separated values). What would take to give glyphs to those characters? Just add them to some font?
  - asdkljasdka 10y ago
    No, what OP is talking about is the fact that Excel formats and parses CSV files differently depending on locale settings. For example, here in mainland Europe, Excel thinks CSV files should look like this: ColA;ColB;ColC 1,618;2,718;3,14159 And in fact, to open an "American" CSV file in Excel you either have to set Windows to American locale settings, or use the text import wizard or the "text to columns" feature. If you don't use the correct settings, Excel will not complain, but corrupt the data (sometimes subtly). I hope you agree that it is INSANE to make a file format depend on locale settings. As someone who writes software for scientists in Europe it leads to endless confusion and annoyance. (And even if you do everything correctly, Excel will still mangle certain CSV files, interpreting phone numbers as dates etc...)
    - 10y ago

by mhd 10y ago

stinos 10y ago

Pretty sure any locale-aware OS treats comma different depending on what locale has been used, no? Anyway commas are historically used as seperators in numbers in some countries, and in text, etc, so if you really want to shed your blame all over something, maybe it shouldn't be Windows here but rather the people who came up with comma as a record seperator in the first place. Probably it was ok for them in that time, at that place (US I'm guessing) and for the usage needed back then, but I'd really wish they chose a different seperator.

Piskvorrr 10y ago
Which "works", in the same way that national character sets "work". Until you start exchanging data across locales ("duh, easy: just don't talk to the Americans, solved!").

boulos 10y ago

No. From the detailed description: > A SYLK file is a text file that begins with "ID" or "ID_xxxx", where xxxx is a text string. The first record of a SYLK file is the ID_Number record. When Excel identifies this text at the beginning of a text file, it interprets the file as a SYLK file. Excel tries to convert the file from the SYLK format, but cannot do so because there are no valid SYLK codes after the "ID" characters. Because Excel cannot convert the file, you receive the error message. This is usually called "magic string" or "magic number" (https://en.m.wikipedia.org/wiki/Magic_number_(programming) https://en.m.wikipedia.org/wiki/Magic_number_(programming)). It has nothing to do with the comma, and everything to do with SYLK using a pretty risky magic string (ID) and Excel not having a "try SYLK, if that fails, try as csv". tl;dr: this is about guessing the input format and has nothing to do with the delimiter.

mhd 10y ago
It's not just ID, there's also mandatory fields after that, separated by a semicolon. Thus 'ID;P' would be a valid header for SYLK and e.g. for German "CSV"s. If CSV would stick to commas (and boohoo, just escape more fields when this causes locale issues), then even such overly simple heuristics would've sufficed, wouldn't it? Although you'd just run into other pathetic cases with such an informal format (CSV, not SYLK). But hey, I need to interchange formats with some IT guys and don't have a big IT department (ready) and they don't want to parse XLS(X), so I just use CSV. Now you have two problems.
- philh 10y ago
  "ID;P" is valid at the start of CSV as well. Fields are allowed to contain semicolons.
  - Frank2312 10y ago
    Depending on locale. I believe in the Canadian French locale (and maybe many other locales), ";" is considered a separator (a "," equivalent).
    - philh 10y ago
      Sure, but mhd was saying that if we stuck to CSV being actually comma-separated, "ID;P" would distinguish CSV from SYLK. I'm saying that it wouldn't.
      - Someone 10y ago
        mhd doesn't say that: Thus 'ID;P' would be a valid header for SYLK and e.g. for German "CSV"s.
        philh 10y ago
        "If CSV would stick to commas ... then even such overly simple heuristics would've sufficed, wouldn't it?"

taneq 10y ago

What's a comma if not a Unicode character that's comma-equivalent in the current locale? I suspect you're thinking of a comma as ASCII 44. If your source text isn't ASCII then it's just not this simple.

derefr 10y ago
I expect what the parent is truly imagining (but hasn't really thought through the usability implications of) is a format where ASCII newline + ASCII comma are used as delimiters, and then everything else is encoded further so as to basically become opaque octet-strings (whose encoding is such that it will never include '\n' or ','). In other words, if you take CSV and make it completely useless for hand-editing in a text-editor by taking it 90% of the way to being a length-prefixed binary encoding, you'll have what the programmer's mind intuitively jumps to as "what CSV should be."
- supergreg 10y ago
  Why not just use ASCII 31/30 instead of comma/new line. Just give it a new name like ASV (ASCII separated values). What would take to give glyphs to those characters? Just add them to some font?
- asdkljasdka 10y ago
  No, what OP is talking about is the fact that Excel formats and parses CSV files differently depending on locale settings. For example, here in mainland Europe, Excel thinks CSV files should look like this: ColA;ColB;ColC 1,618;2,718;3,14159 And in fact, to open an "American" CSV file in Excel you either have to set Windows to American locale settings, or use the text import wizard or the "text to columns" feature. If you don't use the correct settings, Excel will not complain, but corrupt the data (sometimes subtly). I hope you agree that it is INSANE to make a file format depend on locale settings. As someone who writes software for scientists in Europe it leads to endless confusion and annoyance. (And even if you do everything correctly, Excel will still mangle certain CSV files, interpreting phone numbers as dates etc...)
  - 10y ago