4 ms·

> Columns are separated by \u001F (ASCII unit separator) > Rows are separated by \u001E (ASCII record separator) Or, how about columns separated by \u002C, and

by dossy 4y ago

> Columns are separated by \u001F (ASCII unit separator) > Rows are separated by \u001E (ASCII record separator) Or, how about columns separated by \u002C, and rows separated by \u000A. And, for bonus points, we can even define unambiguous ways of escaping those two characters so that they CAN appear within column values, if we wanted to, and not tell people that our encoding format is totally stupid and that they need to use a different format. OP's proposal is equally "highly sub-optimal for the job" for exactly the same imaginary reasons they dislike the currently available encoding formats, but they don't seem to realize it.

LeonB 4y ago
> OP's proposal is equally "highly sub-optimal for the job" for exactly the same imaginary reasons they dislike the currently available encoding formats, but they don't seem to realize it. This is a really unfair appraisal in a bunch of different ways. Removing the ability to embed record delimiters, for example, means you can process the records in parallel. That’s a massive improvement all by itself. Stating that their reasons are “imaginary” is just a needless insult, apart from being wrong. Why be like that?
- tbrownaw 4y ago
  > Removing the ability to embed record delimiters, for example, means you can process the records in parallel. That’s a massive improvement all by itself. I can process records from a csv in parallel just fine. Lexing isn't slow. .csv stream -> tokens -> records (-> aggregate into ~100ms blocks) -> work queue with N workers
  - UncleEntity 4y ago
    You lex the whole file, separate it into records and then process them? Doesn’t sound like you’re processing the file in parallel to me. TFA’s way would allow you to start at any arbitrary file position, just advance until you find a record separator then start parsing from there without having to worry about starting in the middle of a quoted string.
    - tbrownaw 4y ago
      Not like sequential steps, a whole bunch of foreach / yield return loops until the last one dumps things onto a thread-safe queue.
- dossy 4y ago
  > Stating that their reasons are “imaginary” is just a needless insult, apart from being wrong. Why be like that? It's not an insult, let alone a needless one: it is a statement of fact. The reasons OP cites for criticizing CSV are creations of OP's imagination, their subjective beliefs, and not based in objective truth. For example: > "CSV is a mess." Actually, CSV is quite orderly and predictable. OP only imagines it's a mess because of some skewed experience they've personally had. > "One quote in the wrong place and the file is invalid." Any file format that is encoded improperly renders it invalid. Imagining that this is strictly limited to CSV, is just that: a creation of OP's imagination. > "It is difficult to parse efficiently using multiple cores, due to the quoting (you can’t start parsing from part way through a file)." This is absolutely untrue. Anyone who's written a high-volume streaming CSV parser that is multi-process and multi-threaded should be giving OP the side-eye here. Yes, you can absolutely start parsing from any arbitrary point in the file, and yes, in the degenerate worst-case scenario, you may need to rewind back to the beginning of the file if the data exists in such a way that your arbitrary starting point requires it, but that is true of ANY file format: it is possible to craft a worst-case scenario and specific starting position within that data that would require this. CSV is no exception. Hope this helps you better understand what I had written.

by dossy 4y ago

LeonB 4y ago

> OP's proposal is equally "highly sub-optimal for the job" for exactly the same imaginary reasons they dislike the currently available encoding formats, but they don't seem to realize it. This is a really unfair appraisal in a bunch of different ways. Removing the ability to embed record delimiters, for example, means you can process the records in parallel. That’s a massive improvement all by itself. Stating that their reasons are “imaginary” is just a needless insult, apart from being wrong. Why be like that?

tbrownaw 4y ago
> Removing the ability to embed record delimiters, for example, means you can process the records in parallel. That’s a massive improvement all by itself. I can process records from a csv in parallel just fine. Lexing isn't slow. .csv stream -> tokens -> records (-> aggregate into ~100ms blocks) -> work queue with N workers
- UncleEntity 4y ago
  You lex the whole file, separate it into records and then process them? Doesn’t sound like you’re processing the file in parallel to me. TFA’s way would allow you to start at any arbitrary file position, just advance until you find a record separator then start parsing from there without having to worry about starting in the middle of a quoted string.
  - tbrownaw 4y ago
    Not like sequential steps, a whole bunch of foreach / yield return loops until the last one dumps things onto a thread-safe queue.
dossy 4y ago
> Stating that their reasons are “imaginary” is just a needless insult, apart from being wrong. Why be like that? It's not an insult, let alone a needless one: it is a statement of fact. The reasons OP cites for criticizing CSV are creations of OP's imagination, their subjective beliefs, and not based in objective truth. For example: > "CSV is a mess." Actually, CSV is quite orderly and predictable. OP only imagines it's a mess because of some skewed experience they've personally had. > "One quote in the wrong place and the file is invalid." Any file format that is encoded improperly renders it invalid. Imagining that this is strictly limited to CSV, is just that: a creation of OP's imagination. > "It is difficult to parse efficiently using multiple cores, due to the quoting (you can’t start parsing from part way through a file)." This is absolutely untrue. Anyone who's written a high-volume streaming CSV parser that is multi-process and multi-threaded should be giving OP the side-eye here. Yes, you can absolutely start parsing from any arbitrary point in the file, and yes, in the degenerate worst-case scenario, you may need to rewind back to the beginning of the file if the data exists in such a way that your arbitrary starting point requires it, but that is true of ANY file format: it is possible to craft a worst-case scenario and specific starting position within that data that would require this. CSV is no exception. Hope this helps you better understand what I had written.