5 ms·

Nice, until the CSV part really. I mean you have a nice DataFrame there but then it gets serialized using probably the worst format out there for totally variab

by stinos 6y ago

Nice, until the CSV part really. I mean you have a nice DataFrame there but then it gets serialized using probably the worst format out there for totally variable data like what you scrape from a site. Unless the file uses the actual ASCII record separator you'll end up with a CSV file which can only be read by a handful of software, after telling it explicitly what the separator and quoting rules are. And even then it's hit or miss. And likely it does not use the RS because even though the chances it's unambiguous greatly increase and the RS was actually meant for that, software doesn't typically use it because when CSV was invented it's existence was unknown or ignored beacue it's not human readable (I guess, don't really know actually) and so the sad story began. As you can see: I'm not a fan of CSV :) Just today I again had to waste time because at one point in the development of this otherwise fine piece of software I'm working on - even though I knew I'd regret it - I allowed it to export CSV files. Customer moved software to another machine, forgot that they once told the CSV exporter part to use the system settings, and now has CSV files with a comma separator (you know, the C in CSV). Oh the irony, that's not what they wanted.

hetspookjee 6y ago
While I couldnt agree more with your disliking of CSV, I have yet to find another more generic human-readable table file. Of all the pandas dumping methods there is no more widely available format. I find the times a CSV is utterly broken and I have to manually write the parsing logic for it quite rare actually, but then again it varies per field. What format would you rather prefer?
- rabidrat 6y ago
  JSONL (jsonlines.org)
  - micael_dias 6y ago
    http://jsonlines.org http://jsonlines.org
- brokensegue 6y ago
  pandas can directly write to parquet
  - meritt 6y ago
    parquet has a very narrow use case. it's great for internal operations but interoperability is limited unless both sides use one of the same handful of tools. csv is supported by literally every tool ever designed to work with data.
- stinos 6y ago
  Anything unambiguous really, whatever works for the job. So unless the job is looking at very limited amount of data and drawing conclusions from it just using your brain, that means you need some kind of visalisation and processing tool. Like pandas or excel since we're talking tabular data. I.e. actually the whole premise of 'let's look at data in a plain text editor' is sort of wrong, imo. It's too limited and hence not the right tool for the job most of the times. You can't do anything but the simplest things with the data anyway in a plain text editor, so from the moment your data is more than a couple of rows in a table you're better off with another tool anyway.
panos_sa 6y ago
thanks for the feedback, I just update it, you can now have csv, excel or JSONL file
iaabtpbtpnn 6y ago
CSV is totally fine, and I don't understand why people have such an issue with the format. It is very simple, here is the entire specification: the file should be encoded in UTF-8, and fields are separated by the , character. Fields may be quoted with ". If a field contains any of [",\r\n] then it must be quoted. The quote process is: replace all " characters in the field with "" and then surround the field with ". I would expect a competent programmer to be able to implement this specification as an interview question! The existence of software that produces braindead CSVs is beside the point. Don't use that software. Just do the above and everything will work.
- stinos 6y ago
  here is the entire specification: the file should be encoded in UTF-8, and fields are separated by the , character This exactly is part of the problem: how many people do you think would come up with this specific specification? You do realize in real life CSV gets used for anything which is plain text and separated and perhaps even quoted? Also 'should be' UTF-8? By what authority? The existence of software that produces braindead CSVs is beside the point. No, it is not. Especially not because there's so much software. And because it's not braindead, it's just doing something else. Which is bascially fine since there's no standard. Just do the above and everything will work. Yes, in your personal bubble. Beyond that it's totally insufficient.
- panopticon 6y ago
  The answer to your first question is in your last paragraph. CSV is an abysmal format for data exchange because the standards are weak and implementations are extremely varied. If you're writing an (en|de)coder for a specific or limited target, then sure it's fine. Otherwise I would steer clear of it.
- perl4ever 6y ago
  Lots of people like to use backslash escaping, and you are so used to it that you used it in your spec even though it leaves the ambiguity of whether \n and \r are literal or the LF and CR characters. I feel like it's perfectly expressing how it "contains the seeds of its own destruction". How much data expresses a double quote as "\"" instead of """"? It would be easy to use CSV if you never did any data interchange with it.

by stinos 6y ago

hetspookjee 6y ago

While I couldnt agree more with your disliking of CSV, I have yet to find another more generic human-readable table file. Of all the pandas dumping methods there is no more widely available format. I find the times a CSV is utterly broken and I have to manually write the parsing logic for it quite rare actually, but then again it varies per field. What format would you rather prefer?

rabidrat 6y ago
JSONL (jsonlines.org)
- micael_dias 6y ago
  http://jsonlines.org http://jsonlines.org
brokensegue 6y ago
pandas can directly write to parquet
- meritt 6y ago
  parquet has a very narrow use case. it's great for internal operations but interoperability is limited unless both sides use one of the same handful of tools. csv is supported by literally every tool ever designed to work with data.
stinos 6y ago
Anything unambiguous really, whatever works for the job. So unless the job is looking at very limited amount of data and drawing conclusions from it just using your brain, that means you need some kind of visalisation and processing tool. Like pandas or excel since we're talking tabular data. I.e. actually the whole premise of 'let's look at data in a plain text editor' is sort of wrong, imo. It's too limited and hence not the right tool for the job most of the times. You can't do anything but the simplest things with the data anyway in a plain text editor, so from the moment your data is more than a couple of rows in a table you're better off with another tool anyway.

panos_sa 6y ago

thanks for the feedback, I just update it, you can now have csv, excel or JSONL file

iaabtpbtpnn 6y ago

CSV is totally fine, and I don't understand why people have such an issue with the format. It is very simple, here is the entire specification: the file should be encoded in UTF-8, and fields are separated by the , character. Fields may be quoted with ". If a field contains any of [",\r\n] then it must be quoted. The quote process is: replace all " characters in the field with "" and then surround the field with ". I would expect a competent programmer to be able to implement this specification as an interview question! The existence of software that produces braindead CSVs is beside the point. Don't use that software. Just do the above and everything will work.

stinos 6y ago
here is the entire specification: the file should be encoded in UTF-8, and fields are separated by the , character This exactly is part of the problem: how many people do you think would come up with this specific specification? You do realize in real life CSV gets used for anything which is plain text and separated and perhaps even quoted? Also 'should be' UTF-8? By what authority? The existence of software that produces braindead CSVs is beside the point. No, it is not. Especially not because there's so much software. And because it's not braindead, it's just doing something else. Which is bascially fine since there's no standard. Just do the above and everything will work. Yes, in your personal bubble. Beyond that it's totally insufficient.
panopticon 6y ago
The answer to your first question is in your last paragraph. CSV is an abysmal format for data exchange because the standards are weak and implementations are extremely varied. If you're writing an (en|de)coder for a specific or limited target, then sure it's fine. Otherwise I would steer clear of it.
perl4ever 6y ago
Lots of people like to use backslash escaping, and you are so used to it that you used it in your spec even though it leaves the ambiguity of whether \n and \r are literal or the LF and CR characters. I feel like it's perfectly expressing how it "contains the seeds of its own destruction". How much data expresses a double quote as "\"" instead of """"? It would be easy to use CSV if you never did any data interchange with it.