3 ms·
I have my own little toolkit of csv parsing tools that do most of these (using python's csv module.) I suspect most people who do deal with bulk CSV files all
by acveilleux 11y ago
I have my own little toolkit of csv parsing tools that do most of these (using python's csv module.)
I suspect most people who do deal with bulk CSV files all the time do as well. I'd be curious about how gracefully it handles large CSV files.
The lack of quoting support kills it for my use case however. I wrote my own tools, starting with csv_cut because cut(1) didn't do quoting.
- john_kerl 11y agoquoting is key but i've not yet stumbled for lack of it. but it's very much worth doing. miller handles large files well. it's fast enough, and doesn't hold entire file contents unless it has to (e.g. sort). so gigabytes of input data are fine.
- ggrothendieck 11y agoRegarding quoting support, check out https://github.com/dbro/csvquote https://github.com/dbro/csvquote
- acveilleux 11y agoNot sufficient, but close. It substitutes the quoted character for another one.
- dbro 11y agoHi- I'm curious to know what your use case is. Can you explain why substitution is not sufficient?
- acveilleux 11y ago(1) I filter on column content using regex and dealing with a sub character adds complexity. (2) Many of my columns are free-form text containing commas, carriage returns, new lines, tab, vertical tabs and file separator (0x1c). Occasionally, text is in UCS-2/UTF-16 or uses UTF-8 and foreign characters (a non-trivial quantity of the text I process is in French for example.) (If you read between the lines here, some columns can contain MLLP-encoded HL7 messages, others contain free-form text and I'm in the medical field.)