3 ms·

All my data are in TSV files, separated by tabs instead of comma and strings are not quoted. It makes parsing or manually editing data so much easier. You need

All my data are in TSV files, separated by tabs instead of comma and strings are not quoted. It makes parsing or manually editing data so much easier. You need to escape only like 5 characters \0 \n \r \\ \t and even binary data will fit in there. I was checking if sqlite's CSV supports custom separator and unquoted strings for this to work. Found it doesn't.

tanin 5y ago
I built a desktop app that is based on Sqlite, and it is extended to handle other separators. Not sure if this is what you are looking for. You can try it out: https://superintendent.app https://superintendent.app PS. it also loads GBs of file in 10-20s because that part is written in C.
giraffe_lady 5y ago
Interesting I think that makes it the first mainstream real world CSV tool I've used that is truly comma separated values then, rather than delimiter-separated-defaulting-to-comma.
LaszloKv 5y ago
I think you can use sqlite-utils to accomplish this if interested: https://sqlite-utils.datasette.io/en/stable/cli.html?highlight=tsv#running-queries-directly-against-csv-or-json https://sqlite-utils.datasette.io/en/stable/cli.html?highlig...
CRConrad 5y ago
This is gonna blow your mind: There are already ASCII (and Unicode) characters for that![1],[2] Have been since forever; the inception of the ASCII standard. So it turns out everyone has been (ab)using commas, tabs, semi-colons, newlines, carriage returns, carriage-return-and-newlines, single quotes, double quotes, quotes around every field, quotes only around text fields, etc, etc... For fifty years -- totally for nothing. Use the data-delimiter characters actually defined in the ASCII / Unicode specifications to, you know, delimit your frigging data, and you won't have to bother about whether to quote or not to quote, nor about painstakingly "escaping" (and un-escaping) whatever characters you're (mis)using for delimiters. Sure as Hell blew away at least my mind when I came across it. (Not that I got to implement it at work, of course.) ___ [1]: https://en.wikipedia.org/wiki/C0_and_C1_control_codes#Field_separators https://en.wikipedia.org/wiki/C0_and_C1_control_codes#Field_... [2]: https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text

tanin 5y ago

I built a desktop app that is based on Sqlite, and it is extended to handle other separators. Not sure if this is what you are looking for. You can try it out: https://superintendent.app https://superintendent.app PS. it also loads GBs of file in 10-20s because that part is written in C.

giraffe_lady 5y ago

Interesting I think that makes it the first mainstream real world CSV tool I've used that is truly comma separated values then, rather than delimiter-separated-defaulting-to-comma.

LaszloKv 5y ago

I think you can use sqlite-utils to accomplish this if interested: https://sqlite-utils.datasette.io/en/stable/cli.html?highlight=tsv#running-queries-directly-against-csv-or-json https://sqlite-utils.datasette.io/en/stable/cli.html?highlig...

CRConrad 5y ago

This is gonna blow your mind: There are already ASCII (and Unicode) characters for that![1],[2] Have been since forever; the inception of the ASCII standard. So it turns out everyone has been (ab)using commas, tabs, semi-colons, newlines, carriage returns, carriage-return-and-newlines, single quotes, double quotes, quotes around every field, quotes only around text fields, etc, etc... For fifty years -- totally for nothing. Use the data-delimiter characters actually defined in the ASCII / Unicode specifications to, you know, delimit your frigging data, and you won't have to bother about whether to quote or not to quote, nor about painstakingly "escaping" (and un-escaping) whatever characters you're (mis)using for delimiters. Sure as Hell blew away at least my mind when I came across it. (Not that I got to implement it at work, of course.) ___ [1]: https://en.wikipedia.org/wiki/C0_and_C1_control_codes#Field_separators https://en.wikipedia.org/wiki/C0_and_C1_control_codes#Field_... [2]: https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text