3 ms·
Seems like a waste to do CSV->TSV without going all the way: http://www.tsvx.org/ http://www.tsvx.org/ The problem with TSVs and CSVs is that you might get an
by gibba999 7y ago
Seems like a waste to do CSV->TSV without going all the way:
http://www.tsvx.org/ http://www.tsvx.org/
The problem with TSVs and CSVs is that you might get an odd datatype 1TB into a file. For example, what you expect to be an integer value is somehow a string.
TSVx extends TSV to add standard formats for things like headers which allows for strict typing. You can do things like export a database table and import losslessly. You can even export from MySQL and import into PostgreSQL most of the times without pain.
The strict typing also avoids a lot of potential security issues. And in an environment where you control both ends (so you don't need to worry about security of where the file came from), it leads to much nicer APIs: you can refer to things by names rather than column numbers. It's more readable, and if the order or number of columns changes, nothing breaks.
- osrec 7y agoInteresting. Anyone have any experience of using tsvx in the wild?
- benatkin 7y agotsvx apparently isn't tsv. so it's not a good example of going all the way with tsv. it has a header that isn't tab separated. technically all text files are variable length tsvs. But a proper tsv file has the same number of columns per row.
- chubot 7y agoHm interesting, I hadn't seen tsvx. Where is it being used? Are you part of its development? FWIW I have my own TSV enhancement that is more modest: https://github.com/oilshell/oil/wiki/TSV2-Proposal https://github.com/oilshell/oil/wiki/TSV2-Proposal I hope to find time to prototype it in https://www.oilshell.org/ https://www.oilshell.org/ first. Although it's very simple -- about as simple as JSON -- I think it needs some real world use first to prove it. I think adding metadata is orthogonal to cleaning up the problems with TSV, e.g. that you can't represent strings with tabs (or newlines) in fields. Also I wouldn't want a TSV enhancement to depend on YAML, as it's quite a big and confusing (e.g. https://news.ycombinator.com/item?id=17359376 https://news.ycombinator.com/item?id=17359376)
- woofie11 7y agoI think it depends on how YAML is used. Requiring full YAML on an input is cumbersome on users (e.g. if any downstream user MUST parse ANY YAML). On the other hand, a simple format which happens to be YAML-compatible means anyone with a YAML parser can parse it without reinventing wheels. That seems a good thing. An example of that is JSON. Any YAML parser will parse JSON. TSVX merely seems to guarantee that the output is a simplified (but compatible) form of YAML. All this means is that any YAML parser will read it, and you don't need to write your own. But it's just a colon-delimited dictionary with a few constraints.
- enriquto 7y agoIt wouldn't seem that tsvx is "going all the way", it's a completely different thing that dirties your numbers with silly metadata.
- woofie11 7y agoTSVs aren't just numbers. A TSV might be numbers for the first terabyte, and then include a string, just for the heck of it. I'd call that "dirtying numbers," rather than an removable header which allows you to programmatically validate that your numbers have not, in fact, been dirtied.