4 ms·

Author here. I was really hoping to get binaries for Windows/Mac/Linux available before sharing it with others, but clearly I snoozed. I do have them available

Author here. I was really hoping to get binaries for Windows/Mac/Linux available before sharing it with others, but clearly I snoozed. I do have them available for Linux though, so you don't have to install Rust in order to try xsv: https://github.com/BurntSushi/xsv/releases https://github.com/BurntSushi/xsv/releases Otherwise, you could try using rustle[1], which should install `xsv` in one command (but it downloads Rust and compiles everything for you). While I have your attention, if I had to pick one of the cooler features of xsv, I'd tell you about `xsv index`. Its a command that creates a very simple index that permits random access to your CSV data. This makes a lot of operations pretty fast. For example: xsv index worldcitiespop.csv # ~1.5s for 145MB xsv slice -i 500000 worldcitiespop.csv | xsv table # instant, plus elastic tab stops for good measure That second command doesn't have to chug through the first 499,999 records to get the 500,000th record. This can make other commands faster too, like random sampling and statistic gathering. (Parallelism is used when possible!) Finally, have you ever seen a CLI app QuickCheck'd? Yes. It's awesome! :-) https://github.com/BurntSushi/xsv/blob/master/tests/test_sort.rs https://github.com/BurntSushi/xsv/blob/master/tests/test_sor... [1] - https://github.com/brson/rustle https://github.com/brson/rustle

simi_ 12y ago
I'm looking forward to playing with cool languages like Rust, Nim, and Elm. But when I read stuff like this I remember why I love using Go every day. Generating binaries for multiple platforms is braindead easy, as is building from source on any system with Go installed. That aside, really great work OP! I quite like the CSV format and had 2 ideas based on my experience with it that I'd love to get an opinion on: 1. markdown compiler plugin to expand ![title](filename.csv) 2. barebones, imgur-like website for quick CSV file[s] upload, maybe also a public gallery to showcase interesting data (obviously all uploads marked public/unlisted/private)
- burntsushi 12y ago
  > But when I read stuff like this I remember why I love using Go every day. Me too! I wrote a window manager in Go[1] that I've been using for years now. I love that it takes <30 seconds to download and compile the whole thing. No C dependencies (compile or runtime) at all. With that said, doing it with Rust should be almost as easy. There's no `cargo install` command, but I think it's only a matter of time. :-) Your ideas seem cool, by the way! Sharing CSV data would be especially nice. [1] - https://github.com/BurntSushi/wingo https://github.com/BurntSushi/wingo
  - simi_ 12y ago
    Thanks for your support, I'll start working on the idea then. I just checked your Gh repos, your productivity is astonishing. Chapeaux! Also, cute handle. I suspect the problem with CSV will be working around the myriad of broken implementations and making sense of malformed data.
    - burntsushi 12y ago
      > I suspect the problem with CSV will be working around the myriad of broken implementations and making sense of malformed data. Indeed. My CSV parser (and Python's) is pretty interesting in that regard. There are very few things that actually cause a parse error. You can see here[1] that the only two errors occur if there are unequal length records (which can be disabled by enabling the "flexible" option) and invalid UTF-8 data (which can be avoiding by reading everything into plain byte strings). That means that any arbitrary data gets parsed into something. There are various mechanisms in the CSV parser's state machine that make decisions for you. Mostly, I used the same types of decisions that Python makes. For example: >>> import csv >>> from StringIO import StringIO >>> list(csv.reader(StringIO('a, "b,c'))) [['a', ' "b', 'c']] >>> list(csv.reader(StringIO('a,"b,c'))) [['a', 'b,c']] Whaaaa? Yeah, if our CSV parsers were conformant with the spec, then both of these examples should fail. But they succeed and result in slightly different interpretations based on whether a space character precedes the quote. Therefore, "good" CSV parsers tend to implement a superset of RFC 4180 when parsing, but usually implement it strictly when writing. (My CSV parser ends up with the same parse as Python here, because it seemed like a good decision to follow its lead since it is used ubiquitously.) [1] - http://burntsushi.net/rustdoc/csv/enum.ParseErrorKind.html http://burntsushi.net/rustdoc/csv/enum.ParseErrorKind.html
  - steveklabnik 12y ago
    > There's no `cargo install` command, but I think it's only a matter of time. :-) https://github.com/rust-lang/cargo/pull/1318 https://github.com/rust-lang/cargo/pull/1318
- stupidcar 12y ago
  I have never used Rust before, but I was able to install it and compile XSV on OS X in about 5 minutes. So maybe cross-compilation isn't straightforward yet, but building from source definitely is.
- timClicks 12y ago
  FWIW one of the reasons that I have been heavily looking at Nim for the last few weeks is its ease of generating static binaries.
timClicks 12y ago
Sorry for not looking into it myself, but what is the format of your index file (I'm assuming the index is stored as a file somewhere). Could other tools read from that too?
- burntsushi 12y ago
  It's extremely simple. Even if you don't know Rust, this code should be readable: https://github.com/BurntSushi/rust-csv/blob/master/src/index/mod.rs#L52-L69 https://github.com/BurntSushi/rust-csv/blob/master/src/index... Basically, it writes a 64 bit byte offset (big endian) for the start of each record. The very last byte is then the total number of records. (I forget why I'm doing this, since the count should be computable from the size of the index file.) The "start of each record" means "if you point a CSV parser at this byte offset, then it will successfully read a record." So when you want to read record `i`, seek the index to byte offset `i * 8`, read 8 bytes (big endian) into a 64 bit integer, then seek to that byte offset in your CSV data. If the CSV data changes, then you're in trouble. One easy way to handle it (which isn't bulletproof) is to check the last modified times of the index and the CSV file and regenerate the index (or refuse to use it with an error).
  - breck 12y ago
    For an append to a CSV file it shouldn't be too hard to update the index. Remove the last byte in your current format and just parse the additional lines and append to your index. And I'm sure there's a technique out there to handle the update case. I'm not too familiar with hard disk operations so I wouldn't know it. If you insert a byte at position 2 in a 1GB file will the whole file get rewritten? Or are there techniques out there to store updates to a char array without having to rewrite the whole thing?
    - burntsushi 12y ago
      I'm pretty sure you're barking up a long and complex tree. The first problem (for xsv) is that it doesn't control the csv data. So it doesn't know what changes were made. It could have even become an invalid csv file. The second problem is that insertions in sorted sequences are hard and are one of the fundamental problems in a database implementation: http://en.wikipedia.org/wiki/B-tree#The_database_problem http://en.wikipedia.org/wiki/B-tree#The_database_problem

simi_ 12y ago

I'm looking forward to playing with cool languages like Rust, Nim, and Elm. But when I read stuff like this I remember why I love using Go every day. Generating binaries for multiple platforms is braindead easy, as is building from source on any system with Go installed. That aside, really great work OP! I quite like the CSV format and had 2 ideas based on my experience with it that I'd love to get an opinion on: 1. markdown compiler plugin to expand ![title](filename.csv) 2. barebones, imgur-like website for quick CSV file[s] upload, maybe also a public gallery to showcase interesting data (obviously all uploads marked public/unlisted/private)

burntsushi 12y ago
> But when I read stuff like this I remember why I love using Go every day. Me too! I wrote a window manager in Go[1] that I've been using for years now. I love that it takes <30 seconds to download and compile the whole thing. No C dependencies (compile or runtime) at all. With that said, doing it with Rust should be almost as easy. There's no `cargo install` command, but I think it's only a matter of time. :-) Your ideas seem cool, by the way! Sharing CSV data would be especially nice. [1] - https://github.com/BurntSushi/wingo https://github.com/BurntSushi/wingo
- simi_ 12y ago
  Thanks for your support, I'll start working on the idea then. I just checked your Gh repos, your productivity is astonishing. Chapeaux! Also, cute handle. I suspect the problem with CSV will be working around the myriad of broken implementations and making sense of malformed data.
  - burntsushi 12y ago
    > I suspect the problem with CSV will be working around the myriad of broken implementations and making sense of malformed data. Indeed. My CSV parser (and Python's) is pretty interesting in that regard. There are very few things that actually cause a parse error. You can see here[1] that the only two errors occur if there are unequal length records (which can be disabled by enabling the "flexible" option) and invalid UTF-8 data (which can be avoiding by reading everything into plain byte strings). That means that any arbitrary data gets parsed into something. There are various mechanisms in the CSV parser's state machine that make decisions for you. Mostly, I used the same types of decisions that Python makes. For example: >>> import csv >>> from StringIO import StringIO >>> list(csv.reader(StringIO('a, "b,c'))) [['a', ' "b', 'c']] >>> list(csv.reader(StringIO('a,"b,c'))) [['a', 'b,c']] Whaaaa? Yeah, if our CSV parsers were conformant with the spec, then both of these examples should fail. But they succeed and result in slightly different interpretations based on whether a space character precedes the quote. Therefore, "good" CSV parsers tend to implement a superset of RFC 4180 when parsing, but usually implement it strictly when writing. (My CSV parser ends up with the same parse as Python here, because it seemed like a good decision to follow its lead since it is used ubiquitously.) [1] - http://burntsushi.net/rustdoc/csv/enum.ParseErrorKind.html http://burntsushi.net/rustdoc/csv/enum.ParseErrorKind.html
- steveklabnik 12y ago
  > There's no `cargo install` command, but I think it's only a matter of time. :-) https://github.com/rust-lang/cargo/pull/1318 https://github.com/rust-lang/cargo/pull/1318
stupidcar 12y ago
I have never used Rust before, but I was able to install it and compile XSV on OS X in about 5 minutes. So maybe cross-compilation isn't straightforward yet, but building from source definitely is.
timClicks 12y ago
FWIW one of the reasons that I have been heavily looking at Nim for the last few weeks is its ease of generating static binaries.

timClicks 12y ago

Sorry for not looking into it myself, but what is the format of your index file (I'm assuming the index is stored as a file somewhere). Could other tools read from that too?

burntsushi 12y ago
It's extremely simple. Even if you don't know Rust, this code should be readable: https://github.com/BurntSushi/rust-csv/blob/master/src/index/mod.rs#L52-L69 https://github.com/BurntSushi/rust-csv/blob/master/src/index... Basically, it writes a 64 bit byte offset (big endian) for the start of each record. The very last byte is then the total number of records. (I forget why I'm doing this, since the count should be computable from the size of the index file.) The "start of each record" means "if you point a CSV parser at this byte offset, then it will successfully read a record." So when you want to read record `i`, seek the index to byte offset `i * 8`, read 8 bytes (big endian) into a 64 bit integer, then seek to that byte offset in your CSV data. If the CSV data changes, then you're in trouble. One easy way to handle it (which isn't bulletproof) is to check the last modified times of the index and the CSV file and regenerate the index (or refuse to use it with an error).
- breck 12y ago
  For an append to a CSV file it shouldn't be too hard to update the index. Remove the last byte in your current format and just parse the additional lines and append to your index. And I'm sure there's a technique out there to handle the update case. I'm not too familiar with hard disk operations so I wouldn't know it. If you insert a byte at position 2 in a 1GB file will the whole file get rewritten? Or are there techniques out there to store updates to a char array without having to rewrite the whole thing?
  - burntsushi 12y ago
    I'm pretty sure you're barking up a long and complex tree. The first problem (for xsv) is that it doesn't control the csv data. So it doesn't know what changes were made. It could have even become an invalid csv file. The second problem is that insertions in sorted sequences are hard and are one of the fundamental problems in a database implementation: http://en.wikipedia.org/wiki/B-tree#The_database_problem http://en.wikipedia.org/wiki/B-tree#The_database_problem