3 ms·

Here's an example using csvkit: $ time xsv stats --everything /tmp/nfl_all_plays.csv > stats.csv real 5.723 user 14.390 sys 1.914

Here's an example using csvkit: $ time xsv stats --everything /tmp/nfl_all_plays.csv > stats.csv real 5.723 user 14.390 sys 1.914 $ time csvstat /tmp/nfl_all_plays.csv ^C after 2.5 minutes Here's the data: https://burntsushi.net/stuff/nfl_all_plays.csv https://burntsushi.net/stuff/nfl_all_plays.csv It's only 74MB. Let's take a smaller slice to see how long csvstat really takes. This is just the first 10,000 records, which is ~3MB: https://burntsushi.net/stuff/nfl_all_plays_small.csv https://burntsushi.net/stuff/nfl_all_plays_small.csv $ time csvstat /tmp/nfl_all_plays_small.csv > /tmp/stats.csv real 1:01.85 user 1:01.70 sys 0.103 $ time xsv stats --everything /tmp/nfl_all_plays_small.csv > /tmp/stats.csv real 0.308 user 0.576 sys 0.071 Now technically, csvstat is doing more work in that it seems to be computing a frequency table as well. But we can just do the same for xsv and add the time, with the knowledge that it would be faster if it were coupled into `xsv stats`: $ time xsv frequency /tmp/nfl_all_plays_small.csv > /tmp/frequency.csv real 0.251 user 0.187 sys 0.063 Now let's see how xsv fairs on a much larger sample, which is just nfl_all_plays.csv repeated 10 times and is ~800MB: $ ls -lh /tmp/nfl_all_plays_huge.csv -rw-r--r-- 1 andrew users 806M Sep 8 20:34 /tmp/nfl_all_plays_huge.csv $ time xsv index /tmp/nfl_all_plays_huge.csv real 2.041 user 1.876 sys 0.163 $ time xsv stats --everything /tmp/nfl_all_plays_huge.csv > /tmp/stats.csv real 28.336 user 4:36.45 sys 24.212 $ time xsv frequency /tmp/nfl_all_plays_huge.csv > /tmp/frequency.csv real 6.077 user 1:16.51 sys 1.873 That indexing step lets xsv do its processing in parallel. Good luck doing that in Python without blowing your memory budget. :-) csvkit would either take hours to handle that much data or would more likely run out of memory. With that said, I was able to write a Python program that just counted records within an order of magnitude of `xsv count`, but it was still a few times slower.

shawn 8y ago
Mm, using someone else's parser would defeat the spirit of the challenge. I think xsv is worthwhile for being a robust parser, not necessarily for its performance. And my claim is that you'd be able to write it faster, without trading any security guarantees, in Lua, without sacrificing much performance. There's that pesky word, "much" performance. And that's really the interesting part here. How much would you trade away by shedding Rust? My hypothesis is less than 50% additional overhead. Thanks for providing a dataset. I think LuaJIT will match these stats, and it's a good baseline to start with. But yes, the CSV parser is around 15k lines. That'd be the trickiest part.
- burntsushi 8y ago
  xsv doesn't have its own csv parser, it uses a Rust library to parse csv[1], which is almost 4 times the size of xsv itself. I just happen to have written it. In any case, it would be fun to see an implementation in LuaJIT, especially if you did the CSV parser as well. Although, I think that takes you well outside a weekend project unless you cheat. :-) I don't know the performance characteristics of LuaJIT, but I assume they are better than Python's. I don't know how much better. In any case, this challenge was much more interesting to me when you were talking about Python. Also, I don't really care about a claim that says you could write it faster. That's borderline meaningless in my work unless you're talking about an order of magnitude difference, and I sincerely doubt that. [1] - https://crates.io/crates/csv https://crates.io/crates/csv
  - shawn 8y ago
    Also, I don't really care about a claim that says you could write it faster. That's borderline meaningless in my work unless you're talking about an order of magnitude difference, and I sincerely doubt that. Ah, fair point. If there is no benefit to writing software faster, then yes, the discussion is moot. Apologies if it sounded like I was being a dick. I meant to come across as a “player 2 has entered the game,” but it probably just sounded annoying. I’ve been reimplementing some C projects in LuaJIT (more specifically, a dialect of Lisp that compiles to LuaJIT), and it certainly feels an order of magnitude less overhead to get work done. Perhaps it would be interesting to bind the CSV crate to LuaJIT, and then do a direct translation of XSV. The original discussion was about CLI tools, which is one area that scripting languages excel in, and isn’t necessarily enhanced by the benefits of static typing. Creationix has done a lot of excellent work in popularizing LuaJIT’s benefits: https://github.com/luvit/luvit https://github.com/luvit/luvit
    - burntsushi 8y ago
      > I’ve been reimplementing some C projects in LuaJIT (more specifically, a dialect of Lisp that compiles to LuaJIT), and it certainly feels an order of magnitude less overhead to get work done. It is interesting how perspectives change things, because I wouldn't be altogether surprised by that actually. I've used both C and Lua quite a bit myself (although not too much recently), and I can believe that. But if you substituted C for Rust, then I would look at it with great skepticism. That is, I don't see it as types-vs-not-types, but memory-safety-vs-not-memory-safety in addition to having a ecosystem that is very easy to draw on in a granular way. And I didn't think you were too annoying. :-) Building an ffi layer over the csv crate would be cool. It would probably be easiest to do it over csv-core, and then build up the convenience abstractions on the C side (or perhaps even at the Lua level) and therefore avoid the csv crate entirely. csv-core will be simpler because it is designed to work in `no_std` environments, which means it never uses platform specific routines and never allocates. Still though, you'll probably need to find a way to amortize allocation on the Lua side. > and isn’t necessarily enhanced by the benefits of static typing Yeah, I mean I obviously very very very very strongly disagree. We are clearly polar opposites here. If I never write anything substantial in a unityped language again, it will be too soon.

shawn 8y ago

Mm, using someone else's parser would defeat the spirit of the challenge. I think xsv is worthwhile for being a robust parser, not necessarily for its performance. And my claim is that you'd be able to write it faster, without trading any security guarantees, in Lua, without sacrificing much performance. There's that pesky word, "much" performance. And that's really the interesting part here. How much would you trade away by shedding Rust? My hypothesis is less than 50% additional overhead. Thanks for providing a dataset. I think LuaJIT will match these stats, and it's a good baseline to start with. But yes, the CSV parser is around 15k lines. That'd be the trickiest part.

burntsushi 8y ago
xsv doesn't have its own csv parser, it uses a Rust library to parse csv[1], which is almost 4 times the size of xsv itself. I just happen to have written it. In any case, it would be fun to see an implementation in LuaJIT, especially if you did the CSV parser as well. Although, I think that takes you well outside a weekend project unless you cheat. :-) I don't know the performance characteristics of LuaJIT, but I assume they are better than Python's. I don't know how much better. In any case, this challenge was much more interesting to me when you were talking about Python. Also, I don't really care about a claim that says you could write it faster. That's borderline meaningless in my work unless you're talking about an order of magnitude difference, and I sincerely doubt that. [1] - https://crates.io/crates/csv https://crates.io/crates/csv
- shawn 8y ago
  Also, I don't really care about a claim that says you could write it faster. That's borderline meaningless in my work unless you're talking about an order of magnitude difference, and I sincerely doubt that. Ah, fair point. If there is no benefit to writing software faster, then yes, the discussion is moot. Apologies if it sounded like I was being a dick. I meant to come across as a “player 2 has entered the game,” but it probably just sounded annoying. I’ve been reimplementing some C projects in LuaJIT (more specifically, a dialect of Lisp that compiles to LuaJIT), and it certainly feels an order of magnitude less overhead to get work done. Perhaps it would be interesting to bind the CSV crate to LuaJIT, and then do a direct translation of XSV. The original discussion was about CLI tools, which is one area that scripting languages excel in, and isn’t necessarily enhanced by the benefits of static typing. Creationix has done a lot of excellent work in popularizing LuaJIT’s benefits: https://github.com/luvit/luvit https://github.com/luvit/luvit
  - burntsushi 8y ago
    > I’ve been reimplementing some C projects in LuaJIT (more specifically, a dialect of Lisp that compiles to LuaJIT), and it certainly feels an order of magnitude less overhead to get work done. It is interesting how perspectives change things, because I wouldn't be altogether surprised by that actually. I've used both C and Lua quite a bit myself (although not too much recently), and I can believe that. But if you substituted C for Rust, then I would look at it with great skepticism. That is, I don't see it as types-vs-not-types, but memory-safety-vs-not-memory-safety in addition to having a ecosystem that is very easy to draw on in a granular way. And I didn't think you were too annoying. :-) Building an ffi layer over the csv crate would be cool. It would probably be easiest to do it over csv-core, and then build up the convenience abstractions on the C side (or perhaps even at the Lua level) and therefore avoid the csv crate entirely. csv-core will be simpler because it is designed to work in `no_std` environments, which means it never uses platform specific routines and never allocates. Still though, you'll probably need to find a way to amortize allocation on the Lua side. > and isn’t necessarily enhanced by the benefits of static typing Yeah, I mean I obviously very very very very strongly disagree. We are clearly polar opposites here. If I never write anything substantial in a unityped language again, it will be too soon.