8 ms·

Command-line tools for data science

346 points by robdoherty2 13y ago

nonchalance 13y ago
> Ideally, json2csv would have added the header. You can raise an issue (https://github.com/jehiah/json2csv https://github.com/jehiah/json2csv) or just fork and add a flag to include a header line
- daniel-levin 13y ago
  https://github.com/daniel-levin/json2csv https://github.com/daniel-levin/json2csv Obligatory here-I-did-it
  - jeroenjanssens 13y ago
    Nice one!
  - daniel-levin 13y ago
    Update: now merged with the mainline repo
harrytuttle 13y ago
Nice tools. You can actually do most of that with vanilla PowerShell and Excel believe it or not but it's much fuglier and you spend most of your way working around edge cases. The only thing that scares me about this though is that JSON is a terrible format for storing numbers in. There is no way of specifying a decimal type for example so aggregations and calculations have no implied precision past floating point values.
the_cat_kittles 13y ago
Some of these are cool, but I don't understand the preference for command line tools over writing, say, a short python script in vim and then executing it. I see the two methods achieving the same results with about the same amount of text, except with a script, its savable and much easier to type.
- TeMPOraL 13y ago
  It's equivalent if you look at the command line as a (very popular) REPL, and at the tools as functions.
- gnaritas 13y ago
  > except with a script, its savable and much easier to type. That's an odd thing to say as Python scripts are no more savable than shell scripts. Nor are Python scripts any easier to type than shell scripts. Also, as shell scripts tend to pipe commands together, it's multiple process and thus multiple core by default, unlike Python.
  - the_cat_kittles 13y ago
    I wasn't thinking of making them a shell script, I was thinking of using them directly from the prompt- that makes sense. Come to think of it, I should be writing more shell scripts instead of retyping stuff into the prompt.
    - goldfeld 13y ago
      Yeah, that's like saying Python sucks for scripting because all your work in the REPL won't be saved. Goes both ways.
      - zokier 13y ago
        That's why ipython is so nice :)
    - williamcotton 13y ago
      Or better yet, use make.
  - reustle 13y ago
    I could be wrong, but doesn't each piped command wait for the previous one to finish (not multi-core)?
dergachev 13y ago
These are great! I wrote a quick ruby script that converts one or more CSV files into an SQLITE db file, so you can easily query them. https://github.com/dergachev/csv2sqlite https://github.com/dergachev/csv2sqlite
- aw3c2 13y ago
  No (real) need for a script, you can import in the sqlite3 commandline. First make the table, then set a .separator, then .import the file. I am not sure how performant it is though, so creating transactions outside might be better in some cases.
  - kevnin 13y ago
    Depends on how often you have to do this... if it saves time, I'm in
  - mkiwala 13y ago
    The sqlite3 .import command does not handle quoted csv values. "Doe, John", 1234 Pine St, Springfield That would be imported as 4 fields, not 3.
    - SQLite 13y ago
      Used to be true. But recent versions of SQLite fix this.
  - dergachev 13y ago
    csv2sqlite actually generates the table definitions for you, so I think that's a big improvement when you just downloaded a CSV file with 30 columns in it and need to quickly verify it or extract some data from it. Also supports parsing multiple CSV files at once, so you can easily do joins.
th0ma5 13y ago
The ability to script the R stuff with pipes and whatnot is very welcome to me. I tried using R as a part of a system of normalizing package installs across environments. It was sort of kludgey but ultimately worked okay. Setting up the environment as the first part of any script you push into R is necessary, and that makes sense. Python and Java seem to have more support for this kind of thing (environment swapping) on the command line, or at least perhaps I wasn't looking the right places perhaps for R. It has been a year since I've messed with this though.
cgrubb 13y ago
Nice tools. I see some stuff I might add to my own personal repo: https://github.com/clarkgrubb/data-tools https://github.com/clarkgrubb/data-tools Would be nice if some of these tools became standard at the command line. I don't know about "sample" though, since that can easily be implemented in awk: awk 'rand() < 0.2' /etc/passwd
- jeroenjanssens 13y ago
  That's a nice collection of tools you have there. I'll add it to the post.
gav 13y ago
If you are working with XML, XMLStarlet is invaluable: http://xmlstar.sourceforge.net/ http://xmlstar.sourceforge.net/
bcantrill 13y ago
Glad to see that we're not the only ones who are emphatic believers in applying the Unix philosophy to data science! Those interested in this may also be interested in Mark Cavage and Dave Pacheco's presentation last week at Surge on Manta[1], a new storage service that we have developed that draws very direct inspiration from Unix -- and allows you to use the tools mentioned by Jeroen across very large data sets. awk FTMFW! [1] http://us-east.manta.joyent.com/mark.cavage/public/surge2013_manta_final.pdf http://us-east.manta.joyent.com/mark.cavage/public/surge2013...
- mathattack 13y ago
  An open question - why do you prefer doing this in the command line versus via a scripting language like Python? I get the piping philosophy, but why one versus the other?
  - robbles 13y ago
    Most of the well-know "unix-style" command-line tools, such as grep, sort, etc. actually have very high performance. Their relatively constrained use cases allow the authors to implement decent algorithms and optimizations (e.g. sort uses merge sort, grep uses all kinds of optimizations: http://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html http://lists.freebsd.org/pipermail/freebsd-current/2010-Augu...) In contrast, when you're building a custom pipeline in a high-level language, you're optimizing for simple solutions and are not likely to get better performance unless you hit an edge case where the standard tools do really poorly.
    - paulrosenzweig 13y ago
      Actually sort is even better than normal (purely in-memory) merge sort. It looks at available memory and writes out sorted files to merge. http://vkundeti.blogspot.com/2008/03/tech-algorithmic-details-of-unix-sort.html http://vkundeti.blogspot.com/2008/03/tech-algorithmic-detail...
      - robbles 13y ago
jeroenjanssens 13y ago
Suggestions are more than welcome. I'm currently keeping the post up-to-date with suggested tools and repositories.
- dirtyvagabond 13y ago
  nice post! i'd like to make a shameless plug for Drake, a 'make for data' workflow tool. https://github.com/Factual/drake https://github.com/Factual/drake
- susi22 13y ago
  http://www.hdfgroup.org/HDF5/doc/RM/Tools.html http://www.hdfgroup.org/HDF5/doc/RM/Tools.html http://www.hdfgroup.org/tools5desc.html#1 http://www.hdfgroup.org/tools5desc.html#1 The first thing I do when I deal with data which has more than --let's say-- 10,000 rows is to put it in HDF file format and work with that. Saves a ton of time while developing a script. I had a python script do a histogram and it ran ~15sec for a file with 100k rows. With converting it first to HDF it ran in ~0.5sec. The import in python is also much shorter (two lines). HDF is made for high performance numerical I/O. It's great and you can query several structures and even do slices of arrays on the command line (with h5tools). It's also widely used by Octave, Python, R, Matlab... And you don't have a drawback since you can just pipe it into existing command line tools with a h5dump. HDF5: http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Performance.pdf http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Performance.pdf http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Overview.pdf http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Overview.pdf
  - twistedpair 13y ago
    Wow, those benchmarks are old. 195MHz test machine? Still, 15K rows is not much. Just did that 100K rows (read/sum) bit in Octave. Took about 1/4 second to extract and sum. I assume HDF rocks for much larger sets.
- susi22 13y ago
  This also really helpful (pity it's relatively unknown): https://github.com/dbro/csvquote https://github.com/dbro/csvquote And: http://en.wikipedia.org/wiki/GNU_Core_Utilities http://en.wikipedia.org/wiki/GNU_Core_Utilities (section "Text utilities") http://directory.fsf.org/wiki/Textutils http://directory.fsf.org/wiki/Textutils Run "$ info coreutils"
peterwwillis 13y ago
So weird... I didn't know about the csvkit I guess (or maybe it wasn't available for cygwin?), so I wrote my own a year ago. It supports field names (in addition to column numbers) so your data can move around and you'll still reference the right logical data.
- michaelmior 13y ago
  csvkit is awesome. I discovered it by chance a few months ago and I've found it to be really well-written. It correctly handles a lot of weird edge cases (e.g. newlines in the middle of a record) For the record, csvkit supports field names as well :)
revertts 13y ago
You should check out RecordStream. It has a lot of convenient processing built around JSON, and a lot of ways to get data to/from JSON. https://github.com/benbernard/RecordStream/tree/master/doc https://github.com/benbernard/RecordStream/tree/master/doc https://github.com/benbernard/RecordStream/blob/master/doc/RecordStreamStory.pod https://github.com/benbernard/RecordStream/blob/master/doc/R...
- ffk 13y ago
  I second this suggestion. It doesn't just slurp and serialize data, it also is capable of filtering, transforming, aggregating (+ stats) and pretty printing streams of json records.
deleted 13y ago
[deleted]
gerad 13y ago
I'm surprised that the post didn't call out the unix built-ins cut, sort, uniq, and cat. There's an amazing amount of data processing you can do with just those commands.
- sturadnidge 13y ago
  He linked another post that covers those http://www.gregreda.com/2013/07/15/unix-commands-for-data-science/ http://www.gregreda.com/2013/07/15/unix-commands-for-data-sc...
mjn 13y ago
I like Google's crush-tools, which works on delimited data (e.g. tab-delimited), a somewhat simpler and faster format than CSV. Lots of the built-in Unix tools also work on delimited data (cut, join, sort, etc.), but crush-tools fills in some gaps, like being able to do a 'uniq' on specific fields, do outer joins, sum columns, etc.: https://code.google.com/p/crush-tools/ https://code.google.com/p/crush-tools/
- sturadnidge 13y ago
  They look interesting, thanks! Any experience using them over GB or TB datasets?
  - mjn 13y ago
    I've used them on tens-of-GB datasets (not TB), and they're quite fast except for those implemented in Perl, which is kind of a hidden gotcha. For example, calcfield is barely usable, because it loops a perl expression over every record in the file. But things like funiq and cutfield are fast, at least as fast as the traditional Unix tools. And if you have pre-sorted data, aggregate2 is a nice aggregation tool for larger-than-memory datasets.
    - sliverstorm 13y ago
      That's interesting. I've written Perl tools that apply pattern-matching to several-GB flat files, and while it was horrifyingly slow at first, I was able to get the performance down to a minute in the average case. Honestly the whole time I think I/O was a greater limiting factor than Perl's processing speed.
aficionado 13y ago
A very powerful command line tool for Machine Learning: http://bigmler.readthedocs.org/en/latest/ http://bigmler.readthedocs.org/en/latest/ Create a predictive model is as simple as: bigmler --train < data.csv or create a predictive model for Bitcoin volume taking online data from Quandl and parsing it with jq in just one line. curl --silent "http://www.quandl.com/api/v1/datasets/BITCOIN/BTCDEEUR.json" http://www.quandl.com/api/v1/datasets/BITCOIN/BTCDEEUR.json" | jq -c ".data" | bigmler --train --field-attributes <(count=0; for i in `curl --silent "http://www.quandl.com/api/v1/datasets/BITCOIN/BTCDEEUR.json" http://www.quandl.com/api/v1/datasets/BITCOIN/BTCDEEUR.json" | jq -c ".column_names[]"`; do echo "$count, $i"; count=$[$count+1]; done) --name bitcoin More info here: http://blog.bigml.com/2013/01/31/fly-your-ml-cloud-like-a-kite-with-bigmler-the-command-line-tool-for-machine-learning/ http://blog.bigml.com/2013/01/31/fly-your-ml-cloud-like-a-ki...
- diziet 13y ago
  Hmm -- the model did not seem to spot something a simple linear regression did.
  - aficionado 13y ago
    In this particular case, I think that you are right. But isn't it powerful to have it at the command line level? It's great to quickly create models as you can also use them to make predictions at the command line level.
- jeroenjanssens 13y ago
  Impressive. I'll add it to the list.
dima55 13y ago
Here's another that I use all the time: https://github.com/dkogan/feedgnuplot https://github.com/dkogan/feedgnuplot It reads data on stdin, and makes plots. More or less anything that gnuplot can do is supported, which is quite a lot. Realtime plotting of streaming data is supported as well. This is also in Debian, and is apt-gettable. Disclaimer: I am the author
paulgb 13y ago
Great list. Shameless self-plug: I wrote a tool similar to sample available on pip as subsample. It supports the approximate sample algorithm that yours does as well as reservoir sampling and an exact two-pass algorithm that runs in O(1) space. Docs: https://github.com/paulgb/subsample https://github.com/paulgb/subsample
mrcactu5 13y ago
I feel like a dumbass. The two great data science tools I learned for command line are grep and sed :-)
zmmmmm 13y ago
Worth noting that the Rio tool does not seem to account for R's concurrency / race conditions when running RScript. Basically, if you run it a lot in parallel you will get random failures. Although I am sure it is not exactly meant for high concurrency situations, it would be good if it accounted for this.
helloTree 13y ago
Maybe I should take a look at awk/sed but I never found its language orthogonal. For simple scripts I mostly use Python or Haskell via something like ghc -e 'interact ((++"\n") . show . length . lines)' < file EDIT: I know that it's the same like "wc -l" but using that approach I can solve also problems that may not be well suited for awk/sed. Or maybe I just have to see some convincing one-liners.
fluidcruft 13y ago
There is a command-line optimizer. I always forget it's name and can never find it. It is very cool for doing simplex optimizations etc. I will try yet again to find it and edit.
tomrod 13y ago
This post and comment section are tool's gold. I've been wondering where to start expanding my toolkit. Now I have a lot of great stuff to learn!
lardissone 13y ago
Awesome tools. Also interesting color scheme! Can you share this?
kbenson 13y ago
This approach reminded me quite a bit of what Mojo::DOM and Mojo::DOM:CSS modules give you, and the ojo command line tool for perl (called as perl -Mojo). Here's section 5 from that article rewritten: perl -MList::MoreUtils=zip -Mojo -E 'say g("http://en.wikipedia.org/wiki/List_of_countries_and_territories_by_border/area_ratio")->dom->find("table.wikitable http://en.wikipedia.org/wiki/List_of_countries_and_territori... > tr:not(:first-child)")->pluck(sub{ j { country=>$_->find("td")->[1]->text, border=>$_->find("td")->[2]->text, surface=>$_->find("td")->[3]->text, ratio=>$_->find("td")->[4]->text } })' | head {"ratio":"7.2727273","surface":"0.44","country":"Vatican City","border":"3.2"} {"ratio":"2.2000000","surface":"2","country":"Monaco","border":"4.4"} {"ratio":"0.6393443","surface":"61","country":"San Marino","border":"39"} {"ratio":"0.4750000","surface":"160","country":"Liechtenstein","border":"76"} {"ratio":"0.3000000","surface":"34","country":"Sint Maarten (Netherlands)","border":"10.2"} {"ratio":"0.2570513","surface":"468","country":"Andorra","border":"120.3"} {"ratio":"0.2000000","surface":"6","country":"Gibraltar (United Kingdom)","border":"1.2"} {"ratio":"0.1888889","surface":"54","country":"Saint Martin (France)","border":"10.2"} {"ratio":"0.1388244","surface":"2586","country":"Luxembourg","border":"359"} {"ratio":"0.0749196","surface":"6220","country":"Palestinian territories","border":"466"} And here's the original command: curl -s 'http://en.wikipedia.org/wiki/List_of_countries_and_territories_by_border/area_ratio' http://en.wikipedia.org/wiki/List_of_countries_and_territori... | scrape -be 'table.wikitable > tr:not(:first-child)' | xml2json | jq -c '.html.body.tr[] | {country: .td[1][], border: .td[2][], surface: .td[3][], ratio: .td[4][]}' | head Fairly comparable, but there's a whole world of Perl modules for me to pull in and use with the first one.
vram22 13y ago
Good thread. This article may be useful for those who want to learn to write Linux or Unix command-line utilities in C: Developing a Linux command-line utility: http://www.ibm.com/developerworks/library/l-clutil/ http://www.ibm.com/developerworks/library/l-clutil/
mikeme123 13y ago
SmallR: http://github.com/religa/smallR http://github.com/religa/smallR seems to provide quite a bit of flexibility with only few keystrokes needed.

346 points by robdoherty2 13y ago

> Ideally, json2csv would have added the header. You can raise an issue (https://github.com/jehiah/json2csv https://github.com/jehiah/json2csv) or just fork and add a flag to include a header line

daniel-levin 13y ago
https://github.com/daniel-levin/json2csv https://github.com/daniel-levin/json2csv Obligatory here-I-did-it
- jeroenjanssens 13y ago
  Nice one!
- daniel-levin 13y ago
  Update: now merged with the mainline repo

harrytuttle 13y ago

Nice tools. You can actually do most of that with vanilla PowerShell and Excel believe it or not but it's much fuglier and you spend most of your way working around edge cases. The only thing that scares me about this though is that JSON is a terrible format for storing numbers in. There is no way of specifying a decimal type for example so aggregations and calculations have no implied precision past floating point values.

the_cat_kittles 13y ago

Some of these are cool, but I don't understand the preference for command line tools over writing, say, a short python script in vim and then executing it. I see the two methods achieving the same results with about the same amount of text, except with a script, its savable and much easier to type.

TeMPOraL 13y ago
It's equivalent if you look at the command line as a (very popular) REPL, and at the tools as functions.
gnaritas 13y ago
> except with a script, its savable and much easier to type. That's an odd thing to say as Python scripts are no more savable than shell scripts. Nor are Python scripts any easier to type than shell scripts. Also, as shell scripts tend to pipe commands together, it's multiple process and thus multiple core by default, unlike Python.
- the_cat_kittles 13y ago
  I wasn't thinking of making them a shell script, I was thinking of using them directly from the prompt- that makes sense. Come to think of it, I should be writing more shell scripts instead of retyping stuff into the prompt.
  - goldfeld 13y ago
    Yeah, that's like saying Python sucks for scripting because all your work in the REPL won't be saved. Goes both ways.
    - zokier 13y ago
      That's why ipython is so nice :)
  - williamcotton 13y ago
    Or better yet, use make.
- reustle 13y ago
  I could be wrong, but doesn't each piped command wait for the previous one to finish (not multi-core)?

dergachev 13y ago

These are great! I wrote a quick ruby script that converts one or more CSV files into an SQLITE db file, so you can easily query them. https://github.com/dergachev/csv2sqlite https://github.com/dergachev/csv2sqlite

aw3c2 13y ago
No (real) need for a script, you can import in the sqlite3 commandline. First make the table, then set a .separator, then .import the file. I am not sure how performant it is though, so creating transactions outside might be better in some cases.
- kevnin 13y ago
  Depends on how often you have to do this... if it saves time, I'm in
- mkiwala 13y ago
  The sqlite3 .import command does not handle quoted csv values. "Doe, John", 1234 Pine St, Springfield That would be imported as 4 fields, not 3.
  - SQLite 13y ago
    Used to be true. But recent versions of SQLite fix this.
- dergachev 13y ago
  csv2sqlite actually generates the table definitions for you, so I think that's a big improvement when you just downloaded a CSV file with 30 columns in it and need to quickly verify it or extract some data from it. Also supports parsing multiple CSV files at once, so you can easily do joins.

th0ma5 13y ago

The ability to script the R stuff with pipes and whatnot is very welcome to me. I tried using R as a part of a system of normalizing package installs across environments. It was sort of kludgey but ultimately worked okay. Setting up the environment as the first part of any script you push into R is necessary, and that makes sense. Python and Java seem to have more support for this kind of thing (environment swapping) on the command line, or at least perhaps I wasn't looking the right places perhaps for R. It has been a year since I've messed with this though.

cgrubb 13y ago

Nice tools. I see some stuff I might add to my own personal repo: https://github.com/clarkgrubb/data-tools https://github.com/clarkgrubb/data-tools Would be nice if some of these tools became standard at the command line. I don't know about "sample" though, since that can easily be implemented in awk: awk 'rand() < 0.2' /etc/passwd

jeroenjanssens 13y ago
That's a nice collection of tools you have there. I'll add it to the post.

gav 13y ago

If you are working with XML, XMLStarlet is invaluable: http://xmlstar.sourceforge.net/ http://xmlstar.sourceforge.net/

bcantrill 13y ago

Glad to see that we're not the only ones who are emphatic believers in applying the Unix philosophy to data science! Those interested in this may also be interested in Mark Cavage and Dave Pacheco's presentation last week at Surge on Manta[1], a new storage service that we have developed that draws very direct inspiration from Unix -- and allows you to use the tools mentioned by Jeroen across very large data sets. awk FTMFW! [1] http://us-east.manta.joyent.com/mark.cavage/public/surge2013_manta_final.pdf http://us-east.manta.joyent.com/mark.cavage/public/surge2013...

mathattack 13y ago
An open question - why do you prefer doing this in the command line versus via a scripting language like Python? I get the piping philosophy, but why one versus the other?
- robbles 13y ago
  Most of the well-know "unix-style" command-line tools, such as grep, sort, etc. actually have very high performance. Their relatively constrained use cases allow the authors to implement decent algorithms and optimizations (e.g. sort uses merge sort, grep uses all kinds of optimizations: http://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html http://lists.freebsd.org/pipermail/freebsd-current/2010-Augu...) In contrast, when you're building a custom pipeline in a high-level language, you're optimizing for simple solutions and are not likely to get better performance unless you hit an edge case where the standard tools do really poorly.
  - paulrosenzweig 13y ago
    Actually sort is even better than normal (purely in-memory) merge sort. It looks at available memory and writes out sorted files to merge. http://vkundeti.blogspot.com/2008/03/tech-algorithmic-details-of-unix-sort.html http://vkundeti.blogspot.com/2008/03/tech-algorithmic-detail...
    - robbles 13y ago

jeroenjanssens 13y ago

Suggestions are more than welcome. I'm currently keeping the post up-to-date with suggested tools and repositories.

dirtyvagabond 13y ago
nice post! i'd like to make a shameless plug for Drake, a 'make for data' workflow tool. https://github.com/Factual/drake https://github.com/Factual/drake
susi22 13y ago
http://www.hdfgroup.org/HDF5/doc/RM/Tools.html http://www.hdfgroup.org/HDF5/doc/RM/Tools.html http://www.hdfgroup.org/tools5desc.html#1 http://www.hdfgroup.org/tools5desc.html#1 The first thing I do when I deal with data which has more than --let's say-- 10,000 rows is to put it in HDF file format and work with that. Saves a ton of time while developing a script. I had a python script do a histogram and it ran ~15sec for a file with 100k rows. With converting it first to HDF it ran in ~0.5sec. The import in python is also much shorter (two lines). HDF is made for high performance numerical I/O. It's great and you can query several structures and even do slices of arrays on the command line (with h5tools). It's also widely used by Octave, Python, R, Matlab... And you don't have a drawback since you can just pipe it into existing command line tools with a h5dump. HDF5: http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Performance.pdf http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Performance.pdf http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Overview.pdf http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Overview.pdf
- twistedpair 13y ago
  Wow, those benchmarks are old. 195MHz test machine? Still, 15K rows is not much. Just did that 100K rows (read/sum) bit in Octave. Took about 1/4 second to extract and sum. I assume HDF rocks for much larger sets.
susi22 13y ago
This also really helpful (pity it's relatively unknown): https://github.com/dbro/csvquote https://github.com/dbro/csvquote And: http://en.wikipedia.org/wiki/GNU_Core_Utilities http://en.wikipedia.org/wiki/GNU_Core_Utilities (section "Text utilities") http://directory.fsf.org/wiki/Textutils http://directory.fsf.org/wiki/Textutils Run "$ info coreutils"

peterwwillis 13y ago

So weird... I didn't know about the csvkit I guess (or maybe it wasn't available for cygwin?), so I wrote my own a year ago. It supports field names (in addition to column numbers) so your data can move around and you'll still reference the right logical data.

michaelmior 13y ago
csvkit is awesome. I discovered it by chance a few months ago and I've found it to be really well-written. It correctly handles a lot of weird edge cases (e.g. newlines in the middle of a record) For the record, csvkit supports field names as well :)

revertts 13y ago

You should check out RecordStream. It has a lot of convenient processing built around JSON, and a lot of ways to get data to/from JSON. https://github.com/benbernard/RecordStream/tree/master/doc https://github.com/benbernard/RecordStream/tree/master/doc https://github.com/benbernard/RecordStream/blob/master/doc/RecordStreamStory.pod https://github.com/benbernard/RecordStream/blob/master/doc/R...

ffk 13y ago
I second this suggestion. It doesn't just slurp and serialize data, it also is capable of filtering, transforming, aggregating (+ stats) and pretty printing streams of json records.

deleted 13y ago

[deleted]

gerad 13y ago

I'm surprised that the post didn't call out the unix built-ins cut, sort, uniq, and cat. There's an amazing amount of data processing you can do with just those commands.

sturadnidge 13y ago
He linked another post that covers those http://www.gregreda.com/2013/07/15/unix-commands-for-data-science/ http://www.gregreda.com/2013/07/15/unix-commands-for-data-sc...

mjn 13y ago

I like Google's crush-tools, which works on delimited data (e.g. tab-delimited), a somewhat simpler and faster format than CSV. Lots of the built-in Unix tools also work on delimited data (cut, join, sort, etc.), but crush-tools fills in some gaps, like being able to do a 'uniq' on specific fields, do outer joins, sum columns, etc.: https://code.google.com/p/crush-tools/ https://code.google.com/p/crush-tools/

sturadnidge 13y ago
They look interesting, thanks! Any experience using them over GB or TB datasets?
- mjn 13y ago
  I've used them on tens-of-GB datasets (not TB), and they're quite fast except for those implemented in Perl, which is kind of a hidden gotcha. For example, calcfield is barely usable, because it loops a perl expression over every record in the file. But things like funiq and cutfield are fast, at least as fast as the traditional Unix tools. And if you have pre-sorted data, aggregate2 is a nice aggregation tool for larger-than-memory datasets.
  - sliverstorm 13y ago
    That's interesting. I've written Perl tools that apply pattern-matching to several-GB flat files, and while it was horrifyingly slow at first, I was able to get the performance down to a minute in the average case. Honestly the whole time I think I/O was a greater limiting factor than Perl's processing speed.

aficionado 13y ago

A very powerful command line tool for Machine Learning: http://bigmler.readthedocs.org/en/latest/ http://bigmler.readthedocs.org/en/latest/ Create a predictive model is as simple as: bigmler --train < data.csv or create a predictive model for Bitcoin volume taking online data from Quandl and parsing it with jq in just one line. curl --silent "http://www.quandl.com/api/v1/datasets/BITCOIN/BTCDEEUR.json" http://www.quandl.com/api/v1/datasets/BITCOIN/BTCDEEUR.json" | jq -c ".data" | bigmler --train --field-attributes <(count=0; for i in `curl --silent "http://www.quandl.com/api/v1/datasets/BITCOIN/BTCDEEUR.json" http://www.quandl.com/api/v1/datasets/BITCOIN/BTCDEEUR.json" | jq -c ".column_names[]"`; do echo "$count, $i"; count=$[$count+1]; done) --name bitcoin More info here: http://blog.bigml.com/2013/01/31/fly-your-ml-cloud-like-a-kite-with-bigmler-the-command-line-tool-for-machine-learning/ http://blog.bigml.com/2013/01/31/fly-your-ml-cloud-like-a-ki...

diziet 13y ago
Hmm -- the model did not seem to spot something a simple linear regression did.
- aficionado 13y ago
  In this particular case, I think that you are right. But isn't it powerful to have it at the command line level? It's great to quickly create models as you can also use them to make predictions at the command line level.
jeroenjanssens 13y ago
Impressive. I'll add it to the list.

dima55 13y ago

Here's another that I use all the time: https://github.com/dkogan/feedgnuplot https://github.com/dkogan/feedgnuplot It reads data on stdin, and makes plots. More or less anything that gnuplot can do is supported, which is quite a lot. Realtime plotting of streaming data is supported as well. This is also in Debian, and is apt-gettable. Disclaimer: I am the author

paulgb 13y ago

Great list. Shameless self-plug: I wrote a tool similar to sample available on pip as subsample. It supports the approximate sample algorithm that yours does as well as reservoir sampling and an exact two-pass algorithm that runs in O(1) space. Docs: https://github.com/paulgb/subsample https://github.com/paulgb/subsample

mrcactu5 13y ago

I feel like a dumbass. The two great data science tools I learned for command line are grep and sed :-)

zmmmmm 13y ago

Worth noting that the Rio tool does not seem to account for R's concurrency / race conditions when running RScript. Basically, if you run it a lot in parallel you will get random failures. Although I am sure it is not exactly meant for high concurrency situations, it would be good if it accounted for this.

helloTree 13y ago

Maybe I should take a look at awk/sed but I never found its language orthogonal. For simple scripts I mostly use Python or Haskell via something like ghc -e 'interact ((++"\n") . show . length . lines)' < file EDIT: I know that it's the same like "wc -l" but using that approach I can solve also problems that may not be well suited for awk/sed. Or maybe I just have to see some convincing one-liners.

fluidcruft 13y ago

There is a command-line optimizer. I always forget it's name and can never find it. It is very cool for doing simplex optimizations etc. I will try yet again to find it and edit.

tomrod 13y ago

This post and comment section are tool's gold. I've been wondering where to start expanding my toolkit. Now I have a lot of great stuff to learn!

lardissone 13y ago

Awesome tools. Also interesting color scheme! Can you share this?

kbenson 13y ago

This approach reminded me quite a bit of what Mojo::DOM and Mojo::DOM:CSS modules give you, and the ojo command line tool for perl (called as perl -Mojo). Here's section 5 from that article rewritten: perl -MList::MoreUtils=zip -Mojo -E 'say g("http://en.wikipedia.org/wiki/List_of_countries_and_territories_by_border/area_ratio")->dom->find("table.wikitable http://en.wikipedia.org/wiki/List_of_countries_and_territori... > tr:not(:first-child)")->pluck(sub{ j { country=>$_->find("td")->[1]->text, border=>$_->find("td")->[2]->text, surface=>$_->find("td")->[3]->text, ratio=>$_->find("td")->[4]->text } })' | head {"ratio":"7.2727273","surface":"0.44","country":"Vatican City","border":"3.2"} {"ratio":"2.2000000","surface":"2","country":"Monaco","border":"4.4"} {"ratio":"0.6393443","surface":"61","country":"San Marino","border":"39"} {"ratio":"0.4750000","surface":"160","country":"Liechtenstein","border":"76"} {"ratio":"0.3000000","surface":"34","country":"Sint Maarten (Netherlands)","border":"10.2"} {"ratio":"0.2570513","surface":"468","country":"Andorra","border":"120.3"} {"ratio":"0.2000000","surface":"6","country":"Gibraltar (United Kingdom)","border":"1.2"} {"ratio":"0.1888889","surface":"54","country":"Saint Martin (France)","border":"10.2"} {"ratio":"0.1388244","surface":"2586","country":"Luxembourg","border":"359"} {"ratio":"0.0749196","surface":"6220","country":"Palestinian territories","border":"466"} And here's the original command: curl -s 'http://en.wikipedia.org/wiki/List_of_countries_and_territories_by_border/area_ratio' http://en.wikipedia.org/wiki/List_of_countries_and_territori... | scrape -be 'table.wikitable > tr:not(:first-child)' | xml2json | jq -c '.html.body.tr[] | {country: .td[1][], border: .td[2][], surface: .td[3][], ratio: .td[4][]}' | head Fairly comparable, but there's a whole world of Perl modules for me to pull in and use with the first one.

vram22 13y ago

Good thread. This article may be useful for those who want to learn to write Linux or Unix command-line utilities in C: Developing a Linux command-line utility: http://www.ibm.com/developerworks/library/l-clutil/ http://www.ibm.com/developerworks/library/l-clutil/

mikeme123 13y ago

SmallR: http://github.com/religa/smallR http://github.com/religa/smallR seems to provide quite a bit of flexibility with only few keystrokes needed.