6 ms·

Data Science at the Command Line

215 points by kawera 9y ago

rdudekul 9y ago
As a developer I have not paid much attention to the power of command line tools, to get things done. This book is a great resource that uses docker to create simple command line utilities to implement useful Data Science functionality.
jeroenjanssens 9y ago
Believe it or not, it's partly thanks to you, HN, that I wrote this book in 2014 in the first place [1]! It's humbling to see it listed here again now that the text has become available under a CC BY-ND license. Thank you for your help in spreading the word. All this attention (read: likes, shares, and page views) is making me wonder whether it's worthwhile to write an update (or even a second edition). What do you think? What would you like to see changed or added? [1] https://news.ycombinator.com/item?id=6412190 https://news.ycombinator.com/item?id=6412190
- utefan001 9y ago
  sort | uniq -c | sort -n example in chapter 5 is something I use a lot. Do you typically use Mac or Linux? The way Mac does threading made sort 16 times slower on 1GB+ size files compared to my $300 old Lenovo laptop running Ubuntu.
  - vondur 9y ago
    Have you tried installing it from homebrew? brew install coreutils This assumes you have brew installed.
    - utefan001 9y ago
      Yea, same issue with gnu coreutils. Mac uses a fairness threading algorithm, causing lots of time to be wasted deciding whose turn it is. https://stackoverflow.com/questions/28888719/multi-threaded-c-program-much-slower-in-os-x-than-linux https://stackoverflow.com/questions/28888719/multi-threaded-...
minimaxir 9y ago
While the efficiency of a command line is always sexy, but for data science in particular, where reproducibility is important and bugs are subtle and often don't cause a terminal error, it is worth it to sacrifice a little bit of code efficiency for code clarity in the long run by using an IDE/Notebook.
- rabidrat 9y ago
  Is there a tool that would create a notebook from the command-line?
  - marmaduke 9y ago
    Jupyter notebook has Bash kernels
    - jeroenjanssens 9y ago
      http://jeroenjanssens.com/2015/02/19/ibash-notebook.html http://jeroenjanssens.com/2015/02/19/ibash-notebook.html
    - adeelk93 9y ago
      You can also do the same in RMarkdown, if you're more of an R user
  - eivarv 9y ago
    I think that depends on what you mean; Jupyter lets you run magic[0] commands (run bash commands in subprocesses), and there's also a bash kernel[1]. [0]: https://blog.dominodatalab.com/lesser-known-ways-of-using-notebooks/ https://blog.dominodatalab.com/lesser-known-ways-of-using-no... [1]: https://github.com/takluyver/bash_kernel https://github.com/takluyver/bash_kernel
- achompas 9y ago
  Notebooks are not much better than copy-pasting from a notepad or editor into an interpreter. They’re great for reports, but dangerous for presenting the illusion of reproducibility. At best you’re constantly restarting your kernel and clearing output. More likely, output from cell #7 has modified output [138] but you haven’t updated the chart produced in cell #17 (or some similar craziness). Not much better than programming with GOTOs. “But they’re great for reporting and visualization!” you might say. If you’re building any report of value, though, it will influence important decisions. That’s the reason your code shouldn't live in a notebook. It should be in a library, covered by unit tests, so that those decisions aren’t based on faulty logic.
jimnotgym 9y ago
Is there a good way to get this onto an e-reader, am I missing something obvious?
- lonriesberg 9y ago
  You can buy a Kindle version from Amazon: https://www.amazon.com/Data-Science-Command-Line-Time-Tested/dp/1491947853 https://www.amazon.com/Data-Science-Command-Line-Time-Tested...
jhoechtl 9y ago
An indispensable tool for the command line is miller https://github.com/johnkerl/miller https://github.com/johnkerl/miller
fnl 9y ago
Given the number of tools around, and as this book promotes Drake, has anyone got "comparative experience" with some of the following tools: Cookiecutter: https://drivendata.github.io/cookiecutter-data-science https://drivendata.github.io/cookiecutter-data-science DataVersionControl: https://dataversioncontrol.com/ https://dataversioncontrol.com/ Drake: https://github.com/Factual/drake https://github.com/Factual/drake Luigi: https://github.com/spotify/luigi https://github.com/spotify/luigi Pachyderm: http://www.pachyderm.io/ http://www.pachyderm.io/ Sacred: https://github.com/IDSIA/sacred https://github.com/IDSIA/sacred They all focus in slightly different ways on the issue of managing data science/machine learning workflows, so I wonder if someone has a clear preference for one of those over any another. EDIT: added Luigi
- fnl 9y ago
  To amend, why I'm even bringing this up, what worries me about Drake is this: https://github.com/Factual/drake/pulse/monthly https://github.com/Factual/drake/pulse/monthly Its GitHub pulse is - dead; For two years now. Makes me think one of the other projects listed might be better choices.
  - bringtheaction 9y ago
    I was about to say maybe it's finished but it has 70 open issues so maybe it was just abandoned? It is at version 1.0.3 though so it could be that it's considered finished. Seems strange to leave the issues open if it was though.
    - fnl 9y ago
      Maybe so. Yet, I can't find any features in Drake that I don't get with Make, too - in fact, it looks to be rather the opposite. Indeed, for some of the tools I listed, they barely have any more functionality than I'd get out of Make & Git alone. And for Make, I'm pretty sure development & support will stick around for a few more years... To me, only Luigi (Hadoop integration), Pachyderm (containers, production deployments in the Enterprise version) and Sacred (Python & TensorFlow integration) really stick out as differentiating themselves. But maybe I'm overlooking something?
bobivl 9y ago
I also used bash scripts a lot to get quick insights from csv files. Someday I realized that these are mostly sql queries that I encoded into complex scripts. For the sake of trying, I implemented a simple sql to bash transpiler that takes a sql query and returns a bash one-liner that you can execute on csv file(s). Give it a try: http://bigbash.it http://bigbash.it

215 points by kawera 9y ago

As a developer I have not paid much attention to the power of command line tools, to get things done. This book is a great resource that uses docker to create simple command line utilities to implement useful Data Science functionality.

jeroenjanssens 9y ago

Believe it or not, it's partly thanks to you, HN, that I wrote this book in 2014 in the first place [1]! It's humbling to see it listed here again now that the text has become available under a CC BY-ND license. Thank you for your help in spreading the word. All this attention (read: likes, shares, and page views) is making me wonder whether it's worthwhile to write an update (or even a second edition). What do you think? What would you like to see changed or added? [1] https://news.ycombinator.com/item?id=6412190 https://news.ycombinator.com/item?id=6412190

utefan001 9y ago
sort | uniq -c | sort -n example in chapter 5 is something I use a lot. Do you typically use Mac or Linux? The way Mac does threading made sort 16 times slower on 1GB+ size files compared to my $300 old Lenovo laptop running Ubuntu.
- vondur 9y ago
  Have you tried installing it from homebrew? brew install coreutils This assumes you have brew installed.
  - utefan001 9y ago
    Yea, same issue with gnu coreutils. Mac uses a fairness threading algorithm, causing lots of time to be wasted deciding whose turn it is. https://stackoverflow.com/questions/28888719/multi-threaded-c-program-much-slower-in-os-x-than-linux https://stackoverflow.com/questions/28888719/multi-threaded-...

minimaxir 9y ago

While the efficiency of a command line is always sexy, but for data science in particular, where reproducibility is important and bugs are subtle and often don't cause a terminal error, it is worth it to sacrifice a little bit of code efficiency for code clarity in the long run by using an IDE/Notebook.

rabidrat 9y ago
Is there a tool that would create a notebook from the command-line?
- marmaduke 9y ago
  Jupyter notebook has Bash kernels
  - jeroenjanssens 9y ago
    http://jeroenjanssens.com/2015/02/19/ibash-notebook.html http://jeroenjanssens.com/2015/02/19/ibash-notebook.html
  - adeelk93 9y ago
    You can also do the same in RMarkdown, if you're more of an R user
- eivarv 9y ago
  I think that depends on what you mean; Jupyter lets you run magic[0] commands (run bash commands in subprocesses), and there's also a bash kernel[1]. [0]: https://blog.dominodatalab.com/lesser-known-ways-of-using-notebooks/ https://blog.dominodatalab.com/lesser-known-ways-of-using-no... [1]: https://github.com/takluyver/bash_kernel https://github.com/takluyver/bash_kernel
achompas 9y ago
Notebooks are not much better than copy-pasting from a notepad or editor into an interpreter. They’re great for reports, but dangerous for presenting the illusion of reproducibility. At best you’re constantly restarting your kernel and clearing output. More likely, output from cell #7 has modified output [138] but you haven’t updated the chart produced in cell #17 (or some similar craziness). Not much better than programming with GOTOs. “But they’re great for reporting and visualization!” you might say. If you’re building any report of value, though, it will influence important decisions. That’s the reason your code shouldn't live in a notebook. It should be in a library, covered by unit tests, so that those decisions aren’t based on faulty logic.

jimnotgym 9y ago

Is there a good way to get this onto an e-reader, am I missing something obvious?

lonriesberg 9y ago
You can buy a Kindle version from Amazon: https://www.amazon.com/Data-Science-Command-Line-Time-Tested/dp/1491947853 https://www.amazon.com/Data-Science-Command-Line-Time-Tested...

jhoechtl 9y ago

An indispensable tool for the command line is miller https://github.com/johnkerl/miller https://github.com/johnkerl/miller

fnl 9y ago

Given the number of tools around, and as this book promotes Drake, has anyone got "comparative experience" with some of the following tools: Cookiecutter: https://drivendata.github.io/cookiecutter-data-science https://drivendata.github.io/cookiecutter-data-science DataVersionControl: https://dataversioncontrol.com/ https://dataversioncontrol.com/ Drake: https://github.com/Factual/drake https://github.com/Factual/drake Luigi: https://github.com/spotify/luigi https://github.com/spotify/luigi Pachyderm: http://www.pachyderm.io/ http://www.pachyderm.io/ Sacred: https://github.com/IDSIA/sacred https://github.com/IDSIA/sacred They all focus in slightly different ways on the issue of managing data science/machine learning workflows, so I wonder if someone has a clear preference for one of those over any another. EDIT: added Luigi

fnl 9y ago
To amend, why I'm even bringing this up, what worries me about Drake is this: https://github.com/Factual/drake/pulse/monthly https://github.com/Factual/drake/pulse/monthly Its GitHub pulse is - dead; For two years now. Makes me think one of the other projects listed might be better choices.
- bringtheaction 9y ago
  I was about to say maybe it's finished but it has 70 open issues so maybe it was just abandoned? It is at version 1.0.3 though so it could be that it's considered finished. Seems strange to leave the issues open if it was though.
  - fnl 9y ago
    Maybe so. Yet, I can't find any features in Drake that I don't get with Make, too - in fact, it looks to be rather the opposite. Indeed, for some of the tools I listed, they barely have any more functionality than I'd get out of Make & Git alone. And for Make, I'm pretty sure development & support will stick around for a few more years... To me, only Luigi (Hadoop integration), Pachyderm (containers, production deployments in the Enterprise version) and Sacred (Python & TensorFlow integration) really stick out as differentiating themselves. But maybe I'm overlooking something?

bobivl 9y ago

I also used bash scripts a lot to get quick insights from csv files. Someday I realized that these are mostly sql queries that I encoded into complex scripts. For the sake of trying, I implemented a simple sql to bash transpiler that takes a sql query and returns a bash one-liner that you can execute on csv file(s). Give it a try: http://bigbash.it http://bigbash.it