4 ms·
Even better would be to just use standard input/output. Like so (assuming pandas accepts ordinary file handles for reading and writing CSVs; I have very limite
by yellowapple 8y ago
Even better would be to just use standard input/output. Like so (assuming pandas accepts ordinary file handles for reading and writing CSVs; I have very limited experience with it, and I tend to avoid Python in general):
#!/usr/bin/env python
# process-csv.py
import pandas
def expensive_operation(data):
# TODO
return data
input_data = pandas.read_csv(sys.stdin)
results_data = expensive_operation(input_data)
results_data.to_csv(sys.stdout)
Or maybe even shorter (I think this is valid Python?):
#!/usr/bin/env python
import pandas
def expensive_operation(data):
# TODO
return data
expensive_operation(pandas.read_csv(sys.stdin)).to_csv(sys.stdout)
And then use it like so:
$ python process-csv.py <input.csv >results.csv
Or like so:
$ csv-generating-command | python process-csv.py | csv-consuming-command
You could even do a quick
$ chmod +x process-csv.py
And then call it directly
$ ./process-csv.py <in.csv >out.csv
Or copy it to somewhere in your $PATH
$ cp process-csv.py ~/bin/process-csv
At which point you can run from anywhere
$ cd /literally/anywhere/else/
$ sudo pip install csvcat # or something
$ csvcat *.csv | process-csv >~/Documents/results-201904021616.csv
Of course, we could get even crazier with a custom Pip package or whatever the actual terminology is (egg?) and do all the setup.py and requirements.txt and whatnot that Python packaging entails and yadda yadda yadda, but that's probably overkill for a one-off script. Point is: your fancy Macbook has a full-blown actual UNIX™ on it, so might as well put it to good use :)
- IanCal 8y agoThe point is not about how to pass data in, but how to share the data.
- yellowapple 8y agoI mean, the point actually raised in the article is about making the code agnostic to where that data lives. Leveraging plain ol' Unix pipes is pretty hard to beat in that regard. The article doesn't really provide much of an alternative to the actual sharing of the data itself beyond just sticking it in Dropbox. And of course, an stdio-based approach works perfectly fine for Dropbox: $ cd ~/Dropbox/csv/data/folder && process-csv <in.csv >out.csv Or for S3: $ s3fs my-bucket ~/mnt/my-bucket # macOS has FUSE support, right? $ cd ~/mnt/my-bucket/csv/data/folder $ process-csv <in.csv >out.csv Or for pretty much any other filesharing service that provides a folder on the filesystem, really.
- chthonicdaemon 8y ago(author of post) I am a huge fan of this approach and use it extensively in my own code. This natural commandline interaction is completely alien to my students. They are barely holding on with trying to learn Python in the notebook environment, telling them that the solution to their problem is to learn another language has not woked well for me.
- yellowapple 8y agoGotcha. Given that Jupyter feels pretty alien to me coming from a command-line-heavy background, I can understand the stubbornness :) In that case, I'd probably more readily recommend just adding a *.csv line to .gitignore and letting students add/remove files accordingly (unless the lesson is specifically about how configuration files work, of course, in which case by all means your approach is reasonable).
- chthonicdaemon 8y agoAdding the .csv files to .gitignore only avoids the one issue of having checking in data files. How do collaborators ensure they’re working on the same data files? I am aiming to minimise the manual steps involved in running the latest version of the code on the latest version of the input data. With my system you just pull the latest code and run it.
- yellowapple 8y agoYou can have that with gitignored local data files, too, though: since git is ignoring the data, a simple "git pull" will indeed ignore what's already in there. Meanwhile, the answer to "How do collaborators ensure they're working on the same data files?" would be "Drag and drop those files into the source repo from whichever means they wanna use to share them.". Yet another way to skin this particular cat might be to write a script as a drag-and-drop target. I don't think Jupyter supports this, so your students would have to venture beyond it a bit, but your Windows-using students for example can drag files onto actual .py files and those dropped files will show up in sys.argv (I think macOS requires the extra step of using py2app, but that might be a good opportunity for a py2app/py2exe lesson). I'd post sample code like in my other comments, but I'm on my phone at the moment.