4 ms·

Why in the world would you do any of these things as opposed to making your program ask the filename from the user ? You've limited your program to working on e

by dusted 8y ago

Why in the world would you do any of these things as opposed to making your program ask the filename from the user ? You've limited your program to working on exactly one datafile.. If you just ask the user (any way is good, but program arguments required by a long shot), then you allow them to understand the files involved, and you get to run multiple versions of the program against multiple files, to compare their output, or process your data in parallel, or process different sets of data.. Sorry, not buying. Also, the whole post could have been written as "Don't hardcode filepaths, use a configuration file." but again, that's not what configuration files are for.. If you want to run a program on a specific set of files, you're better off wrapping the invocation in a script.

skohan 8y ago
The fact that this article is about jupyter notebooks, and apparently doesn't know that gitignore exists tells me that this is probably written by a data scientist who is in the process of "discovering" solutions to long-solved problems in software development.
- wodenokoto 8y ago
  For a notebook I would ask the user to change a filepath variable in one of the top cells. But I'm also a stupid data scientist. What do you propose as the long-solved solution?
  - pxndx 8y ago
    I'd propose reading data from stdin. Seeing this is about python, one could even consider using fileinput [0] to manage multiple files. [0]: https://docs.python.org/3/library/fileinput.html https://docs.python.org/3/library/fileinput.html
    - wodenokoto 8y ago
      It is a jupyter notebook, not a python script. Your proposal won't work. You generally don't open a notebook from the commandline, you open it from the notebook file picker inside your browser. You generally don't "run" a notebook, like you would a script. It is an interactive programming environment. https://github.com/jupyter/nbconvert/issues/681 https://github.com/jupyter/nbconvert/issues/681
- chthonicdaemon 8y ago
  (author of post here) It's more like someone who started by recommending my students use parametrised scripts, noticing that they will not and working through alternatives. My students are relatively familiar with the notebook interface, but not at all comfortable with writing a script which will accept command line arguments or read from standard in. PS: since I mention .gitignore explicitly in the post, why would you thing I don't know about .gitignore?
IanCal 8y ago
You are missing the use case here. I work with producing some analysis, and the writeup is absolutely tied to the data file used. I want to ensure these are available and tied to the code & output. However, they're too big for git. There are better solutions though. I've been using dvc.org recently.
theamk 8y ago
This is for Jupyter Notebook. For the cases when you start with huge dataset, and want to generate dozens graphs and tables on the screen, each changeable interactively without data reload. And leave python interpreter running, so you can run more code if you have more questions. Oh, and do all the hard work on the cloud machine with hundred of gigs of RAM, while displaying the data on your laptop over slow internet link. I have tried to implement those things by hand (intermediate state, HTML generation, shotgun plotting, pdb interaction), and it is a major pain. Notebooks work great in this situation. There are some downsides -- like inability to ask filenames from user -- but those could be easily bypassed, as this article shows (my solution will be a user-created symlink btw, not an ini file)
chthonicdaemon 8y ago
I am the author of this post. This post was written mostly for my students and is about a particular transition from a "single purpose" notebook where they have solved a problem on one file and trying to collaborate with me on their project. For the situation I show as an example, you wouldn't want to bring up a dialog box every time you run the cell as in most cases you're running it many times to get things right. Even when the code is working perfectly it isn't useful to pop up a dialog during a long-running analysis. The target audience of this post are not comfortable with running scripts from the commandline. I have tried teaching that approach and failed. This is a half-way point.
- dusted 8y ago
  Thank you for clarifying, I regret being overly harsh in my comment, your reason for writing this is perfectly legitimate and I simply lacked the imagination think of the scenario.

by dusted 8y ago

skohan 8y ago

The fact that this article is about jupyter notebooks, and apparently doesn't know that gitignore exists tells me that this is probably written by a data scientist who is in the process of "discovering" solutions to long-solved problems in software development.

wodenokoto 8y ago
For a notebook I would ask the user to change a filepath variable in one of the top cells. But I'm also a stupid data scientist. What do you propose as the long-solved solution?
- pxndx 8y ago
  I'd propose reading data from stdin. Seeing this is about python, one could even consider using fileinput [0] to manage multiple files. [0]: https://docs.python.org/3/library/fileinput.html https://docs.python.org/3/library/fileinput.html
  - wodenokoto 8y ago
    It is a jupyter notebook, not a python script. Your proposal won't work. You generally don't open a notebook from the commandline, you open it from the notebook file picker inside your browser. You generally don't "run" a notebook, like you would a script. It is an interactive programming environment. https://github.com/jupyter/nbconvert/issues/681 https://github.com/jupyter/nbconvert/issues/681
chthonicdaemon 8y ago
(author of post here) It's more like someone who started by recommending my students use parametrised scripts, noticing that they will not and working through alternatives. My students are relatively familiar with the notebook interface, but not at all comfortable with writing a script which will accept command line arguments or read from standard in. PS: since I mention .gitignore explicitly in the post, why would you thing I don't know about .gitignore?

IanCal 8y ago

You are missing the use case here. I work with producing some analysis, and the writeup is absolutely tied to the data file used. I want to ensure these are available and tied to the code & output. However, they're too big for git. There are better solutions though. I've been using dvc.org recently.

theamk 8y ago

This is for Jupyter Notebook. For the cases when you start with huge dataset, and want to generate dozens graphs and tables on the screen, each changeable interactively without data reload. And leave python interpreter running, so you can run more code if you have more questions. Oh, and do all the hard work on the cloud machine with hundred of gigs of RAM, while displaying the data on your laptop over slow internet link. I have tried to implement those things by hand (intermediate state, HTML generation, shotgun plotting, pdb interaction), and it is a major pain. Notebooks work great in this situation. There are some downsides -- like inability to ask filenames from user -- but those could be easily bypassed, as this article shows (my solution will be a user-created symlink btw, not an ini file)

chthonicdaemon 8y ago

I am the author of this post. This post was written mostly for my students and is about a particular transition from a "single purpose" notebook where they have solved a problem on one file and trying to collaborate with me on their project. For the situation I show as an example, you wouldn't want to bring up a dialog box every time you run the cell as in most cases you're running it many times to get things right. Even when the code is working perfectly it isn't useful to pop up a dialog during a long-running analysis. The target audience of this post are not comfortable with running scripts from the commandline. I have tried teaching that approach and failed. This is a half-way point.

dusted 8y ago
Thank you for clarifying, I regret being overly harsh in my comment, your reason for writing this is perfectly legitimate and I simply lacked the imagination think of the scenario.