7 ms·
It's worth nothing that the article does not discuss the reproducibility of results (e.g. with a Jupyter Notebook) and the implementation of said results (e.g.
by minimaxir 9y ago
It's worth nothing that the article does not discuss the reproducibility of results (e.g. with a Jupyter Notebook) and the implementation of said results (e.g. deploying/validating models), both of which matter much more than any code style conventions for data-related projects.
- QuercusMax 9y agoThis 100%. I have read postmortems of some "significant discoveries" which have turned out to only be reproducible on a particular build or software on a single analyst's machine. Or not at all. One "result" turned out to hinge on the iteration order of python dictionaries. And this definitely didn't happen after those working on tools to help said analysts make reproducible results encouraged the analysts to use said tools... No, that would be crazy.
- stornetn 9y agoI completely agree. Almost all of this article appears to have little to do with being a Data Scientist in particular and more to do with some good practices for writing code in general. So the advice itself is fine, just not what I was hoping for based on the title. Reproducibility, like you say, however, is something that is an issue far more particular to data science, and worth more serious consideration and discussion. Hand-in-hand with that is shareability. I'm a fan of what airbnb has open sourced to address some of those issues in their knowledge repo project: https://github.com/airbnb/knowledge-repo https://github.com/airbnb/knowledge-repo
- arcanus 9y ago^ this. I cannot describe how many times I've been shown results and when asking how to reproduce them, after several notes (and sometimes complaints to higher ups) I eventually get a series of command line arguments or a barely functioning R-script. These conclusions are too important to be so sloppily produced. We need verification, validation and uncertainty quantification for any result provided to decision makers.
- cjslep 9y agoLearning uncertainty propagation in engineering statistics was one of those concepts that seemed to be immediately useful and have far more implications than any textbooks emphasized. I was very happy to have that background when I took part developing statistical models used for wind hazard analysis on nuclear powerplants in my first job out of college.
- joe_the_user 9y agoTo put it cynically, if you other people can reproduce your results, you might not be demonstrating that you are x10 more productive than them. Which is to say I think starting with the idea that you're aim for x10 is pernicious and tends to create dysfunctional teams. The claim some developers in some circumstances are ten time more productive than others may or may not be true but software development needs processes whose goal is to help an entire team rather than helping an individual to that "level".
- cheez 9y agoThis is why the only time I use 10x is "your team will become 10x more productive."
- webnrrd2k 9y agoYes, it's better to have an x10 team, rather than an x10 developer. Developers should strive to better themselves, but it's important not to fool yourself, too. Having a strong team is almost always better from a business point of view.
- mdekkers 9y agoIndeed. I have seen several teams with a bunch of (self-styled) "10x" devs, and found that the productivity and quality of the team decreases in direct proportion to the amount of "10x" devs on the team. I shun "rockstar" and "10x" (and whatever other bullshit moniker they will come up with next) team members. Give me a group of smart people that gel well together, and are highly self-confident without egos getting into the way, and we can move mountains.
- flaviuspopan 9y agoNothing feels cleaner than storing everything (notebook, raw data, cleansed data, misc scripts, etc.) in a docker image when you're finished with the project. Data science and docker are meant to be besties.
- Angostura 9y agoPotentially problematic for those who want to check your findings in 30 years time?
- gilbetron 9y agoIs there a good solution to that problem, though? (Serious question). I recently did a laptop refresh and am using it as an opportunity to solidify my approach to ML development, and would love to hear if there is a good solution to long-term reproducibility. I'm currently leaning towards Docker, but maybe Vagrant or another "pure" VM approach is better...
- delazeur 9y agoCarve it onto stone tablets.
- gilbetron 9y agoI prefer gold.
- KirinDave 9y agoYou joke it, but it's a major problem that our tech for very stable WORM media has lagged behind demand. Our use of data has grown so much faster than our network capacity (and indeed, it seems like we're going to hit a series of physical laws and practical engineering constraints here). "Data has gravity" but the only way to "sustainable" hold a non-trivial volume of data for 20 years right now is to run a data center with a big dht that detects faults and replicates data.
- KirinDave 9y ago
- drvdevd 9y agoThus making you a 1x Data Scientist because your result can only be demonstrated once? ;)
- NumberCruncher 9y agoIn my experience if you develop a "data science pipeline" forcing the data scientist to build - reproducible - validated - back-tested - easy to deploy models, they are going to hate it. It just kills the fun and/or makes obvious if they made a mistake.
- CJefferson 9y agoI blame software. I don't understand why we couldn't have some system, perhaps using strace and friends, which tracks everything I ever do, and how every file was created. Then I could just say "how did I make X?"
- carlmr 9y agoMake it happen!
- gipp 9y agoSo we should sacrifice all the things that actually make a Data Scientist's work valuable in the name of fun and obscuring mistakes? Fun I almost get, obviously good for productivity (though I think you'd really be sacrificing productive output for non-productive output), but I just don't get where you're even coming from with the "making mistakes more obvious" angle.
- carlmr 9y agoI think he was being facetious. Of course we need all these things, but data science right now is still not that mature I guess.
- deleted 9y ago[deleted]
- princesspea 9y agoHey thanks for the comment! I'm the author of the talk-turned-post :-) You are completely correct about reproducibility being super important in data science workflows. While I did mention it in the post (and in the talk the post was based on), I mentioned it as a part of version control tools. That said, I think it's not something that is focused on enough (obviously I'm guilty of that too) so I plan on doing a follow up post focused on reproducibility and the tools that can help you recreate your results. Kinda putting the "science" back in data science. Really I want an excuse to play around with tools like https://dataversioncontrol.com/ https://dataversioncontrol.com/ which looks super useful and I mentioned it in the post, but haven't had a chance to use.
- dmpetrov 9y ago'Kinda putting the "science" back in data science. ' Exactly! This is the primary goal in DVC project.