3 ms·

In this case - how do you share results between team members? You can generate files with code and results (numbers, plots) - but for that the easiest step is

• mlthoughts2018 8y ago
  > "In this case - how do you share results between team members?" It depends on what "results" means. Here are some examples. - Say the result is a set of model parameters from a trained model, and accompanying diagnostic info, like precision/recall, ROC graph, accuracy, standard errors, and so forth. In that case, we would probably write the training program in such a way as to store that data into a database table accompanied by some type of model ID that can uniquely track back to the particular git commit of the repo when it was at the state of the experiment. Downstream libraries that we write as "production quality" code could be used to automatically read from the results database and produce plots locally or displayed in some web service. - Say the result is a trained model that fulfills some task, like ranking search results or detecting faces. Then in addition to basic diagnostic metrics like above, we might also provide a web service interface that allows you to run the service on example data for diagnostics. The continuous integration tooling would execute some form of acceptance tests and collate the results so you get trackable feedback about accuracy, and an immutable copy of the model (with an ID that maps back to the git commit) might be uploaded to a remote storage server. While these items require a greater up-front investment for the first few times you do it, it quickly pays off by providing data provenance over exactly how to reproduce a given result, and it also provides more functional display interfaces, rather than mixing them together like a notebook does. Actually that is probably the main point: a notebook takes multiple concerns that should be treated separately and tries to intermix them in one interface. It sounds like a great promise, but in reality it's a bad idea almost always. It's kind of like writing a tic-tac-toe implementation where the source code of the GUI elements directly contains the source code of the game's rule logic or current game state. The sort of mixed-concerns problem that an idea like Model-View-Controller was meant to solve. Only with notebooks it doesn't just mix units of implementation logic with units of display logic -- it also mixes units of configuration (global variables, file paths, parameter settings), units of the dependencies (implicit reliance on the underlying software setup and correctly versioned dependencies) and units of the system overall (like if you needed TensorFlow built from source with optimization flags, or you modified the code for an upgraded version of some library). It's better to state a clear goal, like, "The team needs to take the following type of result / summary / display and share it with others." And then apply standard software craftsmanship like you would for any other software problem. Nobody would say, "well we need to share results with teammates, so we should put embedded plotting functions inside this module that manages the database connection." They're clearly different concerns. Yet both types of code would be put into the same source code if using a notebook, highly coupling things that have no business being coupled that way. > "development -> when working with data virtually all first approaches will go to trash anyway" I agree with this, which is advice going all the way back to The Mythical Man-Month. But that doesn't mean you should omit obvious quality practices when designing that first system. The point of the "plan to throw one away" thinking is that you architectural understanding of the problem will evolve as you try to solve it. And in fact, the better code hygiene you use on that first solution, the better you can distinguish between parts of the solution that are architecturally bad versus parts that are actually good ideas but are just being implemented in a messy / careless way.