6 ms·

Back in the day (3 years ago and earlier) at every company I was at we used the term 'productionization' to describe someone making a model aka a proof of conce

Back in the day (3 years ago and earlier) at every company I was at we used the term 'productionization' to describe someone making a model aka a proof of concept, and then someone else, a machine learning engineer or some kind of engineer rewriting it to work on a server. This process is horrible, and not just because it doubles the work, but because it introduces bugs. When the version up in the cloud does not work as intended, is it a bug in productionizing or is it in the original model? Fixing bugs in this space can take longer than the initial model development and the initial productionization. Many companies have failed over this. So what's the solution? In recent years the industry has turned to deployment over productionization. The idea is you deploy the model to the cloud directly. Both engineers and scientists work together on the process. The scientist defines what cells in the notebook get called for the final algorithm (as there are EDA / plotting cells and documentation cells too). The engineer sets up the amazon IO stuff, database login stuff, and monitoring services. The scientist works with them to create tests and what to monitor so they get notified if there is a problem with the service. No more mystery bugs. The model gets directly deployed, the work load is minimal, and it brings people together. The downside is often the engineers and scientists are on different teams, and sometimes companies will not let them merge for a while, so it becomes a telephone game instead of everyone feeling like they're on the same team working together. imo moving the scientist to the engineering team during this time can be helpful, or moving the engineer to the data team. Some companies have services where entire notebooks get put up into the cloud and all of it gets called, so the scientist has to write the notebook in a way that works for the cloud. It's rarer, but how I prefer it is a wrapper py file is created that calls just the relevant parts of the notebook, kind of like a header file. This process works well for me, but it as far as I know it is not standardized in the industry yet. In short, if you end up in this situation, there is a better way. Import the notebook into a .py file or into the cloud, don't rewrite it. This (hopefully) will remove this scenario you're describing (comment this is replying to) so those issues will become a historical footnote.

carabiner 6y ago
How do you maintain notebooks in production? You use papermill? What about versioning?
- proverbialbunny 6y ago
  Most libraries load entire notebooks from top to bottom when executing, and I believe papermill does too. (Please correct me if I'm wrong, as I've not used papermill.) This is great for making a dashboard, a report, or some other kind of analytics, but when it comes to a service the customer uses, you typically never want to load the whole notebook. This is where the industry standard way of loading the whole notebook tends to fall on its face. What we do is the cells that will end up in prod are written as functions inside of the notebook. This helps reduce globals when writing the notebook, so it is good form when prototyping, but also it allows just those functions to be called from the notebook, instead of running the entire notebook. You will probably want to write your own library to do this, but in the mean time there is one that works for this purpose https://github.com/grst/nbimporter https://github.com/grst/nbimporter (Ironically the author doesn't recognize this use case.) Using nbimporter you can import a notebook without loading it. You can then call functions within that notebook and only those functions get loaded and called. In my notebooks I have a process function which is like main(), but for for feature engineering. On the prod side the process function is called from the notebook. Process calls all of the necessary cells/functions for me in the correct order. This way the py wrapper only has to call one function, then the ML predict function gets called, so it's pretty small on the .py wrapper side. There are tests written on the .py side, IO functions and what not too. Data engineers love their classes, so it's easy to write a class that calls the notebook, and best of all calling a single function this way does not load globals, so the data engineers are happy. It's a nice library, because otherwisw you'd have to write your own (which you may end up wanting to do). This way if the model doesn't work as intended in production it's my fault. We log everything, so I can run the instance prod caught on my local machine, figure out what is going on, update the model, and then it can be deployed instantly. Version numbers on the engineering side I can't comment on as they have their own method, but on my end the second the model writes to a database then I strongly push for having a version number column or a version number metadata table in the database, so it's easy for me to access for future analysis.
  - zwaps 6y ago
    Is rewriting your functions from notebook to a py file really something a research scientist can not do? Or is it infeasible for some other reason? I'd imagine many data scientists want to publish their work as python packages or libraries during their PhD, so they should be familiar with writing classes or functions that work at a bare minimum.
test-account 6y ago
Exactely why I left Dolby. Didn't want to be a part of that process.
Jugurtha 6y ago
Oh my! We were exactly like this many, many years ago. See reply to this thread[0]. The way I view it is frictions and impedance mismatch. People lived in several universes and there were many "taps on shoulders". Data scientist tries to work on a project but the system upgrade messed up their compute environment and their GPU isn't working anymore. Data scientsits ssh'ing into a "powerful workstation" to have their notebooks run on more RAM or more powerful GPUs, having a certain convention to start their notebook servers with specific ports. Building models and then wanting to show results to the client and asking a colleague. Set up a VM on GCP, write a small application, scp the model to the machine, create an environment with the same dependencies to load the model, set up authentication on the machine. Email the client. Client doesn't reply in time. You have a bunch of VMs. Meanwhile the data scientist has produced another model with a notebook and they want the engineer to deploy it. Others want to reproduce it but have the same trouble with running the notebook (libraries, etc.). A complete mess. We ended up building our platform[0]. We wanted our PhDs to do what they were good at, and we wanted to handle a lot for them. In the same time, we wanted our more engineering inclined colleagues not to do that work themselves, and we let the platform do many of these things (building images, deploying, scheduling notebooks, etc). - [0]: https://iko.ai https://iko.ai

carabiner 6y ago

How do you maintain notebooks in production? You use papermill? What about versioning?

proverbialbunny 6y ago
Most libraries load entire notebooks from top to bottom when executing, and I believe papermill does too. (Please correct me if I'm wrong, as I've not used papermill.) This is great for making a dashboard, a report, or some other kind of analytics, but when it comes to a service the customer uses, you typically never want to load the whole notebook. This is where the industry standard way of loading the whole notebook tends to fall on its face. What we do is the cells that will end up in prod are written as functions inside of the notebook. This helps reduce globals when writing the notebook, so it is good form when prototyping, but also it allows just those functions to be called from the notebook, instead of running the entire notebook. You will probably want to write your own library to do this, but in the mean time there is one that works for this purpose https://github.com/grst/nbimporter https://github.com/grst/nbimporter (Ironically the author doesn't recognize this use case.) Using nbimporter you can import a notebook without loading it. You can then call functions within that notebook and only those functions get loaded and called. In my notebooks I have a process function which is like main(), but for for feature engineering. On the prod side the process function is called from the notebook. Process calls all of the necessary cells/functions for me in the correct order. This way the py wrapper only has to call one function, then the ML predict function gets called, so it's pretty small on the .py wrapper side. There are tests written on the .py side, IO functions and what not too. Data engineers love their classes, so it's easy to write a class that calls the notebook, and best of all calling a single function this way does not load globals, so the data engineers are happy. It's a nice library, because otherwisw you'd have to write your own (which you may end up wanting to do). This way if the model doesn't work as intended in production it's my fault. We log everything, so I can run the instance prod caught on my local machine, figure out what is going on, update the model, and then it can be deployed instantly. Version numbers on the engineering side I can't comment on as they have their own method, but on my end the second the model writes to a database then I strongly push for having a version number column or a version number metadata table in the database, so it's easy for me to access for future analysis.
- zwaps 6y ago
  Is rewriting your functions from notebook to a py file really something a research scientist can not do? Or is it infeasible for some other reason? I'd imagine many data scientists want to publish their work as python packages or libraries during their PhD, so they should be familiar with writing classes or functions that work at a bare minimum.

test-account 6y ago

Exactely why I left Dolby. Didn't want to be a part of that process.

Jugurtha 6y ago

Oh my! We were exactly like this many, many years ago. See reply to this thread[0]. The way I view it is frictions and impedance mismatch. People lived in several universes and there were many "taps on shoulders". Data scientist tries to work on a project but the system upgrade messed up their compute environment and their GPU isn't working anymore. Data scientsits ssh'ing into a "powerful workstation" to have their notebooks run on more RAM or more powerful GPUs, having a certain convention to start their notebook servers with specific ports. Building models and then wanting to show results to the client and asking a colleague. Set up a VM on GCP, write a small application, scp the model to the machine, create an environment with the same dependencies to load the model, set up authentication on the machine. Email the client. Client doesn't reply in time. You have a bunch of VMs. Meanwhile the data scientist has produced another model with a notebook and they want the engineer to deploy it. Others want to reproduce it but have the same trouble with running the notebook (libraries, etc.). A complete mess. We ended up building our platform[0]. We wanted our PhDs to do what they were good at, and we wanted to handle a lot for them. In the same time, we wanted our more engineering inclined colleagues not to do that work themselves, and we let the platform do many of these things (building images, deploying, scheduling notebooks, etc). - [0]: https://iko.ai https://iko.ai