2 ms·
I've thought about this a lot in the context of our org. What you're describing is taking modeling -> production (minus the screen shot one). Typically a data
by irjustin 3y ago
I've thought about this a lot in the context of our org. What you're describing is taking modeling -> production (minus the screen shot one).
Typically a data engineer will be the one who helps bridge that gap, but that has a problem of does data engineer's output == scientist output, which can be time consuming to handle.
To shrink the gap from dev->prod - we have 2 notebooks, one for model development and one model deployment in production. We use papermill[0] to execute directly notebooks in production.
Shared functions between the dev/prod that are built by the scientist are put into a separate notebook and then imported via `run`. If I'm honest, our scientist don't do this and simply copy/paste the functions if we don't yell at them to fix it.
This basically allows us to stay within the jupyter environment entirely so that dependancies are isolated.
So, it's far from perfect, but it's allowed us to shrink the dev->prod life cycle time. Love to hear what others have done towards the same end.