3 ms·
I agree, and so many data science and data engineering tools all seem to revolve around using notebooks, much to my frustration. I’ve worked in places whose dat
by FridgeSeal 6y ago
I agree, and so many data science and data engineering tools all seem to revolve around using notebooks, much to my frustration. I’ve worked in places whose data pipelines were built around seemingly infinite notebooks, all containing consistently poor software engineering.
It’s been enough to make me vow to not let people write notebooks that go into prod under my watch lol.
I’m constantly on the watch for software engineering focused tools that solve the issues, rather than data science/engineering focused tools. So many are inextricably linked into python as well, which doesn’t gel nicely with anywhere that has multiple languages in the codebase.
- ricklamers 6y agoWe'll allow your team to move parts of your code base from notebooks to scripts (.py/.sh/.R) to alleviate your frustration. That way you can keep using notebooks only for those parts where it makes most sense. They come together in pipelines that are JSON defined and git-versioned.
- calebkaiser 6y agoShameless plug, but I help maintain Cortex, and "software engineering focused tools that solve (ML) issues" is a neat summary of our entire philosophy. For example, instead of notebooks, our model serving platform (https://github.com/cortexlabs/cortex https://github.com/cortexlabs/cortex) uses YAML to structure deployments, and Python scripts to write inference APIs. It's still inextricably linked to Python, but only for writing your API. It's agnostic as to how the model itself is developed, so long as it can generate predictions.