4 ms·
Ask HN: Tips for software engineering sanity with Databricks notebooks?
Folks, what are your best tips and practices for getting some software engineering sanity into developing Databricks notebooks? In particular larger data pipelines that are orchestrated from something like Azure DataFactory.
It is driving me nuts to use auto-saved versions instead of clearly defined explicitly defined commits that can be proof-read, the lack of my favourite tools from my local environment, like vim, and troubles testing things properly.
I have already found some general hints like modularizing code etc, but I wanted to hear from people in the trenches, if you found some practices or set of practices that actually made a big difference in the development experience?
- PaulHoule 4y agoWith Jupyter you can check things into git. You probably want to have a pre-commit script that deletes all the data and just leaves the code. Some people really hate that because they like having notebooks with results in them in the git repository to read, but if you have data mixed with your code you will have the worst time merging. Tools like databricks lumber on with inadequate version control because people are used to everything being screwed up all the time when it comes to "data science".
- localbolu 4y agoQuite timely post as I'm actively experimenting on this. As for my organization, we're trying out Databricks Repos. Tbd how that goes, as an IDE it's incredibly clunky but it gives our data scientists access to the larger codebase. You might also consider nbdime to manage notebook diffs.
- alexott 4y agoIf you are using IDE you can look to dbx Python package that has the “dbx sync” command that allows to sync local code with Repos and then you can quickly test your code in notebooks
- lucasterra 4y agoDatabricks has Git integration, might be worth checking it out: https://docs.databricks.com/repos/index.html https://docs.databricks.com/repos/index.html
- samuell 4y agoGreat, thanks! Had seen something about this, and we are using repos for production code, but not realizing how easy it was to use personal repos inside the workspace, for development.
- alexott 4y agoYou can use Databricks Repos (https://docs.databricks.com/repos/index.html https://docs.databricks.com/repos/index.html) specifically files in repos (https://docs.databricks.com/repos/work-with-notebooks-other-files.html#work-with-python-and-r-modules https://docs.databricks.com/repos/work-with-notebooks-other-...) functionality that allows to use Python files (not notebooks!) as Python modules. Another alternative is to split notebooks into “library notebooks” that just define transformations, and “orchestration notebooks” that use code library notebooks to execute a “business logic”. In both approaches you can do code testing, etc. P.S. I have a demo of both approaches here: https://github.com/alexott/databricks-nutter-repos-demo https://github.com/alexott/databricks-nutter-repos-demo
- samuell 4y agoSweet, many thanks! Just looked into this, and this will be a huge improvement! Actually pleasantly surprised they store notebooks as .py files. Yay. (Looking at you, .ipynb files :/)
- rk_db 4y ago
- mrpinklestube 4y agoIf you are willing to try alternative options, try RATH: https://github.com/Kanaries/Rath https://github.com/Kanaries/Rath. It is an Open Source Augmented Analytics BI tool that can works great as a Data Bricks alternative.
- samuell 4y agoJust learned one can run debugging via pdb in notebooks: https://docs.databricks.com/_static/notebooks/python-debugger.html https://docs.databricks.com/_static/notebooks/python-debugge...