4 ms·

The last point isn’t so dramatic. There is version control for notebooks, and better version is coming (right now it’s in preview - code name - Projects, it’s i

The last point isn’t so dramatic. There is version control for notebooks, and better version is coming (right now it’s in preview - code name - Projects, it’s in official docs). Tests are possible - either Nutter from Microsoft, or home grown (about 20-30 lines on top of built in unittest). Deployment is also not so complicated - there are tools (provider for terraform, cicd-templates project, Databricks-cli, etc), you can refer MS Learn for short course about CI/CD for Databricks and Azure DevOps, etc...

MrPowers 6y ago
The last point was for teams that only rely on notebooks, sorry if I didn't make that clear. You're right that all those issues can be sidestepped if you build projects in version controlled Git repos, test the code, and deploy JAR / Wheel files. Speaking of testing, can you let me know if this PySpark testing fix worked for you ;) https://github.com/MrPowers/chispa/issues/6 https://github.com/MrPowers/chispa/issues/6
- alexott 6y ago
  I’m sorry for delay, will fix ASAP... My point is that you can do that even without jars/wheels - you can do VC and tests of notebooks. For example, https://github.com/alexott/databricks-nutter-projects-demo https://github.com/alexott/databricks-nutter-projects-demo
- Someone 6y ago
  As alexott says, you can link notebooks to a git repository and commit and rollback them from the Databricks UI (https://docs.databricks.com/notebooks/github-version-control.html https://docs.databricks.com/notebooks/github-version-control...) Problem is that it’s limited. You can’t, for example, commit multiple files in one go (that improves a little bit in https://docs.databricks.com/projects.html https://docs.databricks.com/projects.html), or merge changes a colleague made with your changes. You also have to use the UI Databricks provides. You can’t use a git CLI or whatever GUI you prefer. (all AFAIK, but I’m fairly certain about it)

MrPowers 6y ago

The last point was for teams that only rely on notebooks, sorry if I didn't make that clear. You're right that all those issues can be sidestepped if you build projects in version controlled Git repos, test the code, and deploy JAR / Wheel files. Speaking of testing, can you let me know if this PySpark testing fix worked for you ;) https://github.com/MrPowers/chispa/issues/6 https://github.com/MrPowers/chispa/issues/6

alexott 6y ago
I’m sorry for delay, will fix ASAP... My point is that you can do that even without jars/wheels - you can do VC and tests of notebooks. For example, https://github.com/alexott/databricks-nutter-projects-demo https://github.com/alexott/databricks-nutter-projects-demo
Someone 6y ago
As alexott says, you can link notebooks to a git repository and commit and rollback them from the Databricks UI (https://docs.databricks.com/notebooks/github-version-control.html https://docs.databricks.com/notebooks/github-version-control...) Problem is that it’s limited. You can’t, for example, commit multiple files in one go (that improves a little bit in https://docs.databricks.com/projects.html https://docs.databricks.com/projects.html), or merge changes a colleague made with your changes. You also have to use the UI Databricks provides. You can’t use a git CLI or whatever GUI you prefer. (all AFAIK, but I’m fairly certain about it)