4 ms·
For example, we do a lot of of low-level RDD Operations through databricks. From skimming the Website I feel something like this is not in the scope of this pro
by 0x008 6y ago
For example, we do a lot of of low-level RDD Operations through databricks. From skimming the Website I feel something like this is not in the scope of this project.
In the end, I feel, it is about wording. Databricks is a serverless spark environment with Azure integration and notebooks. Unless the product copies all the aspects (i.e. the hosting) it may not be wise to call it a databricks alternative.
If I reed the title as it is here on HN, I would think is about the infrastructure and not about a custom low-code JSON-based template language on top of spark sql.
- seddonm1 6y agoFair enough. This really replaces the Notebook experience of Databricks and gives you the option to self host and as it is dockerised it is easy to deploy on Kubernetes (with JupyterHub) or other container hosts. If you want to do low-level RDD operations you can easily build a plugin. I think previously Spark was difficult to run, now with Docker and supported Kubernetes it is fairly easy. Edit: I forgot to link to the Deploy repo which does have some scripts to easily deploy to cloud environments (but needs some work): https://github.com/tripl-ai/deploy https://github.com/tripl-ai/deploy