4 ms·
The variation of complexity of transforms across Enterprises and teams within Enterprises is quite large. Spark and Hive (that I was product manager for, and u
by ibains 7y ago
The variation of complexity of transforms across Enterprises and teams within Enterprises is quite large.
Spark and Hive (that I was product manager for, and used by hundreds of Enterprises) are used quite heavily in data engineering. Many transforms fit well into relational/set based model.
In your particular case, perhaps due to the nature of your transforms, it might not have been the best fit.
Also, if your data is small, it's not always a great fit. However, in many large companies, if they have 500 transforms and 30 are small - there is simplicity in using a single technology.
- dvt 7y agoJust to be clear, I'm not discounting the work you're doing with Prophecy. We were using Databricks, and I can definitely see how a platform like Prophecy is a step up. I have a personal rule that I will always prop up any budding startup or entrepreneur (the golden rule and all that). So huge congrats on your release!