3 ms·
I was wondering about Spotify's contributions to Hadoop stack such as Snakebite and Luigi. What will happen to these projects if they're moving to BigQuery?
by icaromedeiros 11y ago
I was wondering about Spotify's contributions to Hadoop stack such as Snakebite and Luigi. What will happen to these projects if they're moving to BigQuery?
- thesandlord 11y agoGoogle Cloud has a managed Hadoop service called Dataproc: https://cloud.google.com/dataproc/ https://cloud.google.com/dataproc/
- crb 11y agoSpotify are not moving to BigQuery alone. They are also using Dataproc, Google's managed Hadoop service (which co-incidentally went GA yesterday), which lets them get value out of all the orchestration tools they already use, like Luigi. (They are also adopting Cloud Dataflow, Cloud Pub/Sub and other parts of our Big Data stack.) Luigi has had BigQuery support for some time: https://github.com/spotify/luigi/pull/1002 https://github.com/spotify/luigi/pull/1002 Spotify will be talking more about the specifics of their data pipelines at GCP Next in SF in March, - I would expect to see a lot more on the new Google Cloud Big Data blog at https://cloud.google.com/blog/big-data/ https://cloud.google.com/blog/big-data/ too. [Disclosure, GCP guy, etc]
- dundun 11y agoNothing in this move should affect contributions of Snakebite and Luigi. If anything, it will just make them easier to use with cloud environments in addition to bare metal. With over 20K jobs/day, Hadoop will be a part of Spotify's data processing stack for quite a while. BigQuery is just a (awesome) piece of the full puzzle. [spotifier]