3 ms·

I've been following Uber's big data platform engineering for a while, this is a really interesting update. Specifically, it's interesting how well their Gen 3 s

by bradhe 8y ago

I've been following Uber's big data platform engineering for a while, this is a really interesting update. Specifically, it's interesting how well their Gen 3 stack held up. Also interesting choice to solve the incremental update problem at storage time instead of inserting another upstream ETL process (which would be incredibly expensive at this level of scale I'm sure). Also interesting: A lot of companies, you look at their big data ecosystem and it's littered with tons of tools. Uber seems like they've always done a good job keeping that pared down, which indicates to me that their team knows what they're doing for sure.

iblaine 8y ago
Being litter with tools is not necessarily a bad thing. A tool is a modular reusable component that acts a building block for something more complex. Consider Netflix. They are big open source contributors to the data industry. Many of their tools interest me because I can see their value outside of Netflix. Uber on the other hand is solving esoteric problems with esoteric solutions.
- kerng 8y ago
  This is the tool they released: https://github.com/uber/hudi https://github.com/uber/hudi
artwr 8y ago
I have only heard second hand, but internally Uber has always had a large litter of tools. There is always a little bit chaos edited out from what is presented to the outside world. They run several container schedulers (YARN, Mesos, Myriad), they used to run several separate workflow orchestrators. The proliferation of tools allows to test out different ideas in production conditions which is not bad per se. But the duplication of efforts or variability has to come with some advantage to continue doing it. Infrastructure engineers wanting to reinvent the wheel can create a lot of technical debt fast, and leveraged one at that. This takes nothing away from the great work Uber has been doing in the area. They have definitely set a great direction for their platform.
- tomnipotent 8y ago
  This is par for the course in any large org - Google, Yahoo, Facebook are all in the same boat. It's just not possible for a single data infrastructure to be everything to everyone.

by bradhe 8y ago

iblaine 8y ago

Being litter with tools is not necessarily a bad thing. A tool is a modular reusable component that acts a building block for something more complex. Consider Netflix. They are big open source contributors to the data industry. Many of their tools interest me because I can see their value outside of Netflix. Uber on the other hand is solving esoteric problems with esoteric solutions.

kerng 8y ago
This is the tool they released: https://github.com/uber/hudi https://github.com/uber/hudi

artwr 8y ago

I have only heard second hand, but internally Uber has always had a large litter of tools. There is always a little bit chaos edited out from what is presented to the outside world. They run several container schedulers (YARN, Mesos, Myriad), they used to run several separate workflow orchestrators. The proliferation of tools allows to test out different ideas in production conditions which is not bad per se. But the duplication of efforts or variability has to come with some advantage to continue doing it. Infrastructure engineers wanting to reinvent the wheel can create a lot of technical debt fast, and leveraged one at that. This takes nothing away from the great work Uber has been doing in the area. They have definitely set a great direction for their platform.

tomnipotent 8y ago
This is par for the course in any large org - Google, Yahoo, Facebook are all in the same boat. It's just not possible for a single data infrastructure to be everything to everyone.