14 ms·

Michelangelo: Uber’s Machine Learning Platform

268 points by holografix 9y ago

jackyinger 9y ago
I expect this will be an unpopular opinion, but a proliferation of frameworks does not create value for anyone besides the developers employed in making it. Sure different frameworks handle different uses better than others, but it doesn't change the fact that in a given genre (machine learning, web frontend, ORM, etc.) that there is a lot of duplication going on. If you want to look cool and impress people, make a tool that doesn't exist, develop a new algorithm, or contribute a great feature to an existing framework.
- xbeta 9y ago
  Sure, you sound like anyone can just have a eureka moment and come up with an algorithm. Uber has a legitimate use-case that fits their needs on ML, so they build it for that need. And why not just also talk about it too as a lesson-learned for others. Do you simply have very high expectation of everything in this world?
  - jackyinger 9y ago
    There are plenty of meaningful ways to contribute to the body of open source software that don't require strokes of genius. Bug fixes and refactoring come to mind. I just picked this thread as a starting point Uber may have a legit need for a particular framework. My main point is there is no need to make frameworks for frameworking's sake. Purposeful OSS collaboration is just as good, if not a better way to gain influence than a PR splash. Of course that assumes that the goal is to make cool things, which seems loosely connect with wooing investors at best. (Again in general, not picking on Uber.)
    - xbeta 9y ago
      I don't work in Uber's team on ML, so I can't answer this, but I also cannot just simply assume everything is built-for-PR-splash. You can argue why can't they simply talk about the reason of not reusing any existing OSS framework than building yourself, but making an assumption that they did it for the sake of PR is lack of factual support. Are we now living in a world assuming everything Uber did was for PR? Last I heard they had 1500+ engineers and with that size, I am sure they can just talk about things they have done in the last few years.
- lwansbrough 9y ago
  That's how I feel about most projects I see on here. Toy languages are my personal pet peeve. So much talent being wasted on useless projects. I'm not really one to speak - none of my projects make a substantially positive impact on society, but they have their niches.
  - gfodor 9y ago
    Do you really feel this way? It seems like a pretty strange world view to think that someone posting a thing they built on a site called hacker news implores you to judge their talent as being wasted, or their project as being useless. Not only does it seem wrong to judge things people post here that way, anyone out there doing good work knows that the path to doing meaningful work begins with "useless projects."
    - lwansbrough 9y ago
      The reality for the people who post on here is that most of us aren't doing anything truly substantial. I include myself in that group. You can believe your IOT AI messenger bot is going to revolutionize an industry, but the more likely situation is that it doesn't. That being said, such a project is not something I'd consider useless, but it's in the same range as the things I do consider useless. Take toy languages for example: could it revolutionize software development? Maybe. Will it? Almost certainly not. How many languages are commonly used in software engineering, and how much ability have you gained by doing it? It's not for me to say where you apply your time and from what you get your enjoyment from, but I do believe there are objectively more important problems that could be solved. There are plenty of problems out there that need to be solved, and engineers here have the capability to solve them -- I think it's fair to say we're more well equipped than most to solve some of the largest problems humans face. Here's a quick list of problems that are worth solving (in my opinion), vs. things that are figuratively useless: - Rising global temperatures will disrupt agriculture and food supplies, how will we farm when desertification consumes the croplands? - Thousands of people walk by homeless people on the street every day, if a small portion of those people stopped and engaged with the homeless, how many lives could be improved? - Children in third world countries struggle to receive proper education, how can we reach them and improve their education? - Governments around the world are failing to represent the needs and desires of their people, what can be done to help governments succeed? - Growing levels of automation are replacing a staggering number of jobs, and if the growth continues, the great depression will be ahead of us, not behind us. - Global financial markets are controlled by companies which you and I have no say in, yet they hold most of the worlds wealth. In contrast: - I want X programming language that does Y because Z doesn't do Y, in all likelihood, I will be the only one who ever uses this language, but it'll be a personal accomplishment. - I don't want to leave the house or stop working for half an hour to make a meal, how can I improve/speed up the time it takes for me to get a meal from my favourite restaurant? - Making X sucks, I want a robot to do it - Bad AI for X - IOT for X - Framework for X - Uber for X How often do I see the latter vs. the former? Imagine if you saw FOSS projects attempting to solve problems on the first list as often as you saw projects attempting to solve problems on the second list. I truly believe the world would be a better place if that was the case. I said earlier I'm guilty of this myself. I build consumer facing supporting software for entertainment media. I like what I do, but I don't think I'm contributing much to society, and that makes me uneasy.
- rockinghigh 9y ago
  Have you read the article? Michelangelo is a platform, not a framework. It abstracts the underlying frameworks.
- deleted 9y ago
  [deleted]
- thewhitetulip 9y ago
  Yes, but in some cases a platform and framework do help in formalizing things.
- deleted 9y ago
  [deleted]
- bayonetz 9y ago
  In this particular genre there are no off-the-shelf platforms or even accepted set of best practices for something as comprehensive as this for generalized ML pipelining infrastructure. It's still very Wild West. I've done a lot of research into what pipelines look like at different companies and cloud providers and I can tell you this is a nice one. Some have a good training story, some have a good deployment story, some have a good experimentation story, and so on but none have all the things. Projects like Tensorflow Serving and PipelineAI are heading there but still quite a lot missing compared with something like what was presented here.
- buremba 9y ago
  Your comment is indeed not helpful and prevents people to open-source their work unless they think that it is awesome. This tool may not be great but can inspire similar tools or may be helpful for developers who want to learn how to develop a software similar to this one, or attack developers to contribute the project and eventually may become a standard tool as UI for ML pipelines etc.
- Maro 9y ago
  This is like FB's FBLearner, which I've used a little bit. This is a platform, so this is not at at the SKL/Tensorflow/Caffe level, this is something which says: put your data in our data warehouse in this format, tell me what model you want to run on it, go have a coffee, and by the time you're back I'll magically run your model on a big cluster quickly; if you want I can also try out different hyperparams/models, and I'll give you standard plots/metrics to evaluate the thing. You can also use 1000s of existing feature vectors/metrics that other people have put in the DWH. You can also clone an existing model and tweak that. If you're happy with the model, you can deploy, either on my API by querying and I'll tell you the prediction, or I can publish the model in binary form to a datastore and you can just import it in your iOS code in 3 lines. You can also schedule nightly re-trains, I'll notify you if something big changes in the accuracy metrics, etc. The point is, this sort of ML platform allows engineers at big companies to leverage existing data/metrics/f.vectors, models, infra, etc. and allows them to move 10-100x faster than startups/hobbyists. When I say 10-100x, I actually mean it, I'm also running ML models at home, and the amount of time I waste on this glue stuff is huge (starting from data prep to f---ing around with tensor ranks). I'm sure there are already startups working on platforms like this for everybody, we'll just have to wait for one of those to become good.
  - MasterScrat 9y ago
    > I'm sure there are already startups working on platforms like this for everybody Does anyone knows of any such open-source product actually usable today? This would solve a real need. (Also I'd love to work on such a thing)
    - Kpourdeilami 9y ago
      I've been working on something very similar to that. It allows you to connect a SQL database, S3 bucket, etc. then write a query for the data that you want to train the model on. It takes a couple minutes and then it'll expose the trained model over a REST endpoint. It also has feedback loops to fine tune the model on real-time data and has an option to store all the predictions made by model into Redshift or S3 to be visualized on a tool like Tableau. The tool itself is not open-source but it can be installed on the user's AWS, Azure, or GCP account to keep their models and data private. It's still in alpha testing (bunch of pilots) and I'll probably do a "Show HN" post when it is ready for a public beta
- gaius 9y ago
  It's a bet similar to a startup - your "framework" might not be popular but if it is - for no particular reason other than a cool name and lots of retweets - then you are sorted. How many JS "frameworks" are there now?
philliphaydon 9y ago
I wonder if this thing is used for Uber Eats in Singapore... Because I use User Eats all the time, the estimated time is MASSIVELY inaccurate. It's not even within 30% to be close to actual time. The estimated time is always less than the actual time, but every minutes the estimated time re-adjusts. By the time it's actually delivered it's about 15 to 20 minutes more than the original estimate. Note: I'm just complaining because the article uses User Eats as an example and as a service I use often I wonder if this ML Platform is used at all...
- strin 9y ago
  Maybe they trained on global data and it biases against sample portion of the dataset that represents Singapore?
  - philliphaydon 9y ago
    Wouldn't the ML learn to that times differ globally tho?
    - d4rti 9y ago
      Only with the right models and features.
    - Quarrelsome 9y ago
      ML is only as good as its inputs and if it doesn't have any Singaporean inputs then its gonna potentially suck for Singapore if its an outlier.
- jdwyah 9y ago
  Chuckle. Same experience. As far as I can tell the estimated time is about as good as the Windows 3.1 file copy time estimation.
- anindha 9y ago
  Maybe they did use ML in your scenario just not the way you thought: It takes 50 mins to get your delivery but to make you order they need to tell you its going to take 30 mins. ML also tells them that the 20 minute discrepancy won't stop you ordering next time. Cycle continues.
linkregister 9y ago
The "subscribe for updates" modal is extremely distracting. It has a long lag before it repositions at the top of the viewport, at least in my browser.
tomerweller 9y ago
Describing your ML platform is nice. Open sourcing it would have been nicer.
- andrew-lucker 9y ago
  I didn't make it past "democratizing" and "internal" in the same sentence. I think their words are different from our words.
- Dystopian 9y ago
  From my experience (and what this looks like) is that there isn't a ton of novel IP to open source here. A lot of the times ML "platforms" are composed of a number of open source components glued together in an elegant way (which I'm not implying is easy, because it's very difficult). From looking at this from the outside the only things in this platform that may be open-sourceable would be the job scheduling and visualizations - and there are already variants of open source tooling which could be repurposed for those tasks (or may even be powering those components). The main purpose of this post seems to be Uber's way of standardizing their workflows + a little extra glue (which they're calling their platform). It still provides a lot of value. Also, Uber does have a few cool open source projects: https://uber.github.io/ https://uber.github.io/ (but could admittedly have more).
  - tomerweller 9y ago
    You have a point. Describing their workflow has value to it. With that said, the title of the post is not "Meet our workflow", It's "Meet X: bla bla". In the world of software, one expects to actually see a product named X that bla blas.
    - mogili 9y ago
      Absolutely! I assumed they had open sourced it and was searching for the Github link :(
    - lwansbrough 9y ago
      > You have a point. Describing their workflow has value to it. This is a really valuable thing to acknowledge. There is some sharing of company philosophies, but seldom do I see companies fully "open sourcing" their workflows and strategies. Perhaps because the people at the top see that as the real value their company brings - that knowledge. Nevertheless, it's extremely valuable and I wish I could see more things like that. Basecamp's book "Getting Real" is close to what that might look like, I think.
pyrophane 9y ago
So, what? Uber is now the new IBM Watson consultancy?
- gipp 9y ago
  I'm not really sure what similarities you see between the two.
- jamesmishra 9y ago
  Former Uber engineer here. I don't think Uber is anything like IBM Watson, mostly because Uber uses machine learning to solve business problems for its own products--not for other companies' products. Most large companies that use machine learning in production will have something similar to Uber's Michelangelo. For example, Facebook has FBLearner Flow[1]. Catherine Dong recently wrote a TechCrunch article describing this broader industry trend[2]. [1]: https://code.facebook.com/posts/1072626246134461/introducing-fblearner-flow-facebook-s-ai-backbone/ https://code.facebook.com/posts/1072626246134461/introducing... [2]: https://techcrunch.com/2017/08/08/the-evolution-of-machine-learning/ https://techcrunch.com/2017/08/08/the-evolution-of-machine-l...
cyrux004 9y ago
TLC anyone ?
- curiousDog 9y ago
  What is TLC?
  - Larrikin 9y ago
    An autocorrect of tldr
    - myhrvold 9y ago
      Also: Tender Loving Care or Taxi & Limousine Commission :0p
    - _pmf_ 9y ago
      I prefer to think he likes a hug.
ahead 9y ago
Is Michelangelo the name of their previously published paper "Scaling Machine Learning as a Service" https://news.ycombinator.com/item?id=14708761 https://news.ycombinator.com/item?id=14708761 , or is this something separate?
- rockinghigh 9y ago
  Yes, it's the same platform.
- deleted 9y ago
  [deleted]
aorloff 9y ago
I am surprised by this - h2o.ai seems to have many if not all of the features of this, and is open source. "Specifically, there were no systems in place to build reliable, uniform, and reproducible pipelines for creating and managing training and prediction data at scale. Prior to Michelangelo, it was not possible to train models larger than what would fit on data scientists’ desktop machines, and there was neither a standard place to store the results of training experiments nor an easy way to compare one experiment to another." Seems like h2o.ai fits a lot of that bill.
- opensandwich 9y ago
  h2o.ai doesn't really do data pipelines, though it does appear they are eager to go into this space through their new driverless.ai tool. However this does not appear to be open source.
- JPKab 9y ago
  Something else that throws me off on this: My team's experience with MLlib has been bad. Especially compared to H2O Sparkling Water. Anyone else on here find MLlib to not be as good as advertised? I was surprised to see Uber using it.
  - thecity2 9y ago
    Spark is good if it fits your use case (or if you can make it fit your use case). We have definitely found that scaling can be a real issue though.
visarga 9y ago
It would be more useful to publish anonymized datasets, if they got any useful data.
sandGorgon 9y ago
>For every model that is trained in Michelangelo, we store a versioned object in our model repository in Cassandra.. >For important model types, we provide sophisticated visualization tools to help modelers understand why a model behaves as it does, as well as to help debug it if necessary How is this done - what serialisation format (pickle, pmml,etc) is being used? And more importantly, how are you tracing into the model ? It looks like this is a Spark based framework
- Eridrus 9y ago
  It's probably being "serialized" as a JAR file which just contains the code you call into. That was Spark's online deployment story last I heard, and jives with them saying the library mode has a Java API.
andrewfong 9y ago
Are there any ML-as-a-service platforms that offers something comparable to what Uber has here? And importantly, do you actually like using them? That is, suppose I'm willing to do the following: * Send said service a firehose of data * Write code for feature selection, data cleanup, transformations, etc. -- i.e. the sort of code you'd expect to find in a data scientist's Jupyter notebook. * Specify the type of ML model and parameters I'd like to experiment with. Is there a service that handles data warehousing, serving up predictions via a REST API, deployment of new models, reporting model performance over time, and general scaling issues? I know it's possible to set up a large portion of this using AWS services, but ideally, it'd be nice to have something with a Heroku-like ease-of-use, but for ML.
- mobileexpert 9y ago
  Azure ML
- mayank 9y ago
  Not quite there, but parts are: * You can send your firehose to BigQuery, and write Javascript UDFs to transform the data. * You can use a service like DataRobot to automatically train a bunch of models. * You can deploy the trained models using a serverless platform like Lambda. Nothing that's one-size-fits-all, but watch out for trends in that direction as more ML workflows get standardized.
- romankolpak 9y ago
  You should take a look at DataRobot, it does most of the things you've listed.
- ANaimi 9y ago
  Algorithmia Enterprise (algorithmia.com/enterprise). Inference- and deep learning-centric. (I'm part of the team).
- wmeddie 9y ago
  Skymind's SKIL does this. (I work on the team.) You don't have to send it your data though, you can just install it inside your system and have it connect to the data source or stream to train models (optionally doing hyper-parameter search), deploy them to REST servers and monitor their performance.
strin 9y ago
A random idea: in this article, they talked about a Feature Store shared across teams. Feature extraction also seems a major bottleneck in their modeling. It seems a very natural and valuable thing to have a platform to share "generic features". Yann LeCun said once a deep neural network predicts many targets, say 1000 classes in ImageNet, it is possible for the model to learn quite generic features. So it makes sense to pre-train on a large amount of data and a reasonable number of targets, and then share the learned feature extractors with others. Could this be a business? Or a community? Thoughts?
- Eridrus 9y ago
  Pretrained models for generic things exist publicly, e.g. models trained on ImageNet, or pretrained word vectors. Not a whole lot of general use in having a feature extractor tuned to Uber's data.
manugarri 9y ago
So I have seen this Michelangelo pop up in other articles. The key question is? Is Uber going to open source it? If not, why bother writing articles about the specifics of their platform?
- sunsetMurk 9y ago
  PR & attracting talent? Though open sourcing would be great and do the same thing, but more so.
nl 9y ago
This solves a real problem. ML pipelines are complicated, and often involve multiple languages and platforms (just as described in this post). But.. there are existing solutions in this area. Notable Luigi[1] (from Spotify) and Airflow[2] (from AirBNB) both seem to have a lot of overlap with this. I'm most familiar with Luigi, and it does many of the things that are listed here. Some of the model visualisations look pretty nice, and don't come out-of-the-box in the other platforms. So I'm not really sure what makes this unique. [1] https://github.com/spotify/luigi https://github.com/spotify/luigi [2] https://airflow.incubator.apache.org/ https://airflow.incubator.apache.org/
- gipp 9y ago
  What? Luigi and Airflow are ETL frameworks. They express dependencies between tasks, schedule their execution, and that's about it. Model storage and versioning, feature definition and extraction (batch and real-time), model deployment... Luigi and Airflow do almost nothing described in this post.
  - nl 9y ago
    They do model/feature storage and versioning just fine. Or at least Luigi does - I use it daily. And feature extraction and modelling is my main usecase. It works really well, across Spark, Scikit and R, saving data in HDFS. Some examples: https://blog.dominodatalab.com/luigi-pipelines-domino/ https://blog.dominodatalab.com/luigi-pipelines-domino/ http://blog.richardweiss.org/2016/10/13/kaggle-with-luigi.html http://blog.richardweiss.org/2016/10/13/kaggle-with-luigi.ht... https://github.com/Atreya22/luigi_rosmann_sales https://github.com/Atreya22/luigi_rosmann_sales
    - gipp 9y ago
      Yeah, it does all those things, as long as you write all the code, set up all the systems for it to interact with, and manage all the logic yourself. That's like saying Python does all those things.
      - 9y ago
atonse 9y ago
This reminds me of LivingSocial. For years, LivingSocial was like a mini black hole of really good startup-like talent in the DC area. But when you have that many smart people under one roof, you have to keep them busy. So you get these kinds of reinventing-a-beautiful-wheel projects. Not knocking the quality, just knocking the growing distraction and temptation to build in-house rather than use existing tech. LivingSocial had such good quality output, but they were all over the place. They had to keep all those productive people busy after all. And the best way was to keep churning out new products.
- JPKab 9y ago
  Yeah, as a techie that used to live in DC, LivingSocial people were always at the Meetups. I was always blow away by the talent being used to work on a shitty coupon app.
- hitekker 9y ago
  I agree. "Keeping them busy" also raises an uncomfortable question: if Uber and other valuable Unicorns can be executed 99.999% with 25% of the engineers they hire, do they need the next 75% of engineers? Even considering crunch-time and potential diversification/expansion, full-time employment of non-critical engineers is obviously inefficient. And, as Uber proved, politically toxic and consequently harmful to the productivity of the business. More pressing is the existential tension underlying this quandary. That our economy will generally and increasingly forgo the skilled and unskilled labor provided by our society. Framed concretely: one day, we may define a "10x engineer" as an engineer who can send ten other engineers to the soup kitchen. I'm still looking for a well-thought-out solution.
  - xixi77 9y ago
    This is not really new; just think of all the stuff that came out of AT&T labs...
rodionos 9y ago
> batch jobs run on a schedule or a trigger and are integrated with data quality monitoring tools to quickly detect regressions in the pipeline–either due to local or upstream code or data issues I'm curious what tooling they're using to ensure data quality, in particular time series data.
strictnein 9y ago
SAP has Leonardo. Uber has Michelangelo. Who has Donatello and Raphael?
- oculusthrift 9y ago
  Personally i don't get the obsession with ninja turtles.... /s
  - omot 9y ago
    Funny you thought of Ninja Turtles, I was thinking Renaissance artists.
    - strictnein 9y ago
      Then this Epic Rap Battle of History is for you (salty language warning): https://www.youtube.com/watch?v=6HZ5V9rT96M https://www.youtube.com/watch?v=6HZ5V9rT96M
- tomdre 9y ago
  Splinter is the best
thecity2 9y ago
This doesn't seem to be an open source project, unless I'm missing something. It's kind of a show-and-tell basically.

268 points by holografix 9y ago

I expect this will be an unpopular opinion, but a proliferation of frameworks does not create value for anyone besides the developers employed in making it. Sure different frameworks handle different uses better than others, but it doesn't change the fact that in a given genre (machine learning, web frontend, ORM, etc.) that there is a lot of duplication going on. If you want to look cool and impress people, make a tool that doesn't exist, develop a new algorithm, or contribute a great feature to an existing framework.

xbeta 9y ago
Sure, you sound like anyone can just have a eureka moment and come up with an algorithm. Uber has a legitimate use-case that fits their needs on ML, so they build it for that need. And why not just also talk about it too as a lesson-learned for others. Do you simply have very high expectation of everything in this world?
- jackyinger 9y ago
  There are plenty of meaningful ways to contribute to the body of open source software that don't require strokes of genius. Bug fixes and refactoring come to mind. I just picked this thread as a starting point Uber may have a legit need for a particular framework. My main point is there is no need to make frameworks for frameworking's sake. Purposeful OSS collaboration is just as good, if not a better way to gain influence than a PR splash. Of course that assumes that the goal is to make cool things, which seems loosely connect with wooing investors at best. (Again in general, not picking on Uber.)
  - xbeta 9y ago
    I don't work in Uber's team on ML, so I can't answer this, but I also cannot just simply assume everything is built-for-PR-splash. You can argue why can't they simply talk about the reason of not reusing any existing OSS framework than building yourself, but making an assumption that they did it for the sake of PR is lack of factual support. Are we now living in a world assuming everything Uber did was for PR? Last I heard they had 1500+ engineers and with that size, I am sure they can just talk about things they have done in the last few years.
lwansbrough 9y ago
That's how I feel about most projects I see on here. Toy languages are my personal pet peeve. So much talent being wasted on useless projects. I'm not really one to speak - none of my projects make a substantially positive impact on society, but they have their niches.
- gfodor 9y ago
  Do you really feel this way? It seems like a pretty strange world view to think that someone posting a thing they built on a site called hacker news implores you to judge their talent as being wasted, or their project as being useless. Not only does it seem wrong to judge things people post here that way, anyone out there doing good work knows that the path to doing meaningful work begins with "useless projects."
  - lwansbrough 9y ago
    The reality for the people who post on here is that most of us aren't doing anything truly substantial. I include myself in that group. You can believe your IOT AI messenger bot is going to revolutionize an industry, but the more likely situation is that it doesn't. That being said, such a project is not something I'd consider useless, but it's in the same range as the things I do consider useless. Take toy languages for example: could it revolutionize software development? Maybe. Will it? Almost certainly not. How many languages are commonly used in software engineering, and how much ability have you gained by doing it? It's not for me to say where you apply your time and from what you get your enjoyment from, but I do believe there are objectively more important problems that could be solved. There are plenty of problems out there that need to be solved, and engineers here have the capability to solve them -- I think it's fair to say we're more well equipped than most to solve some of the largest problems humans face. Here's a quick list of problems that are worth solving (in my opinion), vs. things that are figuratively useless: - Rising global temperatures will disrupt agriculture and food supplies, how will we farm when desertification consumes the croplands? - Thousands of people walk by homeless people on the street every day, if a small portion of those people stopped and engaged with the homeless, how many lives could be improved? - Children in third world countries struggle to receive proper education, how can we reach them and improve their education? - Governments around the world are failing to represent the needs and desires of their people, what can be done to help governments succeed? - Growing levels of automation are replacing a staggering number of jobs, and if the growth continues, the great depression will be ahead of us, not behind us. - Global financial markets are controlled by companies which you and I have no say in, yet they hold most of the worlds wealth. In contrast: - I want X programming language that does Y because Z doesn't do Y, in all likelihood, I will be the only one who ever uses this language, but it'll be a personal accomplishment. - I don't want to leave the house or stop working for half an hour to make a meal, how can I improve/speed up the time it takes for me to get a meal from my favourite restaurant? - Making X sucks, I want a robot to do it - Bad AI for X - IOT for X - Framework for X - Uber for X How often do I see the latter vs. the former? Imagine if you saw FOSS projects attempting to solve problems on the first list as often as you saw projects attempting to solve problems on the second list. I truly believe the world would be a better place if that was the case. I said earlier I'm guilty of this myself. I build consumer facing supporting software for entertainment media. I like what I do, but I don't think I'm contributing much to society, and that makes me uneasy.
rockinghigh 9y ago
Have you read the article? Michelangelo is a platform, not a framework. It abstracts the underlying frameworks.
deleted 9y ago
[deleted]
thewhitetulip 9y ago
Yes, but in some cases a platform and framework do help in formalizing things.
deleted 9y ago
[deleted]
bayonetz 9y ago
In this particular genre there are no off-the-shelf platforms or even accepted set of best practices for something as comprehensive as this for generalized ML pipelining infrastructure. It's still very Wild West. I've done a lot of research into what pipelines look like at different companies and cloud providers and I can tell you this is a nice one. Some have a good training story, some have a good deployment story, some have a good experimentation story, and so on but none have all the things. Projects like Tensorflow Serving and PipelineAI are heading there but still quite a lot missing compared with something like what was presented here.
buremba 9y ago
Your comment is indeed not helpful and prevents people to open-source their work unless they think that it is awesome. This tool may not be great but can inspire similar tools or may be helpful for developers who want to learn how to develop a software similar to this one, or attack developers to contribute the project and eventually may become a standard tool as UI for ML pipelines etc.
Maro 9y ago
This is like FB's FBLearner, which I've used a little bit. This is a platform, so this is not at at the SKL/Tensorflow/Caffe level, this is something which says: put your data in our data warehouse in this format, tell me what model you want to run on it, go have a coffee, and by the time you're back I'll magically run your model on a big cluster quickly; if you want I can also try out different hyperparams/models, and I'll give you standard plots/metrics to evaluate the thing. You can also use 1000s of existing feature vectors/metrics that other people have put in the DWH. You can also clone an existing model and tweak that. If you're happy with the model, you can deploy, either on my API by querying and I'll tell you the prediction, or I can publish the model in binary form to a datastore and you can just import it in your iOS code in 3 lines. You can also schedule nightly re-trains, I'll notify you if something big changes in the accuracy metrics, etc. The point is, this sort of ML platform allows engineers at big companies to leverage existing data/metrics/f.vectors, models, infra, etc. and allows them to move 10-100x faster than startups/hobbyists. When I say 10-100x, I actually mean it, I'm also running ML models at home, and the amount of time I waste on this glue stuff is huge (starting from data prep to f---ing around with tensor ranks). I'm sure there are already startups working on platforms like this for everybody, we'll just have to wait for one of those to become good.
- MasterScrat 9y ago
  > I'm sure there are already startups working on platforms like this for everybody Does anyone knows of any such open-source product actually usable today? This would solve a real need. (Also I'd love to work on such a thing)
  - Kpourdeilami 9y ago
    I've been working on something very similar to that. It allows you to connect a SQL database, S3 bucket, etc. then write a query for the data that you want to train the model on. It takes a couple minutes and then it'll expose the trained model over a REST endpoint. It also has feedback loops to fine tune the model on real-time data and has an option to store all the predictions made by model into Redshift or S3 to be visualized on a tool like Tableau. The tool itself is not open-source but it can be installed on the user's AWS, Azure, or GCP account to keep their models and data private. It's still in alpha testing (bunch of pilots) and I'll probably do a "Show HN" post when it is ready for a public beta
gaius 9y ago
It's a bet similar to a startup - your "framework" might not be popular but if it is - for no particular reason other than a cool name and lots of retweets - then you are sorted. How many JS "frameworks" are there now?

philliphaydon 9y ago

I wonder if this thing is used for Uber Eats in Singapore... Because I use User Eats all the time, the estimated time is MASSIVELY inaccurate. It's not even within 30% to be close to actual time. The estimated time is always less than the actual time, but every minutes the estimated time re-adjusts. By the time it's actually delivered it's about 15 to 20 minutes more than the original estimate. Note: I'm just complaining because the article uses User Eats as an example and as a service I use often I wonder if this ML Platform is used at all...

strin 9y ago
Maybe they trained on global data and it biases against sample portion of the dataset that represents Singapore?
- philliphaydon 9y ago
  Wouldn't the ML learn to that times differ globally tho?
  - d4rti 9y ago
    Only with the right models and features.
  - Quarrelsome 9y ago
    ML is only as good as its inputs and if it doesn't have any Singaporean inputs then its gonna potentially suck for Singapore if its an outlier.
jdwyah 9y ago
Chuckle. Same experience. As far as I can tell the estimated time is about as good as the Windows 3.1 file copy time estimation.
anindha 9y ago
Maybe they did use ML in your scenario just not the way you thought: It takes 50 mins to get your delivery but to make you order they need to tell you its going to take 30 mins. ML also tells them that the 20 minute discrepancy won't stop you ordering next time. Cycle continues.

linkregister 9y ago

The "subscribe for updates" modal is extremely distracting. It has a long lag before it repositions at the top of the viewport, at least in my browser.

tomerweller 9y ago

Describing your ML platform is nice. Open sourcing it would have been nicer.

andrew-lucker 9y ago
I didn't make it past "democratizing" and "internal" in the same sentence. I think their words are different from our words.
Dystopian 9y ago
From my experience (and what this looks like) is that there isn't a ton of novel IP to open source here. A lot of the times ML "platforms" are composed of a number of open source components glued together in an elegant way (which I'm not implying is easy, because it's very difficult). From looking at this from the outside the only things in this platform that may be open-sourceable would be the job scheduling and visualizations - and there are already variants of open source tooling which could be repurposed for those tasks (or may even be powering those components). The main purpose of this post seems to be Uber's way of standardizing their workflows + a little extra glue (which they're calling their platform). It still provides a lot of value. Also, Uber does have a few cool open source projects: https://uber.github.io/ https://uber.github.io/ (but could admittedly have more).
- tomerweller 9y ago
  You have a point. Describing their workflow has value to it. With that said, the title of the post is not "Meet our workflow", It's "Meet X: bla bla". In the world of software, one expects to actually see a product named X that bla blas.
  - mogili 9y ago
    Absolutely! I assumed they had open sourced it and was searching for the Github link :(
  - lwansbrough 9y ago
    > You have a point. Describing their workflow has value to it. This is a really valuable thing to acknowledge. There is some sharing of company philosophies, but seldom do I see companies fully "open sourcing" their workflows and strategies. Perhaps because the people at the top see that as the real value their company brings - that knowledge. Nevertheless, it's extremely valuable and I wish I could see more things like that. Basecamp's book "Getting Real" is close to what that might look like, I think.

pyrophane 9y ago

So, what? Uber is now the new IBM Watson consultancy?

gipp 9y ago
I'm not really sure what similarities you see between the two.
jamesmishra 9y ago
Former Uber engineer here. I don't think Uber is anything like IBM Watson, mostly because Uber uses machine learning to solve business problems for its own products--not for other companies' products. Most large companies that use machine learning in production will have something similar to Uber's Michelangelo. For example, Facebook has FBLearner Flow[1]. Catherine Dong recently wrote a TechCrunch article describing this broader industry trend[2]. [1]: https://code.facebook.com/posts/1072626246134461/introducing-fblearner-flow-facebook-s-ai-backbone/ https://code.facebook.com/posts/1072626246134461/introducing... [2]: https://techcrunch.com/2017/08/08/the-evolution-of-machine-learning/ https://techcrunch.com/2017/08/08/the-evolution-of-machine-l...

cyrux004 9y ago

TLC anyone ?

curiousDog 9y ago
What is TLC?
- Larrikin 9y ago
  An autocorrect of tldr
  - myhrvold 9y ago
    Also: Tender Loving Care or Taxi & Limousine Commission :0p
  - _pmf_ 9y ago
    I prefer to think he likes a hug.

ahead 9y ago

Is Michelangelo the name of their previously published paper "Scaling Machine Learning as a Service" https://news.ycombinator.com/item?id=14708761 https://news.ycombinator.com/item?id=14708761 , or is this something separate?

rockinghigh 9y ago
Yes, it's the same platform.
deleted 9y ago
[deleted]

aorloff 9y ago

I am surprised by this - h2o.ai seems to have many if not all of the features of this, and is open source. "Specifically, there were no systems in place to build reliable, uniform, and reproducible pipelines for creating and managing training and prediction data at scale. Prior to Michelangelo, it was not possible to train models larger than what would fit on data scientists’ desktop machines, and there was neither a standard place to store the results of training experiments nor an easy way to compare one experiment to another." Seems like h2o.ai fits a lot of that bill.

opensandwich 9y ago
h2o.ai doesn't really do data pipelines, though it does appear they are eager to go into this space through their new driverless.ai tool. However this does not appear to be open source.
JPKab 9y ago
Something else that throws me off on this: My team's experience with MLlib has been bad. Especially compared to H2O Sparkling Water. Anyone else on here find MLlib to not be as good as advertised? I was surprised to see Uber using it.
- thecity2 9y ago
  Spark is good if it fits your use case (or if you can make it fit your use case). We have definitely found that scaling can be a real issue though.

visarga 9y ago

It would be more useful to publish anonymized datasets, if they got any useful data.

sandGorgon 9y ago

>For every model that is trained in Michelangelo, we store a versioned object in our model repository in Cassandra.. >For important model types, we provide sophisticated visualization tools to help modelers understand why a model behaves as it does, as well as to help debug it if necessary How is this done - what serialisation format (pickle, pmml,etc) is being used? And more importantly, how are you tracing into the model ? It looks like this is a Spark based framework

Eridrus 9y ago
It's probably being "serialized" as a JAR file which just contains the code you call into. That was Spark's online deployment story last I heard, and jives with them saying the library mode has a Java API.

andrewfong 9y ago

Are there any ML-as-a-service platforms that offers something comparable to what Uber has here? And importantly, do you actually like using them? That is, suppose I'm willing to do the following: * Send said service a firehose of data * Write code for feature selection, data cleanup, transformations, etc. -- i.e. the sort of code you'd expect to find in a data scientist's Jupyter notebook. * Specify the type of ML model and parameters I'd like to experiment with. Is there a service that handles data warehousing, serving up predictions via a REST API, deployment of new models, reporting model performance over time, and general scaling issues? I know it's possible to set up a large portion of this using AWS services, but ideally, it'd be nice to have something with a Heroku-like ease-of-use, but for ML.

mobileexpert 9y ago
Azure ML
mayank 9y ago
Not quite there, but parts are: * You can send your firehose to BigQuery, and write Javascript UDFs to transform the data. * You can use a service like DataRobot to automatically train a bunch of models. * You can deploy the trained models using a serverless platform like Lambda. Nothing that's one-size-fits-all, but watch out for trends in that direction as more ML workflows get standardized.
romankolpak 9y ago
You should take a look at DataRobot, it does most of the things you've listed.
ANaimi 9y ago
Algorithmia Enterprise (algorithmia.com/enterprise). Inference- and deep learning-centric. (I'm part of the team).
wmeddie 9y ago
Skymind's SKIL does this. (I work on the team.) You don't have to send it your data though, you can just install it inside your system and have it connect to the data source or stream to train models (optionally doing hyper-parameter search), deploy them to REST servers and monitor their performance.

strin 9y ago

A random idea: in this article, they talked about a Feature Store shared across teams. Feature extraction also seems a major bottleneck in their modeling. It seems a very natural and valuable thing to have a platform to share "generic features". Yann LeCun said once a deep neural network predicts many targets, say 1000 classes in ImageNet, it is possible for the model to learn quite generic features. So it makes sense to pre-train on a large amount of data and a reasonable number of targets, and then share the learned feature extractors with others. Could this be a business? Or a community? Thoughts?

Eridrus 9y ago
Pretrained models for generic things exist publicly, e.g. models trained on ImageNet, or pretrained word vectors. Not a whole lot of general use in having a feature extractor tuned to Uber's data.

manugarri 9y ago

So I have seen this Michelangelo pop up in other articles. The key question is? Is Uber going to open source it? If not, why bother writing articles about the specifics of their platform?

sunsetMurk 9y ago
PR & attracting talent? Though open sourcing would be great and do the same thing, but more so.

nl 9y ago

This solves a real problem. ML pipelines are complicated, and often involve multiple languages and platforms (just as described in this post). But.. there are existing solutions in this area. Notable Luigi[1] (from Spotify) and Airflow[2] (from AirBNB) both seem to have a lot of overlap with this. I'm most familiar with Luigi, and it does many of the things that are listed here. Some of the model visualisations look pretty nice, and don't come out-of-the-box in the other platforms. So I'm not really sure what makes this unique. [1] https://github.com/spotify/luigi https://github.com/spotify/luigi [2] https://airflow.incubator.apache.org/ https://airflow.incubator.apache.org/

gipp 9y ago
What? Luigi and Airflow are ETL frameworks. They express dependencies between tasks, schedule their execution, and that's about it. Model storage and versioning, feature definition and extraction (batch and real-time), model deployment... Luigi and Airflow do almost nothing described in this post.
- nl 9y ago
  They do model/feature storage and versioning just fine. Or at least Luigi does - I use it daily. And feature extraction and modelling is my main usecase. It works really well, across Spark, Scikit and R, saving data in HDFS. Some examples: https://blog.dominodatalab.com/luigi-pipelines-domino/ https://blog.dominodatalab.com/luigi-pipelines-domino/ http://blog.richardweiss.org/2016/10/13/kaggle-with-luigi.html http://blog.richardweiss.org/2016/10/13/kaggle-with-luigi.ht... https://github.com/Atreya22/luigi_rosmann_sales https://github.com/Atreya22/luigi_rosmann_sales
  - gipp 9y ago
    Yeah, it does all those things, as long as you write all the code, set up all the systems for it to interact with, and manage all the logic yourself. That's like saying Python does all those things.
    - 9y ago

atonse 9y ago

This reminds me of LivingSocial. For years, LivingSocial was like a mini black hole of really good startup-like talent in the DC area. But when you have that many smart people under one roof, you have to keep them busy. So you get these kinds of reinventing-a-beautiful-wheel projects. Not knocking the quality, just knocking the growing distraction and temptation to build in-house rather than use existing tech. LivingSocial had such good quality output, but they were all over the place. They had to keep all those productive people busy after all. And the best way was to keep churning out new products.

JPKab 9y ago
Yeah, as a techie that used to live in DC, LivingSocial people were always at the Meetups. I was always blow away by the talent being used to work on a shitty coupon app.
hitekker 9y ago
I agree. "Keeping them busy" also raises an uncomfortable question: if Uber and other valuable Unicorns can be executed 99.999% with 25% of the engineers they hire, do they need the next 75% of engineers? Even considering crunch-time and potential diversification/expansion, full-time employment of non-critical engineers is obviously inefficient. And, as Uber proved, politically toxic and consequently harmful to the productivity of the business. More pressing is the existential tension underlying this quandary. That our economy will generally and increasingly forgo the skilled and unskilled labor provided by our society. Framed concretely: one day, we may define a "10x engineer" as an engineer who can send ten other engineers to the soup kitchen. I'm still looking for a well-thought-out solution.
- xixi77 9y ago
  This is not really new; just think of all the stuff that came out of AT&T labs...

rodionos 9y ago

> batch jobs run on a schedule or a trigger and are integrated with data quality monitoring tools to quickly detect regressions in the pipeline–either due to local or upstream code or data issues I'm curious what tooling they're using to ensure data quality, in particular time series data.

strictnein 9y ago

SAP has Leonardo. Uber has Michelangelo. Who has Donatello and Raphael?

oculusthrift 9y ago
Personally i don't get the obsession with ninja turtles.... /s
- omot 9y ago
  Funny you thought of Ninja Turtles, I was thinking Renaissance artists.
  - strictnein 9y ago
    Then this Epic Rap Battle of History is for you (salty language warning): https://www.youtube.com/watch?v=6HZ5V9rT96M https://www.youtube.com/watch?v=6HZ5V9rT96M
tomdre 9y ago
Splinter is the best

thecity2 9y ago

This doesn't seem to be an open source project, unless I'm missing something. It's kind of a show-and-tell basically.