9 ms·

Scaling Machine Learning at Uber with Michelangelo

112 points by dpandya 8y ago

marmaduke 8y ago
I love what Uber does with machines (ML), hate what it (currently) does to people. We recently potted some models from Stan to Pyro (SVI on PyTorch), and it’s been reallly exciting (except for the dark corner of poutines), it really has the performance of something being used in production, except the occasional nan explosion. edit we are lazy and use our GitLab CI/CD to drive model development iteration. It’s not as fully featured as what’s in the article but it’s a zero effort start.
- mmq 8y ago
  Can you elaborate a bit more about your usage of GitLab CI/CD for model management/development. I am currently working on a platform [1] that tries to solve some of the issues mentioned in the article, i.e. improving data scientists' productivity and velocity, compare models, solve reproducibility issues... [1] https://github.com/polyaxon/polyaxon https://github.com/polyaxon/polyaxon
  - marmaduke 8y ago
    We uh treat models as code, but also have NFS shares setup for the storage and GitLab runner talking to a Slurm cluster to run the models. Results and cross validation upload to GitLab. Main thing we haven’t built out yet are performance dashboards for showing improvement across commits, but with the GitLab APIs that’s a script away (currently we do it by hand)
    - sandGorgon 8y ago
      Thanks for your reply. Actually the question was more around "how do you create your models and what do you mean treating them as code", "why slurm and not something like airflow" , "what is the test/performance setup - backtesting, smoke test" etc etc The Gitlab stuff is easier to understand.
      - marmaduke 8y ago
        Ah right, > how do you create your models and what do you mean treating them as code we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data, where we assess performance with LOO approximations for Bayesian models (notably PSIS) and some labeling from experts (which is not taken too seriously tbh) > why slurm and not something like airflow because the HPC resources we have access to are built with Slurm, which is super fast, supports DAGs of jobs, schedules our jobs reliably and quickly. I don't really want the other stuff on the Airflow feature list to be honest.
        sandGorgon 8y ago
        super interesting. thanks for sharing. >we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data This is the part that everyone seems reinventing. Have you looked at PyML (https://eng.uber.com/michelangelo-pyml/ https://eng.uber.com/michelangelo-pyml/). What are some of your learnings around jupyter -> production code. A lot of these are around conventions - "write a function called train(), fit(), test()". Is that the basis of your pipeline as well ?
        marmaduke 8y ago
        It’s not so simple for our models (hierarchical Bayesian time series models, often nonlinear, which may not be typical): we spend a lot of time digging through the data itself, forward simulations of model, and refactoring/tweaking model structure. PyML (as described in the link you provided) doesn’t appear to support the first two parts, which are prerequisites to improving the model IMO. Usually when we are doing more of the train/fit/test cycle, there’s an argparse script to quickly try different parameter values succinctly (which is run and tracked by the above CI setup) I wouldn’t say we’re reinventing since a better solution isn’t very clear (though PyML et al look interesting) edit forward simulation isn't a frequent thing in posts on generic ML algorithms, so just as an example: suppose you run a model and see an oscillatory component along a temporal dimensions in your residual error, and you add a oscillatory component to your model, and rerun it but still see a residual with an oscillation. You can run a forward simulation of your model to see what frequency it's predicting and check against what's seen in the data, and fix it. This is a contrived example but when you have multiple competing priors or model components, this is an effective way to debug their behavior.
    - ptd 8y ago
      What do you think are the best dashboard options for showing improvements?
      - marmaduke 8y ago
        I'd probably set up Grafana talking to Elasticsearch or PostGres, but I haven't thought too hard about it.
  - marmaduke 8y ago
    Polyaxon looks nice but we don’t admin the majority of the GPU resources we use (which is why being able to tell GitLab-runner to invoke Slurm is cool) Pachyderm is another one I’ve looked at but we don’t have the sys admin bandwidth for that stuff right now.
- sandGorgon 8y ago
  would love to know what is your model development iteration. especially how you do testing, etc
  - marmaduke 8y ago
    See my comment here, but I can answer other questions if you have them https://news.ycombinator.com/item?id=18376567 https://news.ycombinator.com/item?id=18376567
- mlthoughts2018 8y ago
  Why would you do this instead of using pymc3?
  - marmaduke 8y ago
    PyMC3 didn’t run well on GPUs last I tried. That may have changed but I find PyTorch easier to work with than Theano or TensorFlow.
    - mlthoughts2018 8y ago
      Just in case other readers stumble by, neither of these perceptions of pymc is accurate. GPU operability is well-supported, and much like Keras, pymc provides well-designed abstractions over top of TensorFlow, making the downsides of raw TensorFlow mostly irrelevant. I like PyTorch a lot too, but any time I see someone say PyTorch is easier than TensorFlow, it usually just means that person only tried PyTorch, learned some special knowledge about it, and now they don’t want to admit using a different framework might be the better choice, even if it requires giving up some of what’s nice about PyTorch.
      - marmaduke 8y ago
        That’s a fairly aggressive response. Both TF and Theano require static graph while PyTorch lets you use Python’s regular control flows (if, for, while, etc). This makes building modular model components much easier, since you can reason about execution mostly as if it’s normal numerical Python code. I have tried running PyMC3 models on GPUs (when they were on Theano; not sure if they have transitioned since) and it is slower than CPUs, not for small models but the big, SIMD-wide ones. When I ported the same thing to Pyro/PyTorch, it was clearly making good use of the GPU, not bottlenecked by useless CPU-GPU transfers Maybe that’s changed now, so as they say the only useful benchmark is your own code.
        mlthoughts2018 8y ago
        > “I have tried running PyMC3 models on GPUs (when they were on Theano; not sure if they have transitioned since) and it is slower than CPUs, not for small models but the big, SIMD-wide ones.“ Can you post a link to your code with some synthetic data of the sizes you’re talking about to demonstrate this? I hear it as a criticism a lot, but have never found it to be true (full disclosure: I work on a large-scale production system that uses pymc for huge Bayesian logistic regression and huge hierarchical models, both in GPU mode out of necessity). > “Both TF and Theano require static graph while PyTorch lets you use Python’s regular control flows (if, for, while, etc). This makes building modular model components much easier, since you can reason about execution mostly as if it’s normal numerical Python code.” I can’t tell if you’ve looked into pymc or not based on this (or Keras either for that matter), since in pymc, GPU mode is just a Theano setting, you don’t actually write any Theano code, manipulate any graphs or sessions directly, or anything else. You just call pm.sample with the appropriate mode settings at it is executed on the GPU. Much like with Keras, where you can also easily use Python native control flow, context managers and so on, pymc doesn’t require low-level usage of underlying computation graph abstractions. Again, I really like PyTorch too, but people just seem to have only ever tried PyTorch, liked one or two things about it, forgive the parts that are bad about it (like needing to explicitly write a wrapper for the backwards calculation for custom layers, which you don’t need to do in Keras for example), and generalize to criticize other tools.
- cocobongo 8y ago
  What does Uber currently do to people that you hate? Uber currently provides people with more than 2 Million jobs [1]. Uber drivers/couriers made almost $13 Billion in the US alone last year [2]. [1] https://medium.com/@gc/ubers-path-forward-b59ec9bd4ef6 https://medium.com/@gc/ubers-path-forward-b59ec9bd4ef6 [2] https://www.sfchronicle.com/business/article/Uber-drivers-in-Bay-Area-made-1-07-billion-last-13105699.php https://www.sfchronicle.com/business/article/Uber-drivers-in...
  - deleted 8y ago
    [deleted]
  - marmaduke 8y ago
    Treat people like freelancers while paying them like low-wage waiters? $13bn/2mn jobs is $6k/job/yr, not so impressive compared to welfare.
    - cocobongo 8y ago
      Personally I think Uber drivers/couriers are freelancers. Uber drivers can work whatever hours they like. If they want to work 1 hour a day, they can. If they want to work 10 hours a day, they can. That's not something that non-freelancers can do. For the same reason, I don't think it's fair to compare the average Uber driver salary to a full-time salary. Some Uber drivers work full-time, but I'd guess most don't. Lots probably only work a few hours a week. Uber provides students/parents/anyone with a way to make extra money on the side. Also, from the article I linked to, 900k US drivers make $13 Billion a year. So $14k-$15k/job/year. When you factor in that many (probably most) Uber drivers are only working part-time, that's significant income from a super flexible job.
      - marmaduke 8y ago
        > I think Uber drivers/couriers are freelancers yep so they should be compensated more, not less, in sight of the precarity of their job > When you factor in that many (probably most) Uber drivers are only working part-time, that's significant income from a super flexible job super flexible for whom? the drivers? or for Uber? There are two sides to the gig style work, one side is a corp that's got teams of PhDs and a cloud calculating its optimal risk-reward strategy, and the other side some poor people trying to make money. How this can possibly turn out well for the latter is a pipe dream.
- jquery 8y ago
  I like what Uber (currently) does to people. Gets passengers from point A to point B efficiently while saving them significant money in the process over alternatives. Metaphorically puts dinner on the table of hundreds of thousands of drivers. Literally puts dinner on the table of millions (UberEats). Has a business model that doesn't rely exposing more eyeballs to more ads, corrupting the press, media, and privacy in the process. Reduces car ownership and dependence. Moving towards encouraging people to ride green vehicles. Literally saves lives (reducing DUI). Yeah, I'm okay with the Uber of 2018.* *Disclaimer: I work at Uber, and my opinions are solely my own. We're hiring.
  - platz 8y ago
    "Silicon Valley innovation now is directly aimed at oppressing the underclass, and everybody knows it and can see it. They hate Uber. People hate Uber. It means the death of the era of good feelings that came with this constant Moore's Law style innovation. And that was an unforced error, by Silicon Valley. It was in their DNA. They didn't have to give Travis Kalanick, a guy they despised and never trusted, for good reason—They didn't have to give him all that venture capital. But they saw him as an expendable probe, so they cynically gave him money, to see how much law-breaking he could get away with in the name of their disruption activities. That was hubris—and nemesis is well on the way." - NEXT17 | Bruce Sterling | Live from 2027
    - jquery 8y ago
      In the same talk, Bruce Sterling also said, "Do what China says. It’s the ascendant model. It’s destroys the California ideology. The Silicon Valley companies can’t get a toe-hold there." As far as I can tell, the guy doesn't like America or even representative democracy very much. Take that for what you will.
    - woolvalley 8y ago
      Is it Uber / gig-economy apps you don't like, or the general idea of low income relatively unskilled labor jobs?
      - 8y ago
paulie_a 8y ago
It's kinda funny they tout their usage of GPS. I use Uber on a near daily basis and drivers by an large use Google maps. They have out right said "Uber sucks for directions" And if you use express pools it will always say to go the wrong side of an intersection. I like uber because of the drivers, but their fancy technology is flawed.
- srean 8y ago
  Indeed. What is even more strange about the use of google maps is that Uber bought Bing maps, I am sure for a hefty sum.
- martinald 8y ago
  I've never seen an Uber driver not use Waze in London.
  - melling 8y ago
    I was in Colombia and South Africa last year. Those Uber drivers also used Waze.
  - magoghm 8y ago
    In Mexico City I always see them use Waze.
- freyir 8y ago
  I believe they’re using GPS data here more for analytics, rather than navigation. They can use GPS data to chart usage metrics, plan pool rides, check for anomalies, and harass journalists, for example.
- googlemike 8y ago
  Please do not conflate GPS with navigation. There is a massive set of problems you can solve with high fidelity GPS Data (Uber knows it is a driver in a car, verifies it with another GPS entity (rider app reports GPS also), etc). There is not that much overlap between great GPS data and great maps - no amount of great GPS data will give you a good basemap. Please let me know if I am not making sense, I am more than happy to provide examples / explain further!
Tickon 8y ago
This is not a product, nor is it open sourced - so this is basically just a PR stunt. Or am I missing anything??
- typon 8y ago
  Looks like a blog post about an internal tool. Not sure why this is interesting to people

112 points by dpandya 8y ago

I love what Uber does with machines (ML), hate what it (currently) does to people. We recently potted some models from Stan to Pyro (SVI on PyTorch), and it’s been reallly exciting (except for the dark corner of poutines), it really has the performance of something being used in production, except the occasional nan explosion. edit we are lazy and use our GitLab CI/CD to drive model development iteration. It’s not as fully featured as what’s in the article but it’s a zero effort start.

mmq 8y ago
Can you elaborate a bit more about your usage of GitLab CI/CD for model management/development. I am currently working on a platform [1] that tries to solve some of the issues mentioned in the article, i.e. improving data scientists' productivity and velocity, compare models, solve reproducibility issues... [1] https://github.com/polyaxon/polyaxon https://github.com/polyaxon/polyaxon
- marmaduke 8y ago
  We uh treat models as code, but also have NFS shares setup for the storage and GitLab runner talking to a Slurm cluster to run the models. Results and cross validation upload to GitLab. Main thing we haven’t built out yet are performance dashboards for showing improvement across commits, but with the GitLab APIs that’s a script away (currently we do it by hand)
  - sandGorgon 8y ago
    Thanks for your reply. Actually the question was more around "how do you create your models and what do you mean treating them as code", "why slurm and not something like airflow" , "what is the test/performance setup - backtesting, smoke test" etc etc The Gitlab stuff is easier to understand.
    - marmaduke 8y ago
      Ah right, > how do you create your models and what do you mean treating them as code we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data, where we assess performance with LOO approximations for Bayesian models (notably PSIS) and some labeling from experts (which is not taken too seriously tbh) > why slurm and not something like airflow because the HPC resources we have access to are built with Slurm, which is super fast, supports DAGs of jobs, schedules our jobs reliably and quickly. I don't really want the other stuff on the Airflow feature list to be honest.
      - sandGorgon 8y ago
        super interesting. thanks for sharing. >we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data This is the part that everyone seems reinventing. Have you looked at PyML (https://eng.uber.com/michelangelo-pyml/ https://eng.uber.com/michelangelo-pyml/). What are some of your learnings around jupyter -> production code. A lot of these are around conventions - "write a function called train(), fit(), test()". Is that the basis of your pipeline as well ?
        marmaduke 8y ago
        It’s not so simple for our models (hierarchical Bayesian time series models, often nonlinear, which may not be typical): we spend a lot of time digging through the data itself, forward simulations of model, and refactoring/tweaking model structure. PyML (as described in the link you provided) doesn’t appear to support the first two parts, which are prerequisites to improving the model IMO. Usually when we are doing more of the train/fit/test cycle, there’s an argparse script to quickly try different parameter values succinctly (which is run and tracked by the above CI setup) I wouldn’t say we’re reinventing since a better solution isn’t very clear (though PyML et al look interesting) edit forward simulation isn't a frequent thing in posts on generic ML algorithms, so just as an example: suppose you run a model and see an oscillatory component along a temporal dimensions in your residual error, and you add a oscillatory component to your model, and rerun it but still see a residual with an oscillation. You can run a forward simulation of your model to see what frequency it's predicting and check against what's seen in the data, and fix it. This is a contrived example but when you have multiple competing priors or model components, this is an effective way to debug their behavior.
  - ptd 8y ago
    What do you think are the best dashboard options for showing improvements?
    - marmaduke 8y ago
      I'd probably set up Grafana talking to Elasticsearch or PostGres, but I haven't thought too hard about it.
- marmaduke 8y ago
  Polyaxon looks nice but we don’t admin the majority of the GPU resources we use (which is why being able to tell GitLab-runner to invoke Slurm is cool) Pachyderm is another one I’ve looked at but we don’t have the sys admin bandwidth for that stuff right now.
sandGorgon 8y ago
would love to know what is your model development iteration. especially how you do testing, etc
- marmaduke 8y ago
  See my comment here, but I can answer other questions if you have them https://news.ycombinator.com/item?id=18376567 https://news.ycombinator.com/item?id=18376567
mlthoughts2018 8y ago
Why would you do this instead of using pymc3?
- marmaduke 8y ago
  PyMC3 didn’t run well on GPUs last I tried. That may have changed but I find PyTorch easier to work with than Theano or TensorFlow.
  - mlthoughts2018 8y ago
    Just in case other readers stumble by, neither of these perceptions of pymc is accurate. GPU operability is well-supported, and much like Keras, pymc provides well-designed abstractions over top of TensorFlow, making the downsides of raw TensorFlow mostly irrelevant. I like PyTorch a lot too, but any time I see someone say PyTorch is easier than TensorFlow, it usually just means that person only tried PyTorch, learned some special knowledge about it, and now they don’t want to admit using a different framework might be the better choice, even if it requires giving up some of what’s nice about PyTorch.
    - marmaduke 8y ago
      That’s a fairly aggressive response. Both TF and Theano require static graph while PyTorch lets you use Python’s regular control flows (if, for, while, etc). This makes building modular model components much easier, since you can reason about execution mostly as if it’s normal numerical Python code. I have tried running PyMC3 models on GPUs (when they were on Theano; not sure if they have transitioned since) and it is slower than CPUs, not for small models but the big, SIMD-wide ones. When I ported the same thing to Pyro/PyTorch, it was clearly making good use of the GPU, not bottlenecked by useless CPU-GPU transfers Maybe that’s changed now, so as they say the only useful benchmark is your own code.
      - mlthoughts2018 8y ago
        > “I have tried running PyMC3 models on GPUs (when they were on Theano; not sure if they have transitioned since) and it is slower than CPUs, not for small models but the big, SIMD-wide ones.“ Can you post a link to your code with some synthetic data of the sizes you’re talking about to demonstrate this? I hear it as a criticism a lot, but have never found it to be true (full disclosure: I work on a large-scale production system that uses pymc for huge Bayesian logistic regression and huge hierarchical models, both in GPU mode out of necessity). > “Both TF and Theano require static graph while PyTorch lets you use Python’s regular control flows (if, for, while, etc). This makes building modular model components much easier, since you can reason about execution mostly as if it’s normal numerical Python code.” I can’t tell if you’ve looked into pymc or not based on this (or Keras either for that matter), since in pymc, GPU mode is just a Theano setting, you don’t actually write any Theano code, manipulate any graphs or sessions directly, or anything else. You just call pm.sample with the appropriate mode settings at it is executed on the GPU. Much like with Keras, where you can also easily use Python native control flow, context managers and so on, pymc doesn’t require low-level usage of underlying computation graph abstractions. Again, I really like PyTorch too, but people just seem to have only ever tried PyTorch, liked one or two things about it, forgive the parts that are bad about it (like needing to explicitly write a wrapper for the backwards calculation for custom layers, which you don’t need to do in Keras for example), and generalize to criticize other tools.
cocobongo 8y ago
What does Uber currently do to people that you hate? Uber currently provides people with more than 2 Million jobs [1]. Uber drivers/couriers made almost $13 Billion in the US alone last year [2]. [1] https://medium.com/@gc/ubers-path-forward-b59ec9bd4ef6 https://medium.com/@gc/ubers-path-forward-b59ec9bd4ef6 [2] https://www.sfchronicle.com/business/article/Uber-drivers-in-Bay-Area-made-1-07-billion-last-13105699.php https://www.sfchronicle.com/business/article/Uber-drivers-in...
- deleted 8y ago
  [deleted]
- marmaduke 8y ago
  Treat people like freelancers while paying them like low-wage waiters? $13bn/2mn jobs is $6k/job/yr, not so impressive compared to welfare.
  - cocobongo 8y ago
    Personally I think Uber drivers/couriers are freelancers. Uber drivers can work whatever hours they like. If they want to work 1 hour a day, they can. If they want to work 10 hours a day, they can. That's not something that non-freelancers can do. For the same reason, I don't think it's fair to compare the average Uber driver salary to a full-time salary. Some Uber drivers work full-time, but I'd guess most don't. Lots probably only work a few hours a week. Uber provides students/parents/anyone with a way to make extra money on the side. Also, from the article I linked to, 900k US drivers make $13 Billion a year. So $14k-$15k/job/year. When you factor in that many (probably most) Uber drivers are only working part-time, that's significant income from a super flexible job.
    - marmaduke 8y ago
      > I think Uber drivers/couriers are freelancers yep so they should be compensated more, not less, in sight of the precarity of their job > When you factor in that many (probably most) Uber drivers are only working part-time, that's significant income from a super flexible job super flexible for whom? the drivers? or for Uber? There are two sides to the gig style work, one side is a corp that's got teams of PhDs and a cloud calculating its optimal risk-reward strategy, and the other side some poor people trying to make money. How this can possibly turn out well for the latter is a pipe dream.
jquery 8y ago
I like what Uber (currently) does to people. Gets passengers from point A to point B efficiently while saving them significant money in the process over alternatives. Metaphorically puts dinner on the table of hundreds of thousands of drivers. Literally puts dinner on the table of millions (UberEats). Has a business model that doesn't rely exposing more eyeballs to more ads, corrupting the press, media, and privacy in the process. Reduces car ownership and dependence. Moving towards encouraging people to ride green vehicles. Literally saves lives (reducing DUI). Yeah, I'm okay with the Uber of 2018.* *Disclaimer: I work at Uber, and my opinions are solely my own. We're hiring.
- platz 8y ago
  "Silicon Valley innovation now is directly aimed at oppressing the underclass, and everybody knows it and can see it. They hate Uber. People hate Uber. It means the death of the era of good feelings that came with this constant Moore's Law style innovation. And that was an unforced error, by Silicon Valley. It was in their DNA. They didn't have to give Travis Kalanick, a guy they despised and never trusted, for good reason—They didn't have to give him all that venture capital. But they saw him as an expendable probe, so they cynically gave him money, to see how much law-breaking he could get away with in the name of their disruption activities. That was hubris—and nemesis is well on the way." - NEXT17 | Bruce Sterling | Live from 2027
  - jquery 8y ago
    In the same talk, Bruce Sterling also said, "Do what China says. It’s the ascendant model. It’s destroys the California ideology. The Silicon Valley companies can’t get a toe-hold there." As far as I can tell, the guy doesn't like America or even representative democracy very much. Take that for what you will.
  - woolvalley 8y ago
    Is it Uber / gig-economy apps you don't like, or the general idea of low income relatively unskilled labor jobs?
    - 8y ago

paulie_a 8y ago

It's kinda funny they tout their usage of GPS. I use Uber on a near daily basis and drivers by an large use Google maps. They have out right said "Uber sucks for directions" And if you use express pools it will always say to go the wrong side of an intersection. I like uber because of the drivers, but their fancy technology is flawed.

srean 8y ago
Indeed. What is even more strange about the use of google maps is that Uber bought Bing maps, I am sure for a hefty sum.
martinald 8y ago
I've never seen an Uber driver not use Waze in London.
- melling 8y ago
  I was in Colombia and South Africa last year. Those Uber drivers also used Waze.
- magoghm 8y ago
  In Mexico City I always see them use Waze.
freyir 8y ago
I believe they’re using GPS data here more for analytics, rather than navigation. They can use GPS data to chart usage metrics, plan pool rides, check for anomalies, and harass journalists, for example.
googlemike 8y ago
Please do not conflate GPS with navigation. There is a massive set of problems you can solve with high fidelity GPS Data (Uber knows it is a driver in a car, verifies it with another GPS entity (rider app reports GPS also), etc). There is not that much overlap between great GPS data and great maps - no amount of great GPS data will give you a good basemap. Please let me know if I am not making sense, I am more than happy to provide examples / explain further!

Tickon 8y ago

This is not a product, nor is it open sourced - so this is basically just a PR stunt. Or am I missing anything??

typon 8y ago
Looks like a blog post about an internal tool. Not sure why this is interesting to people