4 ms·

I am surprised that massive computation in one place is currently the best performing approach to this. We are searching for an algorithm which turns camera da

by TekMol 5y ago

I am surprised that massive computation in one place is currently the best performing approach to this. We are searching for an algorithm which turns camera data into a vector model, right? This sounds very parallelizable. Have a lot of computers travel the search space individually. When one finds a algorithm better than the current best solution, have it call the others "Hey guys, this algo works better then what we got so far. Everyone iterate on this one now!".

bdamm 5y ago
To farm out to cloud computing resources? Very expensive. It isn’t surprising to me that a company with a massive and known compute load would choose to pack that compute resource as efficiently as possible. Supercomputers are still a business.
- dannyw 5y ago
  Cloud computing is also like 100-200% surcharged, even after volume discounting. If you are not rapidly growing with uncertainty over resource requirements, you're going to get far better financial outcomes by taking your infra in-house. Example: Uber.
  - tetsuto 5y ago
    I'd slightly disagree with this - the real benefits from cloud are in the things you don't need to do. If you're just using the cloud as a normal datacentre (but in another location) you're not going to have a good time. EG using a managed Database as opposed to rolling your own saves on patching, management etc...
    - dannyw 5y ago
      If you're not rapidly growing and are paying millions in cloud compute, you should hire your own infra team and start managing your own database, so you can save millions of dollars.
krasin 5y ago
ML training is a latency sensitive problem that benefits from locality: most of the learning algorithms wait for all parallel workers to send gradients back before updating weights. While workarounds exist, life is easier without them.
- Mizza 5y ago
  Why is this a necessary constraint? (Not trolling, just curious if this has been investigated.) Our neurons don't have a global clock, it doesn't seem to be a problem for us. My intuition is that as long as the input is changing continuously and not through random presentation, and at a rate where big changes happen at a lower order of magnitude than the average compute rate, it wouldn't matter all that much in terms of accuracy.
modeless 5y ago
Stochastic gradient descent is not embarrassingly parallel, because it's an iterative process where each step depends on the previous one. If you figure out a better optimization algorithm that is more parallelizable, the world will beat a path to your door.
- TekMol 5y ago
  So we build self driving software by creating one big brain that does gradient descent. Would we have been more effective if we built other things like this? 400,000 people were involved in the Apollo program. Would it have been better if we had just one person with a really big brain? How about the Linux kernel and Wikipedia? And why should I figure out a better algorithm? Shouldn't that be the job of the one guy with the biggest brain who comes up with everything?
  - IshKebab 5y ago
    Uhm it's a lot easier to make a single supercomputer a million times faster than it is to make a single brain a million times cleverer.
    - TekMol 5y ago
      What is the difference? Couldn't evolution have settled on bigger brains if they are an advantage? Why all that slow interpersonal communication if it is more efficient to have the combined thought processes inside a single brain?
      - alpaca128 5y ago
        Evolution doesn't care about brains. They're just an accidental byproduct that happens to improve our chances of survival.
      - BenoitP 5y ago
        I think you might be interested in learning about Amdahl's law: https://en.wikipedia.org/wiki/Amdahl%27s_law https://en.wikipedia.org/wiki/Amdahl%27s_law Some parts of an algorithm are parallelizable, but only up to a certain limit; and that's where the endgame bottleneck is: the last remaining 'in series' parts and the communication overhead for the parallelized parts. Note that their Dojo system has absolutely massive networking equipment: that's for having a better endgame at Amdahl's law. ---- On another note, one can modify the algorithm itself so that is has more inherent parallelism. For example DNA sequencing: you match short reads, them send them off to be reconciled together. The way to have more parallelism is to generate more matches -by lowering the threshold for matching-. Having more matches means more communication costs, but better parallelism. For gradient descent, one of the tricks is learning in batches of input data: the neural network doesn't learn from the freshest point of view, but it can be done in parallel (also the way to aggregate new knowledge is to do an average of the weights, which must limit the amount of things learned)
gzer0 5y ago
This is of relevance: "To get rid of the dependency on the radar sensor for the pilot, we generated over 10 billion labels across two and a half million clips. And so to do that, we had to skill our huge offline neural networks and our simulation engine across 1000s of GPUs, and just a little bit shy of 20,000 CPU cores. On top of that, we also included over 2000, actual autopilot full self driving computers in the loop with our simulation engine. And that's our smallest compute cluster. So I'd like to give you some idea of what it takes to take our neural networks and move them in the car. And so the the two main constraints that we're working on there here are mostly latency and framerate, which are very important for safety, but also to get proper estimates of acceleration and velocity all of our surroundings. And so the meat of the problem really is around the AI compiler that we write and extend here within the group that essentially maps the compute operations from a pytorch model, to a set of dedicated and accelerated pieces of hardware. And we do that by figuring out a schedule that's optimized for throughput while working on very severe SRAM constraints. And so by the way, we're not doing that just on one engine, but across two engines on the autopilot computer. And the way we use those engines here at Tesla is such that, at any given time, only one of them will actually output control commands to the vehicle, while the other one is used as an extension of compute. But those rules are interchangeable, both on the hardware and software level. So how do we very quickly together as a group to this AI development cycles? Well, first, we have been scaling our capacity to evaluate our software neural network dramatically over the past few years. And today, we are running over a million evaluations per week on any code change that the team is producing. And those evaluations run on over 3000 actual full self driving computers that are hooked up together in a dedicated cluster."

by TekMol 5y ago

bdamm 5y ago

To farm out to cloud computing resources? Very expensive. It isn’t surprising to me that a company with a massive and known compute load would choose to pack that compute resource as efficiently as possible. Supercomputers are still a business.

dannyw 5y ago
Cloud computing is also like 100-200% surcharged, even after volume discounting. If you are not rapidly growing with uncertainty over resource requirements, you're going to get far better financial outcomes by taking your infra in-house. Example: Uber.
- tetsuto 5y ago
  I'd slightly disagree with this - the real benefits from cloud are in the things you don't need to do. If you're just using the cloud as a normal datacentre (but in another location) you're not going to have a good time. EG using a managed Database as opposed to rolling your own saves on patching, management etc...
  - dannyw 5y ago
    If you're not rapidly growing and are paying millions in cloud compute, you should hire your own infra team and start managing your own database, so you can save millions of dollars.

krasin 5y ago

ML training is a latency sensitive problem that benefits from locality: most of the learning algorithms wait for all parallel workers to send gradients back before updating weights. While workarounds exist, life is easier without them.

Mizza 5y ago
Why is this a necessary constraint? (Not trolling, just curious if this has been investigated.) Our neurons don't have a global clock, it doesn't seem to be a problem for us. My intuition is that as long as the input is changing continuously and not through random presentation, and at a rate where big changes happen at a lower order of magnitude than the average compute rate, it wouldn't matter all that much in terms of accuracy.

modeless 5y ago

Stochastic gradient descent is not embarrassingly parallel, because it's an iterative process where each step depends on the previous one. If you figure out a better optimization algorithm that is more parallelizable, the world will beat a path to your door.

TekMol 5y ago
So we build self driving software by creating one big brain that does gradient descent. Would we have been more effective if we built other things like this? 400,000 people were involved in the Apollo program. Would it have been better if we had just one person with a really big brain? How about the Linux kernel and Wikipedia? And why should I figure out a better algorithm? Shouldn't that be the job of the one guy with the biggest brain who comes up with everything?
- IshKebab 5y ago
  Uhm it's a lot easier to make a single supercomputer a million times faster than it is to make a single brain a million times cleverer.
  - TekMol 5y ago
    What is the difference? Couldn't evolution have settled on bigger brains if they are an advantage? Why all that slow interpersonal communication if it is more efficient to have the combined thought processes inside a single brain?
    - alpaca128 5y ago
      Evolution doesn't care about brains. They're just an accidental byproduct that happens to improve our chances of survival.
    - BenoitP 5y ago
      I think you might be interested in learning about Amdahl's law: https://en.wikipedia.org/wiki/Amdahl%27s_law https://en.wikipedia.org/wiki/Amdahl%27s_law Some parts of an algorithm are parallelizable, but only up to a certain limit; and that's where the endgame bottleneck is: the last remaining 'in series' parts and the communication overhead for the parallelized parts. Note that their Dojo system has absolutely massive networking equipment: that's for having a better endgame at Amdahl's law. ---- On another note, one can modify the algorithm itself so that is has more inherent parallelism. For example DNA sequencing: you match short reads, them send them off to be reconciled together. The way to have more parallelism is to generate more matches -by lowering the threshold for matching-. Having more matches means more communication costs, but better parallelism. For gradient descent, one of the tricks is learning in batches of input data: the neural network doesn't learn from the freshest point of view, but it can be done in parallel (also the way to aggregate new knowledge is to do an average of the weights, which must limit the amount of things learned)

gzer0 5y ago

This is of relevance: "To get rid of the dependency on the radar sensor for the pilot, we generated over 10 billion labels across two and a half million clips. And so to do that, we had to skill our huge offline neural networks and our simulation engine across 1000s of GPUs, and just a little bit shy of 20,000 CPU cores. On top of that, we also included over 2000, actual autopilot full self driving computers in the loop with our simulation engine. And that's our smallest compute cluster. So I'd like to give you some idea of what it takes to take our neural networks and move them in the car. And so the the two main constraints that we're working on there here are mostly latency and framerate, which are very important for safety, but also to get proper estimates of acceleration and velocity all of our surroundings. And so the meat of the problem really is around the AI compiler that we write and extend here within the group that essentially maps the compute operations from a pytorch model, to a set of dedicated and accelerated pieces of hardware. And we do that by figuring out a schedule that's optimized for throughput while working on very severe SRAM constraints. And so by the way, we're not doing that just on one engine, but across two engines on the autopilot computer. And the way we use those engines here at Tesla is such that, at any given time, only one of them will actually output control commands to the vehicle, while the other one is used as an extension of compute. But those rules are interchangeable, both on the hardware and software level. So how do we very quickly together as a group to this AI development cycles? Well, first, we have been scaling our capacity to evaluate our software neural network dramatically over the past few years. And today, we are running over a million evaluations per week on any code change that the team is producing. And those evaluations run on over 3000 actual full self driving computers that are hooked up together in a dedicated cluster."