4 ms·

>A disturbing thing is that the architecture is much less novel than I originally thought it would be, so this shows perhaps one of the major difficulties was h

>A disturbing thing is that the architecture is much less novel than I originally thought it would be, so this shows perhaps one of the major difficulties was having the resources to try different things on a massive set of multiple alignments. A similar concern has sparked some worries about "AI overhang" https://www.lesswrong.com/posts/75dnjiD8kv2khe9eQ/measuring-hardware-overhang https://www.lesswrong.com/posts/75dnjiD8kv2khe9eQ/measuring-... Most of the compute in ML research seems to be going into architecture search. Once the architecture is found, training and net finetuning/transfer learning is comparatively cheap, and then inference is cheaper still. This implies we could see 10-100x gains in AI algorithms using today's hardware, or sudden surprising appearance of AI dominance in an unexpected field. (Object grasping in unstructured environments? Art synthesis?) A task could go from totally impossible to trivial in a year. In retrospect, the EfficientNet scaling graph should have alarmed more people than it did: https://learnopencv.com/wp-content/uploads/2019/06/EfficientNet-performance-1024x771.png https://learnopencv.com/wp-content/uploads/2019/06/Efficient... Waymo has been puttering along for years, not announcing much of interest. This may have caused some complacency about self-driving cars, which is a mistake. Algorithms only get better, while humans stay the same. Once Waymo can replace some human drivers some of the time, things will start changing very quickly.

nl 5y ago
> Most of the compute in ML research seems to be going into architecture search. No it's not. Only Google spends significant time with automatic architecture search, and many people think this is really to try to sell cloud capacity. > Once the architecture is found, training and net finetuning/transfer learning is comparatively cheap Training isn't cheap for significant problems. Getting the data is very expensive, and compute is a significant expense for large datasets. > This implies we could see 10-100x gains in AI algorithms using today's hardware Actually, most of the time we see 10-100% (percent! not times) gains from architecture improvements, whether they be manual or automatic. But that is very significant, because a 10% improvement can suddenly make something useful that wasn't before.
- neolog 5y ago
  parent's EfficientNet graph seems really dramatic. Is it misleading somehow?
  - nl 5y ago
    It's not misleading assuming you know the field. If you don't you might not realize they are comparing ResNet (designed for ultimate performance) vs EffcientNet to show how close it gets in accuracy in the FLOPs budget. Note that the best accuracy for EffcientNet is roughly 1-2% better than ResNet/ResNeXT/SENet/etc but does have a much better FLOPs budget. But these other architectures were never optimised for FLOPs. It wasn't event a consideration when designing them. And EffcientNet is about a (manually designed) technique for scaling neural networks up in accuracy. Only EffcientNet-B0 is designed by AutoML, the others are scaled up. See the paper[1] for complete details. Like-for-like should be against MobileNet etc. EfficentNet is better, but the comparison is more reasonable in general. https://machinethink.net/blog/mobile-architectures/ https://machinethink.net/blog/mobile-architectures/ is a really good overview. [1] https://arxiv.org/pdf/1905.11946.pdf https://arxiv.org/pdf/1905.11946.pdf
- Tenoke 5y ago
  >No it's not. Only Google spends significant time with automatic architecture search, and many people think this is really to try to sell cloud capacity. Maybe not automatic architecture search but a lot does go into testing different architectures and changes to them in a more manual manner. Though yes, those tests are run on a smaller scale so for those huge models, training will be a bigger portion compared to architecture search than for smaller models.
jefftk 5y ago
> Once Waymo can replace some human drivers some of the time, things will start changing very quickly. But that happened 1y+ ago [1][2] without much changing since? [1] https://www.theverge.com/2019/12/9/21000085/waymo-fully-driverless-car-self-driving-ride-hail-service-phoenix-arizona https://www.theverge.com/2019/12/9/21000085/waymo-fully-driv... [2] https://blog.waymo.com/2020/10/waymo-is-opening-its-fully-driverless.html https://blog.waymo.com/2020/10/waymo-is-opening-its-fully-dr...
evouga 5y ago
Self-driving is a cursed problem, but the thorniest obstacles relate to optics and politics and not technology. AI can already replace some human drivers some of the time; but it doesn’t matter because each news story about a Tesla killing a passenger or driving into a parked fire truck sets back public acceptance of self-driving cars by years.
- boublepop 5y ago
  True, and Tesla is so far behind competitors when it comes to self driving that they knowingly push the envelope, because either they succeed, or they kill their customers but hurt all self-driving companies. So both scenarios work to their advantage and they just pay the minor fines and compensations to the people the kill in the process. And the Elon posts some Tweets blaming the people who got killed and his fandome happily cheers. There is a reason Waymo is progressing at what looks like snail speed from outside observers, and that reason is that it’s the only ethical pace for development and testing of what could eventually become broadly available fully autonomous vehicles.
  - pelorat 5y ago
    Teslas philosophy is that self driving is useless if it can't handle every road and route out there, while Waymo who already has a ride hailing service, relies on pre-mapped routes and databases. Both have their advantages and disadvantages. Waymo requires huge databases and a constant network connection. But it's good enough to be used in real-life without a backup driver, in certain select locations that is. If there's a road construction somewhere, don't expect the Waymo car to handle it unless they update their database. Tesla on the other hand attempts to solve the problem with minimal database use, where most of the info coming from road markings and traffic signs. A much harder problem.
    - plankers 5y ago
      Tesla's approach, however, kills people. Waymo's has not as of yet.

nl 5y ago

> Most of the compute in ML research seems to be going into architecture search. No it's not. Only Google spends significant time with automatic architecture search, and many people think this is really to try to sell cloud capacity. > Once the architecture is found, training and net finetuning/transfer learning is comparatively cheap Training isn't cheap for significant problems. Getting the data is very expensive, and compute is a significant expense for large datasets. > This implies we could see 10-100x gains in AI algorithms using today's hardware Actually, most of the time we see 10-100% (percent! not times) gains from architecture improvements, whether they be manual or automatic. But that is very significant, because a 10% improvement can suddenly make something useful that wasn't before.

neolog 5y ago
parent's EfficientNet graph seems really dramatic. Is it misleading somehow?
- nl 5y ago
  It's not misleading assuming you know the field. If you don't you might not realize they are comparing ResNet (designed for ultimate performance) vs EffcientNet to show how close it gets in accuracy in the FLOPs budget. Note that the best accuracy for EffcientNet is roughly 1-2% better than ResNet/ResNeXT/SENet/etc but does have a much better FLOPs budget. But these other architectures were never optimised for FLOPs. It wasn't event a consideration when designing them. And EffcientNet is about a (manually designed) technique for scaling neural networks up in accuracy. Only EffcientNet-B0 is designed by AutoML, the others are scaled up. See the paper[1] for complete details. Like-for-like should be against MobileNet etc. EfficentNet is better, but the comparison is more reasonable in general. https://machinethink.net/blog/mobile-architectures/ https://machinethink.net/blog/mobile-architectures/ is a really good overview. [1] https://arxiv.org/pdf/1905.11946.pdf https://arxiv.org/pdf/1905.11946.pdf
Tenoke 5y ago
>No it's not. Only Google spends significant time with automatic architecture search, and many people think this is really to try to sell cloud capacity. Maybe not automatic architecture search but a lot does go into testing different architectures and changes to them in a more manual manner. Though yes, those tests are run on a smaller scale so for those huge models, training will be a bigger portion compared to architecture search than for smaller models.

jefftk 5y ago

> Once Waymo can replace some human drivers some of the time, things will start changing very quickly. But that happened 1y+ ago [1][2] without much changing since? [1] https://www.theverge.com/2019/12/9/21000085/waymo-fully-driverless-car-self-driving-ride-hail-service-phoenix-arizona https://www.theverge.com/2019/12/9/21000085/waymo-fully-driv... [2] https://blog.waymo.com/2020/10/waymo-is-opening-its-fully-driverless.html https://blog.waymo.com/2020/10/waymo-is-opening-its-fully-dr...

evouga 5y ago

Self-driving is a cursed problem, but the thorniest obstacles relate to optics and politics and not technology. AI can already replace some human drivers some of the time; but it doesn’t matter because each news story about a Tesla killing a passenger or driving into a parked fire truck sets back public acceptance of self-driving cars by years.

boublepop 5y ago
True, and Tesla is so far behind competitors when it comes to self driving that they knowingly push the envelope, because either they succeed, or they kill their customers but hurt all self-driving companies. So both scenarios work to their advantage and they just pay the minor fines and compensations to the people the kill in the process. And the Elon posts some Tweets blaming the people who got killed and his fandome happily cheers. There is a reason Waymo is progressing at what looks like snail speed from outside observers, and that reason is that it’s the only ethical pace for development and testing of what could eventually become broadly available fully autonomous vehicles.
- pelorat 5y ago
  Teslas philosophy is that self driving is useless if it can't handle every road and route out there, while Waymo who already has a ride hailing service, relies on pre-mapped routes and databases. Both have their advantages and disadvantages. Waymo requires huge databases and a constant network connection. But it's good enough to be used in real-life without a backup driver, in certain select locations that is. If there's a road construction somewhere, don't expect the Waymo car to handle it unless they update their database. Tesla on the other hand attempts to solve the problem with minimal database use, where most of the info coming from road markings and traffic signs. A much harder problem.
  - plankers 5y ago
    Tesla's approach, however, kills people. Waymo's has not as of yet.