3 ms·

Does tesla use transformers for the auto pilot?

sillysaurusx 5y ago
Doubtful. The biggest downside of transformers for vision is how ungodly long they take to produce results. Tesla has to operate in realtime.
- gamegoblin 5y ago
  In Karpathy’s recent AI day presentation he specifically stated they use transformers. But not on the raw camera input — they use regnets for that. The transformers come higher up the stack: https://youtu.be/j0z4FweCy4M https://youtu.be/j0z4FweCy4M Transformers mentioned on the slide at timestamp 1:00:18.
  - sillysaurusx 5y ago
    Thank you!
  - tandem5000 5y ago
    They use the key-value lookup/routing mechanism from Transformers to predict pixel-wise labels in bird view (lane, car, obstacle, intersection etc.). The motivation here is that some of the predictions may temporarily be occluded, so for predicting these occluded areas it may be particularly helpful to attend to remote regions in the input images which requires long-range dependencies that highly depend on the input itself (e.g. on whether there is an occlusion), which is exactly where the key-value mechanism excels. Not sure they even process past camera frames at this point. They only mention that later in the pipline they have an LSTM-like NN incorporating past camera frames (Schmidhuber will be proud!!). Edit: A random observation which just occurred to me is that their predictions seem surprisingly temporally unstable. Observe, for example, the lane layout wildly changing while it drives makes a left-turn at the intersection (https://youtu.be/j0z4FweCy4M?t=2608 https://youtu.be/j0z4FweCy4M?t=2608). You can use the comma and period keys to step through the video frame-by-frame.
dexter89_kp3 5y ago
They do. Karpathy spoke about it in on Tesla AI day. They use it for transforming image space to a vector space. See: https://youtu.be/j0z4FweCy4M https://youtu.be/j0z4FweCy4M (timestamp 54.40 onwards)
- sillysaurusx 5y ago
  Can you provide a more specific timestamp? 54.40 doesn't seem to mention anything about transformers, and "onwards" is two hours. I'd be really surprised if they use transformers due to how computationally expensive they are for anything involving vision. EDIT: Found it. 1h: https://www.youtube.com/watch?v=j0z4FweCy4M?t=1h https://www.youtube.com/watch?v=j0z4FweCy4M?t=1h Fascinating. I guess transformers are efficient.
  - dexter89_kp3 5y ago
    I gave the timestamp where they start talking about the problem they are trying to solve using transformers. As you said it is around 1hr mark
deleted 5y ago
[deleted]

Doubtful. The biggest downside of transformers for vision is how ungodly long they take to produce results. Tesla has to operate in realtime.

gamegoblin 5y ago
In Karpathy’s recent AI day presentation he specifically stated they use transformers. But not on the raw camera input — they use regnets for that. The transformers come higher up the stack: https://youtu.be/j0z4FweCy4M https://youtu.be/j0z4FweCy4M Transformers mentioned on the slide at timestamp 1:00:18.
- sillysaurusx 5y ago
  Thank you!
- tandem5000 5y ago
  They use the key-value lookup/routing mechanism from Transformers to predict pixel-wise labels in bird view (lane, car, obstacle, intersection etc.). The motivation here is that some of the predictions may temporarily be occluded, so for predicting these occluded areas it may be particularly helpful to attend to remote regions in the input images which requires long-range dependencies that highly depend on the input itself (e.g. on whether there is an occlusion), which is exactly where the key-value mechanism excels. Not sure they even process past camera frames at this point. They only mention that later in the pipline they have an LSTM-like NN incorporating past camera frames (Schmidhuber will be proud!!). Edit: A random observation which just occurred to me is that their predictions seem surprisingly temporally unstable. Observe, for example, the lane layout wildly changing while it drives makes a left-turn at the intersection (https://youtu.be/j0z4FweCy4M?t=2608 https://youtu.be/j0z4FweCy4M?t=2608). You can use the comma and period keys to step through the video frame-by-frame.

dexter89_kp3 5y ago

They do. Karpathy spoke about it in on Tesla AI day. They use it for transforming image space to a vector space. See: https://youtu.be/j0z4FweCy4M https://youtu.be/j0z4FweCy4M (timestamp 54.40 onwards)

sillysaurusx 5y ago
Can you provide a more specific timestamp? 54.40 doesn't seem to mention anything about transformers, and "onwards" is two hours. I'd be really surprised if they use transformers due to how computationally expensive they are for anything involving vision. EDIT: Found it. 1h: https://www.youtube.com/watch?v=j0z4FweCy4M?t=1h https://www.youtube.com/watch?v=j0z4FweCy4M?t=1h Fascinating. I guess transformers are efficient.
- dexter89_kp3 5y ago
  I gave the timestamp where they start talking about the problem they are trying to solve using transformers. As you said it is around 1hr mark

deleted 5y ago

[deleted]