3 ms·
In Karpathy’s recent AI day presentation he specifically stated they use transformers. But not on the raw camera input — they use regnets for that. The transfo
by gamegoblin 5y ago
In Karpathy’s recent AI day presentation he specifically stated they use transformers.
But not on the raw camera input — they use regnets for that. The transformers come higher up the stack:
https://youtu.be/j0z4FweCy4M https://youtu.be/j0z4FweCy4M
Transformers mentioned on the slide at timestamp 1:00:18.
- sillysaurusx 5y agoThank you!
- tandem5000 5y agoThey use the key-value lookup/routing mechanism from Transformers to predict pixel-wise labels in bird view (lane, car, obstacle, intersection etc.). The motivation here is that some of the predictions may temporarily be occluded, so for predicting these occluded areas it may be particularly helpful to attend to remote regions in the input images which requires long-range dependencies that highly depend on the input itself (e.g. on whether there is an occlusion), which is exactly where the key-value mechanism excels. Not sure they even process past camera frames at this point. They only mention that later in the pipline they have an LSTM-like NN incorporating past camera frames (Schmidhuber will be proud!!). Edit: A random observation which just occurred to me is that their predictions seem surprisingly temporally unstable. Observe, for example, the lane layout wildly changing while it drives makes a left-turn at the intersection (https://youtu.be/j0z4FweCy4M?t=2608 https://youtu.be/j0z4FweCy4M?t=2608). You can use the comma and period keys to step through the video frame-by-frame.