4 ms·
Unless you've demonstrated that FSD follows scaling laws, this comment is pretty meaningless.
by learndeeply 4y ago
Unless you've demonstrated that FSD follows scaling laws, this comment is pretty meaningless.
- rafaelero 4y agoI am not aware of any autoregressive transformer model that doesn't follow scaling laws. Tesla only needs to tokenize actions and images and voi la, self-driving capabilities. The problem, of course, is how you deploy such a big model so every car can run it locally.
- ShamelessC 4y agoScaling parameter count requires a similar increase in the amount of (accurate, labeled) data. This can be mitigated by “bootstrapping” techniques that make labeling new data easier, but is still likely the bottleneck for training such a model effectively (assuming they can probably spin up a supercomputer to scale their models otherwise).
- rafaelero 4y agoYou are correct. A Masked Language Model architecture would work very well here. Train it on labeled data and then bidirectionally infer missed spots. That's fun. We should join their AI team.
- ShamelessC 4y agoI don’t think I would do very well under Elon’s management style ha - the whole butts-in- seats from 8-6 thing is basically a dead end for me (personally). I’m not sure what architecture they use, but they do indeed already have a pre trained “auto-labeler” that their annotators use. My understanding is that due to hallucinations from the model and the risks involved with driving, they still need to be vetted manually before being added to the dataset.
- rafaelero 4y agoMakes sense. Fortunately, scale is less of a problem for the autolabeler, so they can put a lot of processing power on this step. But yeah, they will need human labelers for the edge cases where the model is unsure. I wish Tesla would publish their results so we could understand what they are doing and how much it is improving.