4 ms·
It's fun to see everybody succumbing to scaling laws. We will only have FSD when we can run realtime inference with a huge model. No amount of creativity seems
by rafaelero 4y ago
It's fun to see everybody succumbing to scaling laws. We will only have FSD when we can run realtime inference with a huge model. No amount of creativity seems to overcome the fact that bigger is better. So, let's wait Moore's law do its job.
- password321 4y agoWell look at this guy. Maybe you should take charge as the new director of AI seen as you figured it out.
- rafaelero 4y agoThanks, such an honor.
- learndeeply 4y agoUnless you've demonstrated that FSD follows scaling laws, this comment is pretty meaningless.
- rafaelero 4y agoI am not aware of any autoregressive transformer model that doesn't follow scaling laws. Tesla only needs to tokenize actions and images and voi la, self-driving capabilities. The problem, of course, is how you deploy such a big model so every car can run it locally.
- ShamelessC 4y agoScaling parameter count requires a similar increase in the amount of (accurate, labeled) data. This can be mitigated by “bootstrapping” techniques that make labeling new data easier, but is still likely the bottleneck for training such a model effectively (assuming they can probably spin up a supercomputer to scale their models otherwise).
- rafaelero 4y agoYou are correct. A Masked Language Model architecture would work very well here. Train it on labeled data and then bidirectionally infer missed spots. That's fun. We should join their AI team.
- ShamelessC 4y agoI don’t think I would do very well under Elon’s management style ha - the whole butts-in- seats from 8-6 thing is basically a dead end for me (personally). I’m not sure what architecture they use, but they do indeed already have a pre trained “auto-labeler” that their annotators use. My understanding is that due to hallucinations from the model and the risks involved with driving, they still need to be vetted manually before being added to the dataset.
- rafaelero 4y agoMakes sense. Fortunately, scale is less of a problem for the autolabeler, so they can put a lot of processing power on this step. But yeah, they will need human labelers for the edge cases where the model is unsure. I wish Tesla would publish their results so we could understand what they are doing and how much it is improving.