18 ms·
Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]
- mkagenius 7y agoOh he's no longer with OpenAI? Sam Altman must be worried about this..
- cyrux004 7y agofor over 2 years now
- timdorr 7y agoHe's been at Tesla for almost 2 and a half years now: https://techcrunch.com/2017/06/20/tesla-hires-deep-learning-expert-andrej-karpathy-to-lead-autopilot-vision/ https://techcrunch.com/2017/06/20/tesla-hires-deep-learning-...
- spectramax 7y agoWithout meaning offense to Sam, I thought he was an investor / YC head. What credentials does he have to be at OpenAI?
- jayparth 7y agoPerhaps founding the initiative....
- visarga 7y agoTeaching the CS231n course at Stanford, for one.
- spectramax 7y agoThanks, that makes sense. So now the question is - how did Sam become a YC/Angel investor from someone who taught a class at Stanford? I think we need an interview with Sam.
- mkagenius 7y agoHe did? I only knew of 183B which is "How to start a startup"
- _cs2017_ 7y agoAndrej Karpathy taught cs231n. The question is about Sam Altman who didn't teach anything related to NL. He's on the board of OpenAI because he was one of its founding investors.
- new_realist 7y agoElon Musk poached him, and for that was kicked off the OpenAI board.
- Pazzaz 7y agoI've never heard that that was the reason for Elon leaving the OpenAI board. The official announcement said "As Tesla continues to become more focused on AI, this will eliminate a potential future conflict for Elon." Do you have a source?
- slim 7y agothat statement is the politically correct version of "poached an employee"
- adamnemecek 7y agoThe trick for level 5 is learning the mapping between the lidar point cloud and the video stream. It’s the best of both worlds.
- pgodzin 7y agoTesla doesn't have have a lidar point cloud at all
- adamnemecek 7y agoThat’s the point. You train with lidar, deploy with cameras.
- djebdbebeejrrn 7y agoThe trick for level 5 is creating artificial general intelligence. In other words, level 5 will almost certainly not happen any time in the next 50 years. Musk is a con artist and Karpathy is one of his enablers.
- ojn 7y agoThat falls apart as soon as the map and the real world deviates and you need to drive based on what’s in front of you. Lidar helps you spot obstructions, but won’t tell you what they are and won’t help you figure out what to do to avoid them. Want an example? Cruise’s first real world demo got stuck behind a simple taco truck in downtown SF.
- antpls 7y agoThe parent meant : gather training data with Lidar and Camera, then build a model with that data to learn to reconstruct a 3D space only from Camera data, and then embed that model in the cars. Tesla is already using a model to rebuild a 3D space from Camera data only, the parent suggests to improve the quality of the transformation with high quality 3D representations from Lidar. It's deep learning all the way down.
- ben_w 7y ago
- jfoster 7y agoI wonder if the environment the car discovers includes elevation. Would be necessary for handling many carparks.
- timzaman 7y agoHis team is hiring; https://www.tesla.com/careers/job/software-engineerdeeplearning-49779 https://www.tesla.com/careers/job/software-engineerdeeplearn... https://www.tesla.com/careers/job/machine-learninginfrastructureengineerautopilot-48125 https://www.tesla.com/careers/job/machine-learninginfrastruc... https://www.tesla.com/careers/job/machine-learningscientistautopilot-48414 https://www.tesla.com/careers/job/machine-learningscientista...
- sdan 7y agoThanks Tim! Would love to apply, but still a student. Hoping to join in the future given how nicely orchestrated your team has been training nets.
- ojn 7y agoApply for internship. Doing well on an internship is a great way to get a foot in the door after graduation.
- throwaway010718 7y agoAny guess what the compensation is like for these positions ?
- drchewbacca 7y agoIf you get rejected just edit 10 words on your resume and then resubmit. Do this a 100hz and you'll get in eventually :)
- choppaface 7y agoLess than Uber, but more than Waymo, who is only offering ~$20k-ish stock packages like a late start-up who expects to 10x. Depends on how you value Tesla stock, though. See levels.fyi
- deleted 7y ago[deleted]
- sdan 7y agoReally liked this talk. Looks like they are really nicely orchestrating workloads and training on numerous nets asynchronously. As a person in the AV industry I think Tesla's ability to control the entire stack is great for Tesla... maybe not for everyone who can't afford/doesn't have a Tesla.
- natch 7y ago>maybe not for everyone who can't afford/doesn't have a Tesla. Affordability is not as much of an issue as some make it out to be. Cost-wise it's like owning a Camry or an Accord, if you go for the lower end models. If you mean not everyone can afford a new car, then sure I agree with you. Edit: if you think I'm wrong about this, please explain or ask me to clarify anything?
- sdan 7y agoAs a small anecdote, my parents couldn't afford/didn't want to spent over $30k for a car. Surely we could've gotten a Tesla for $5k+ more, but given the relatively new infrastructure with electric charging stations (and the fact that none are available in the apartment I live in) my parents didn't find all the new cool features appealing and instead got a regular Toyota Sienna that has nothing fancy, just enough to take the family around. Similarly, the infrastructure around electric charging stations I believe hasn't fully matured yet and as a result many people who've already owned a car, I believe will stick with gas cars since there's no huge incentive to change, unless it becomes easier to charge (faster, more convenient). Do note that I don't have a drivers license. I never intend on getting one (I believe in what I do in the AV industry). I'm just guessing on the habits of people, not that I have any real experience in buying a gas/electric car. Also note I didn't think you were wrong, not sure why the downvotes.
- natch 7y agoThanks for the reply. Living in an apartment is not a big issue for me... we have two, live in an apartment with no charger. That's interesting that you relate something you do in the AV industry to not ever getting a car... what's that about? I do think that it's possible in the future the majority of people will never need to own a car.
- suehebfbfjfrifh 7y agoI'd rather he talk about why he works for a con artist who will never create self driving cars (not that anyone else will in the next few decades) but keeps promising to do it.
- kegan 7y agoAnyone knows why Andrej's team chooses PyTorch (as oppose to say TensorFlow?)
- ankeshanand 7y agohttps://twitter.com/karpathy/status/868178954032513024 https://twitter.com/karpathy/status/868178954032513024
- tigershark 7y agoNot at expert, but as far as I understood PyTorch is much better to build new models, while with tensorflow it’s easier to assemble the predefined blocks. Source: somewhere in the motivations on why Fast.Ai courses switched to PyTorch for the second edition.
- jeffshek 7y agoSome potential reasons: - TensorFlow is great at deployment, but not the easiest to code. PyTorch isn't frequently used in production until recently. - If you have the resources for great AI engineers and researchers, your team will be good enough to build and deploy both frameworks. - Preference toward the easier framework your tech leads prefer. - Lots of new academic research is coming in PyTorch - TensorFlow is undergoing a massive change from 1.1x to 2.0; if you choose TensorFlow, write on 1.1x just to then refactor to TF 2.0? Or write on TF 2.0 now and deal with all new edge cases? Or write in PyTorch (easier) but handle the more difficult deployment process. - ML code quickly rots. Bad PyTorch code is just bad Python code. Bad TensorFlow code can be a nightmare to debug. - PyTorch's eager execution makes coding NNs much easier to prototype and build.
- m0zg 7y agoBecause PyTorch literally triples researcher productivity. Imagine a deep learning framework which you can actually debug when something goes wrong and which you don't have to fight every step of the way to do even simple things. That's PyTorch.
- new_realist 7y agoMeanwhile Waymo is way ahead.
- grecy 7y agoDo you belittle everyone that gets second place in the Olympics because the winner is "way ahead"? Your comment just reeks of anger and hostility. It seems like you'd rather Tesla didn't try at all, and instead we all just give up and go back to the status quo.
- Fricken 7y agoMeanwhile General Motors is way ahead.
- adrr 7y agoElon belittles lidar saying it is doomed and will never work yet Waymo and Cruise will probably be operating self driving taxi fleets in California next year. Tesla deserves getting dumped on for those comments because they are no where near self driving.
- xiphias2 7y agoAndrej Karpathy just started working on Tesla's software 2 years ago, before what Chris Lattner did was a mess (he wanted to just have 1 task that learns magically everything), Andrej had to start everything from scratch. Waymo had a 20 year advantage, but Google lost many key people there in the meantime as Larry Page didn't want to launch partial self driving. I think both approaches are great and I wouldn't want to choose between the 2, just be a happy user of the end result of the competition.
- kayoone 7y ago> before what Chris Lattner did was a mess (he wanted to just have 1 task that learns magically everything), Andrej had to start everything from scratch any source for that?
- mindfulplay 7y agoJust listening to this talk scares me. The amount of errors - even in a seemingly normal, sunny day - is mind boggling to think people trust this crap. How can we rely on the output of eight cameras? This is not a kid's science project. It's all fancy neural networks until someone dies. Pretty callous and Silicon valley-mindset for such an important and critical function of the car. Will never buy a Tesla after having seen this.
- optimiz3 7y agoYour comments read like /r/RealTesla | TSLAQ fud. Please explain, in detail, what your specific objections are, and how you are more qualified on this subject matter than the presenter.
- mindfulplay 7y agoI am honestly scared even of WayMo that I think unless NHTSA gets its acts together this shit shouldn't be allowed on roads. It's often reactionary - people wait for someone to die and then suddenly you have nervous Musk in front of Congress and such crap. Why wait? Why can't these be regulated before allowing on roads? And no, failover control is not acceptable given the past incidents and deaths.
- Joky 7y agoYou seem to use "regulated" as in "forbidden" right now, but maybe I am misreading so can you elaborate? Today thousands are dying every year on the road, are we just forbidding cars entirely? > And no, failover control is not acceptable given the past incidents and deaths. How many incidents/deaths per miles driven? How does it compare to all the other transportation systems?
- JohnJamesRambo 7y agohttps://www.greencarreports.com/news/1119936_tesla-fatal-crash-rate-with-autopilot-still-no-better-than-with-human-drivers https://www.greencarreports.com/news/1119936_tesla-fatal-cra... “Perhaps that’s because, as it turns out, Teslas on Autopilot could have a higher fatal accident rate than those driven entirely by humans.”
- alexnewman 7y agoJeeze and I can't get my pytorch to stop leaking memory. I couldn't imagine trying to drive a car with it
- Joky 7y agoPytorch is used to train models on servers/cloud, not to drive the car later. The trained model is converted to something native to the embedded environment of the car.
- ngcc_hk 7y agoWow
- eanzenberg 7y agoOne thing I didn't quite understand is how training sub-graphs in parallel works. If you are editing a sub-graph of a monolith type model, aren't you affecting other graphs that have dependencies on the one you're editing? If these are independent graphs, then what's a "sub-graph" even mean?
- punnerud 7y agoIn PyTorch you have full control on the graph and weight, everything feels like Python. So feeding some of the learning between “sub-graph” is easy. Not sure if this is possible on Tensorflow/Keras? He describes the sub-graph training in the context that they they have all the predictors in one big model, and with control of the network can feedforward and train sub-graph (read sub-parts) of the model.
- eanzenberg 7y agoThis is possible in keras, just drive new models that are functions of a monolith model and train independently. I still don’t understand the point though. If you train a “subgraph”, the other tasks dependent on the part of the graph will have to get retrained anyways, since those edits will affect the other tasks.
- paraschopra 7y agoI think their architecture might be their secret sauce. But I'm curious about this too.
- antpls 7y agoFirst time I read about "sub-network" is in this AI Google blog post : https://ai.googleblog.com/2019/09/recursive-sketches-for-modular-deep.html?m=1 https://ai.googleblog.com/2019/09/recursive-sketches-for-mod... They talk about the concept of "modular network". The article itself links to the Wikipedia page : https://en.wikipedia.org/wiki/Modular_neural_network https://en.wikipedia.org/wiki/Modular_neural_network Not sure it's exactly the same idea, but it looks similar.
- modeless 7y agoAwesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case for doing their own edge hardware for the cars (barely), but I really don't think doing training hardware will pay off for them. If they're serious about it, they should spin it out as a separate business to spread the development cost over a larger customer base. Also, I'm really curious whether the custom hardware in the cars is benefiting them at all yet. Every feature they've released so far works fine on the previous generation hardware with 1/10 the compute power. At some point won't they need to start training radically larger networks to take advantage of all that untapped compute power?
- sdan 7y agoDid they say they were building their own training hardware? I thought it was just their inference hardware (the boards on the teslas)?
- m0zg 7y agoNothing crazy about it. TPU-like stuff is ~10x the energy efficiency of GPUs and several times the speed. When you're spending megawatt-hours and days to train a single model, it adds up in both real and opportunity costs. Also, Google TPU TOS prohibits the use of TPUs for stuff that competes with Google (and I'm assuming with other companies under Alphabet umbrella), at Google's sole determination. Not that it would be a good idea to upload Tesla's proprietary data into Google Cloud even if it did not. Cloud, after all, is just somebody else's computer.
- jacquesm 7y agoThe competition in this space is great but I can't help but wonder what would happen if instead all these companies pooled their resources and went after the goal collectively. There is so much duplication going on and the paths do not seem to me - as an outsider - to be all that divergent, which is usually a pre-condition for having a lot of independent efforts one of which will succeed. It's as if everybody wants to be the one to exclusively own the tech. Imagine every car manufacturer having a completely different take on what a car should be like from a safety perspective. We have standards bodies for a reason and given the fact that there are plenty of lives at stake here maybe for once the monetary angle should get a back-seat (pun intended) to safety and a joint effort is called for. That would also stop people dying because operators of unsafe software are trying to make up for their late entry by 'moving fast and breaking things' where in this case the things are pedestrians, cyclists and other traffic participants who have no share in the monetary gain.
- paraschopra 7y agoStandards shouldn't emerge too soon. I think for self driving tech, at the current stage, competition is good because there are lots of unsolved questions. Competition will ensure the best tech is ultimately available to consumers. Of course, it's not a binary choice. Things like data should probably be pooled but the use of data in tech should compete.
- jacquesm 7y agoDitto validation tech frameworks. If those are not standardized then people will not be able to make an informed choice about which solution is the safest other than to wait a decade and do a bodycount.
- konschubert 7y agoI think that there is still a need for some brilliant insights and breakthroughs, it isn't just a matter of getting the work done. So actually, I think it's one of these situations where having a lot of independent efforts might be worth it.
- 7y ago
- protomikron 7y agoFun fact for all of you: Some time ago (around ~10 years) this guy (the presenter) was internet famous for being a Rubik cube speed solver and making tutorials and videos about that: https://www.youtube.com/watch?v=609nhVzg-5Q https://www.youtube.com/watch?v=609nhVzg-5Q
- nsilvestri 7y agoI'll always know him as badmephisto. In a recentish reddit AMA, he says he still keeps a cube on his desk so he can practice a bit and not forget his algorithms.
- lanius 7y agoCan you provide the link to that AMA?
- deleted 7y ago[deleted]
- eyeundersand 7y agoWow! Thank you. What a blast from the past! I learned how to solve the Rubik's Cube blindfolded by watching him, back in the day. His tutorials are perfect and I've probably recommended his channel to ~50 people myself. Crazy he only has 36.6k subs. Glad to see he's doing well.
- laichzeit0 7y agoThe good news for me is that the upper bound for fully autonomous self-driving cars is no more than 50 years away. What a time to be alive. If it happens before then, that will be an absolute bonus.
- fyp 7y agoFor those who want to learn more, I would start with Mask-RCNN where you have a very similar architecture: one shared backbone with multiple heads that can be retrained for various tasks (bounding boxes, masks, keypoints, etc): https://youtu.be/g7z4mkfRjI4?t=628 https://youtu.be/g7z4mkfRjI4?t=628
- Gravityloss 7y agoInteresting that they don't have a full 3D world model. I'm certainly not a machine learning expert. I'm still amazed the route from image recognition to a 2D map of "what's drivable" to autonomous driving is so direct. One would expect to hit a ceiling really soon with that approach. To me it seems we're still in really early days.
- spyder 7y agoThey're doing 3D for the road path, and even predicting it beyond corners: https://youtu.be/Ucp0TTmvqOE?t=8137 https://youtu.be/Ucp0TTmvqOE?t=8137 And later in the video they show 3D reconstruction from cameras and saying they use it in the car. Watching the full talk is recommended if you have the time (talk starts around 1:10:00 in the video)
- Gravityloss 7y agoThanks, seems my original comment is wrong then!
- williesleg 7y agoTesla is the best!
- SloopJon 7y agoThe description of SmartSummon about halfway through the talk is interesting. One of the views looks like SLAM using a particle filter, but Andrej seems to say that it's done entirely within a neural net.
- londons_explore 7y agoI'm still amazed that Teslas team isn't using a map... I know maps get outdated and are sometimes wrong, but having inaccurate knowledge of what's around the corner is far far more helpful than not having any clue whats around the corner. The smart solution would be to consider a map a probabilistic thing, which neural networks are really good at handling.
- anonu 7y agoI'm still amazed Tesla has decided not to use lidar and instead just stick with cheap cameras. Better sensors are there, they're available, they're cheap and they can probably "see" better than plain old cameras... it doesn't make too much sense not to use them IMHO. But then again, I am not coding NNs for Tesla...
- JoeSmithson 7y agoI think they advertised self-driving as a future feature of the Model 3, so I think they are limited to whatever Model 3's currently have.
- option 7y agolidars are used to generate humongous amounts of labeled training data for depth perception networks so that you don’t have to use them during inference
- londons_explore 7y agoWhile LIDAR's are certainly 'better' from a technological standpoint than not having anything, from a business standpoint it's less clear. LIDAR's are cheap, but not cheap enough yet to not seriously affect the bottom line if you put them into every car. It also will kill the resale price of cars without it, which in turn hurts the companies image and stock price.
- nielsole 7y agoIf you are first to market with a level 4 autonomous taxi, unit economics will likely be great regardless of whether you put LIDAR or cameras in.
- diveanon 7y agoAndrej Karpathy is such a treasure. He is an excellent presenter who really has a passion for teaching. Im not really involved with the industry, so I cant really speak to how he holds up to other experts. However he is by far the most digestable resource I have found for learning about NN and science behind them. If you are just discovering him now, google his name and just start reading. His work is truly binge worthy in the most meaningful way.