8 ms·
Neural Networks: Zero to Hero
- deleted 4y ago[deleted]
- agentofoblivion 4y agoWonderful! Just went through the GPT video the other day and it was great. Andrej has a talent for pedagogy via simplification.
- yuuuuyu 4y agoAnd he presents all this stuff with humility. Many people that present are just showing off and are pretty much full of themselves. I suppose they need the ego boost, who knows. But Andrej could be the nice guy next door in the dorm who is studying the same course as you, just that hr is a lecture or two ahead. (Until you figure out he is the former VP of AI at Tesla or whatever his title ended up being before he left.) I can even recommend his interview with Lex Fridman.
- meling 4y agoAbsolutely agree with this.
- 0cf8612b2e1e 4y agoOnly finished the first video, but he even made two minor blunders in his code, but kept the footage. Really helps your confidence when you see a pro make a mistake rather than a perfectly polished but unattainable ideal standard.
- Yajirobe 4y agoHe also is quite good at teaching and solving the Rubik's cube
- abraxas 4y agoHe is a master educator. While at Stanford he developed their undergrad machine learning intro course named cs231n which immediately became legendary. It's somewhat out of date on some details but it's still well worth watching especially as delivered by Andrej. You can find all 11 lectures on YouTube.
- polycaster 4y agohttps://youtube.com/playlist?list=PLkt2uSq6rBVctENoVBg1TpCC7OQi31AlC https://youtube.com/playlist?list=PLkt2uSq6rBVctENoVBg1TpCC7...
- frankcort 4y agoWhich GPT Video?
- spaceman_2020 4y agoA little offtopic, but is this something that someone with only webdev experience can get started with?
- nomel 4y agoGetting started with it takes exactly zero experience. Being productive in it does, but that's unrelated to the starting point, and shouldn't discourage you, if you really want to do it. There are several open courses, online.
- jfisher4024 4y agoIt might make sense to get a handle on Python first
- thealchemistdev 4y agoYes. The journey will teach you everything you need to know or it'll kill you. The likelihood of that happening here or any path in IT is negligible at best. So just show up everyday and collect more fuck ups than the next guy (or your former self).
- mhh__ 4y agoThe tools are basically irrelevant conceptually. Its all about the mathematics
- whiplash451 4y agoYou need some fluency in python and basic knowledge of algebra (matrix multiplication etc.) If you want, you can also start with the first lessons of course.fast.ai
- yacine_ 4y agoWhat I appreciate about karpathy's videos is that it doesn't make things any more complicated than they need to be. Simple, engineering language is used. No gatekeeping! It's reassuring, and lets everyone know that anyone can do it. Thanks karpathy!
- bilsbie 4y agoI just don’t know what he means by logits. Everything else seems like straightforward language.
- airstrike 4y agoHaving not watched the series, I can only assume he means logit as in a probability function from 0 to 1 https://deepai.org/machine-learning-glossary-and-terms/logit#:~:text=What%20is%20a%20Logit%3F,rather%20than%20the%20X%2Daxis https://deepai.org/machine-learning-glossary-and-terms/logit....
- PheonixPharts 4y agologit is not a "probability function", quite the opposite. You can see this in the image in the link you posted (the x-axis is from 0-1, the y-axis is from -inf to inf). It transforms probabilities into log odds which is a linear space, and make combining probabilities much nicer. The inverse logit or logistic function takes log odds and transforms them back into probabilities. Most machine learning relies heavily on manipulating probabilities, but since probabilities are not linear, the logit/logistic transformations become essential to correctly modeling complex problems involving probabilities.
- airstrike 4y agoprobability-related* function
- joshvm 4y agoWhen people mention logits, they're usually referring to the raw output of the model before it gets transformed/normalised into a probability distribution (i.e. sums to 1, range [0,1]). Logits can take any value. The naming might not be mathematically strict, because it assumes(?) that you're going to apply softmax (which interprets the output of the model as logits), but that's how the term is used. For example in many classification problems you get a 1D vector of logits from the final layer, you apply softmax to normalise, then argmax to extract the predicted class. It extends to other tasks like semantic segmentation (predict pixel classes) where the "logit" output is the same size as the image with a channel for each class and you apply the same process to get a single channel image with class-per-pixel. Here's a nice explanation: https://stackoverflow.com/a/66804099/395457 https://stackoverflow.com/a/66804099/395457
- 7373737373 4y agoThis was the first time I actually grokked backpropagation, just the first video alone is more lucid and valuable than any other resource about machine learning I had seen before, in fact it's so well explained that i managed to implement the library almost completely from memory after watching it - I cannot recommend it highly enough, especially for programmers without a math background! The only aspect I could see being non-ideal for some is that it uses some Python-specific cleverness/advanced syntax and semantics (__call__(), list comprehensions with two for's, **kwargs, __add__, __repr__, subclasses, (nested) functions as variables etc.), but if you are familiar with these it might seem more compact and elegant as well.
- whiplash451 4y agoTo be fair, the older Andrew Ng’s online course was also fantastic to explain backprop. But this does not remove any credit to Andrej’s class.
- mhh__ 4y agoI find it genuinely stunning how ugly python gets in these projects. Almost immediately, even in a toy project (not tinygrad which is deliberately super dense).
- fuddle 4y agoI agree. I'm looking forward to re-watching it to as it was so information dense.
- sirodoht 4y agoI'm doing an ML apprenticeship [1] these weeks and Karpathy's videos are part of it. We've been deep down into them. I found them excellent. All concepts he illustrates are crystal clear in his mind (even though they are complicated concepts themselves) and that shows in his explanations. Also, the way he builds up everything is magnificent. Starting from basic python classes, to derivatives and gradient descent, to micrograd [2] and then from a bigram counting model [3] to makemore [4] and nanoGPT [5] [1]: https://www.foundersandcoders.com/ml https://www.foundersandcoders.com/ml [2]: https://github.com/karpathy/micrograd https://github.com/karpathy/micrograd [3]: https://github.com/karpathy/randomfun/blob/master/lectures/makemore/makemore_part1_bigrams.ipynb https://github.com/karpathy/randomfun/blob/master/lectures/m... [4]: https://github.com/karpathy/makemore https://github.com/karpathy/makemore [5]: https://github.com/karpathy/nanoGPT https://github.com/karpathy/nanoGPT
- bilsbie 4y agoDo you run the code as you watch? I’ve been simply watching them on a palm from a hammock and I’m worried I’m not getting the full experience.
- sirodoht 4y agoI've found that actually running the code has been very beneficial in understanding. This, along with reasoning for each line of code and spending a lot of time with the video paused and discussing and explaining to each other what we understood.
- whiplash451 4y agoSame. I also found the exercises to be useful.
- iKlsR 4y agoSomething I discovered not so long ago that I wish I had years ago is to watch the video first and then code along after. So simple but makes a world of difference, you can skip errors, fluff and foresee what's next, all around you'd think it's watching two 2 hour videos but it works out in terms of getting the most out of the content and drilling it into your head.
- sourcecodeplz 4y agoThis is really cool and I am so glad my math teacher was a hard ball and I still remember some Calculus. edit: Python really was/is made for this numbers/calculation/visualization thing. Kinda kicking myself now for not investing more in it and sticking with PHP, although PHP has its merits when building different things, Python is a beast with numbers.
- sourcecodeplz 4y agoI am at graphwiz now and it is getting better and better. Also using ChatGTP to ask questions where I don't get something. Wow what a time we live in to learn things.
- auggierose 4y agoThis course together with the new fastai ones [1] seem to be exactly what I was looking for. The micrograd video is excellent. [1] https://course.fast.ai https://course.fast.ai
- throwbackhere 4y agoI tried this course a few weeks ago but quickly got stuck after finetuning the first model that detects cats (first example). The finetuning part works but I was never able to get it to infer (python kept crashing complaining that an object didn't have a read function). Hopefully, I'll manage to get further with this course.
- zakki 4y agoAnybody tried the lessons in an Apple MBA/MBP M1/M2? Is it easily applicable?
- whiplash451 4y agoAndrej’s course is brilliant and so nice to follow. His explanation of attention is the most accessible I have ever seen.
- Buttons840 4y agoMy hive mind connection must be good because I literally finished this course yesterday. It was very satisfying to learn how transformers worked, to finally be able to turn the obscure glyphs of the research papers into real code, but I think transformers are too big for what I can do on my own computer. The author mentioned that the toy transformer he was building in the final video took 15 minutes to train on his A100 GPU (a $10,000 GPU), and the results weren't even that good; the transformer was spelling words correctly using character level tokens, I guess that's something, but it's not GTP4. Even so, there were a lot of good tips to pick up along the way. This is a great series that I'm thankful to have. The "Backprop Ninja" video was hard work, you manually calculate the gradients and then compare your calculations against PyTorch. It's great to have instant feedback telling you whether your gradients are correct or not.
- callistus 4y ago> his A100 GPU (a $10,000 GPU) These are available to rent per hour at much lower costs. The author mentions this in the video description.
- Buttons840 4y agoTrue, as much as I enjoy owning and controlling my own hardware, buying an A100 and then letting it sit idle while I procrastinate and play video games probably isn't the best use of resources. He did say "my GPU" (or similar) at one point, and I thought maybe he does enough ML stuff that he bought his own.
- nl 4y agoIf you have an NVIDIA gaming GPU you can train reasonable transformers.
- oh_sigh 4y agoApproximately 40 cents USD for 15 minutes from cursory research.
- abhishekjha 4y agoI am always lost in these blogs. Is there any gradual progression of understanding/exercises that one can follow to apply ML/NN/DL practically? Having these disconnected pieces of information with no clear link to one another feels like a lot of noise to me.
- Buttons840 4y agoI've been reading The Little Learner, which builds machine learning knowledge on top of Scheme / Racket. After that book, you could watch this series, which will immediately begin explaining how the automatic differentiation works.
- porknubbins 4y agoI’m almost done with both of the Andrew Ng coursera specializations they are the not at all disconnected or unclear. I don’t think I’ve ever learned so much so quickly in fact.
- Buttons840 4y agoThose interested in this might also be interested in some notes from the University of Amsterdam: https://uvadlc-notebooks.readthedocs.io/en/latest/index.html https://uvadlc-notebooks.readthedocs.io/en/latest/index.html Tutorial 6 covers transformers.
- adamnemecek 4y ago[flagged]
- qumpis 4y agoHow is it relevant here?
- adamnemecek 4y agoIt provides a unifying framework for how NNs work.
- zwaps 4y agoYou should check out this https://cats.for.ai/program/ https://cats.for.ai/program/
- mickael-kerjean 4y agoDiscovering this kind of ressources is a big reason I come on HN. Another recent and good one on the same topic is MIT 6.S191: https://www.youtube.com/watch?v=ySEx_Bqxvvo https://www.youtube.com/watch?v=ySEx_Bqxvvo
- ethicalsmacker 4y agoThe simplest guide to NNs I have ever read is this one: http://www.ai-junkie.com/ann/evolved/nnt1.html http://www.ai-junkie.com/ann/evolved/nnt1.html It's an old site and guide, but probably still the easiest to understand if you're coming from a programming background.
- sanketskasar 4y agoCan other members suggest courses/resources, that help prepare with basic Maths/Stats/ML/Programming skills required for this one?
- Alifatisk 4y agoI wonder, how do people visualize their neural network?
- fulafel 4y agoAre there GPU hardware requirements for the exercises?