8 ms·
Understanding Deep Learning
- WeMoveOn 3y agolit
- msie 3y agoThis book looks impressive. There's a chapter on the unreasonable effectiveness of Deep Learning which I love. Any other books I should be on the lookout for?
- nootopian 3y agohttps://news.ycombinator.com/item?id=38425368 https://news.ycombinator.com/item?id=38425368
- theogravity 3y agoI wish it wasn't an X post. Can't see responses at all without an account.
- nextos 3y agoUse nitter to go around X authwalls: https://nitter.net/suhail/status/1728676402864812466 https://nitter.net/suhail/status/1728676402864812466
- nextos 3y agoThis presentation from Deep Mind outlines some foundational ML books: https://drive.google.com/file/d/1lPePNMGMEKoaDvxiftc8hcy-rFpobXeJ/view https://drive.google.com/file/d/1lPePNMGMEKoaDvxiftc8hcy-rFp... For the impatient, look into slide #123. Essentially, the recommendations are Murphy, Gelman, Barber, and Deisenroth. Note these slides have a Bayesian bias. In spite of that, Murphy is a great DL book. Besides, going through GLMs is a great way to get into DL.
- mitthrowaway2 3y agoWhat is a "Bayesian bias"?
- nextos 3y agoI meant the presenter is discussing ML from a Bayesian point of view, which is interesting, but not something you need if your aim is just to understand deep learning.
- fastneutron 3y agoReality has a well known Bayesian bias… Joking aside, these slides are excellent! Is there an associated video or course that they were a part of?
- nextos 3y agoNo, these were part of a conference held at Tubingen in 2020.
- teleforce 3y agoYes, it looks very impressive indeed and it has the potential to be the seminal textbook on the subject. Fun facts, the infamous Attention paper is closing in to reach the 10K citations, and it should reach this milestone by the end of this year. It's probably the fastest paper ever to reach this significant milestone. Any deep learning book written before the Attention paper should be considered out of date, and needs updating. The situation is not unlike an outdated Physics textbook with Newton's laws but devoid of the infamous Einstein's equation of energy equivalence.
- ldjkfkdsjnv 3y agoI spent a decade working on various machine learning platforms at well known tech companies. Everything I ever worked on became obsolete pretty fast. From the ML algorithm to the compute platform, all of it was very transitory. That coupled with the fact that a few elite companies are responsible for all ML innovation, its oxymoronic to me to even learn a lot of this material.
- drBonkers 3y agoWhat would you recommend someone read instead?
- ldjkfkdsjnv 3y agoBetter to understand the bounds of whats currently possible. And then recognize when that changes. Much more economically valuable
- probablynish 3y agoDo you think there's a better way to do this than spending some time playing around with the latest releases of different tools?
- trace_id 3y agoEven better: change the bounds of whats possible ;)
- mindcrime 3y ago> Even better: change the bounds of whats possible ;) ... which will be easier it you have a solid grasp of the foundations of the field. If you only ever focus on the "latest shiny" you'll be lost and left floundering when the landscape changes out from underneath you.
- reqo 3y agoVery few things stay the same in Technology. You should think of technology as another type of evolution! It is driven by the same type of forces as evolution IMO. I think even Linus Torvalds once stated that Linux evolved trough natural selection.
- water-your-self 3y agoNo chapter on RNNs, but one on transformers is interesting, having last read Deep learning by ian goodfellow in 2016
- nothrowaways 3y agoYeah, content looks interesting.
- PeterisP 3y agoRNNs have "lost the hardware lottery" by being structurally not that efficient to train on the cost-effective hardware that's available. So they're not really used for much right now - though IMHO they are conceptually sufficiently interesting enough to cover in such a course.
- nextos 3y agoThat is not completely true. There are RNNs with transformer/LLM-like performance. See e.g. https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM. They are less popular, and less explored. But an interesting route ahead.
- mindcrime 3y ago> RNNs have "lost the hardware lottery" by being structurally not that efficient to train on the cost-effective hardware that's available. Which suggests two obvious paths forward: 1. Don't bother learning / using RNN's 2. Co-develop new hardware / new RNN architectures that work together to provide great performance per unit of price. Now of course nobody is saying (well, I am not saying) that (2) would be easy... or even necessarily possible. But somebody should at least be intrigued by the idea. And in the world we live in today where FPGA's and other devices make it easier than ever to experiment with custom hardware architectures... it might be worth taking a stab at it.
- nsxwolf 3y agoAs someone who missed the boat on this, is learning about this just for historical purposes now, or is there still relevance to future employment? I just imagine the OpenAI eats everyone's lunch in regards to anything AI related, am I way off base?
- ksherlock 3y agoMaybe last week's drama should have been a left-pad moment. For many things you can train your own NN and be just as good without being dependent on internet access, third parties, etc. Knowing how things work should give you insight into using them better.
- lamroger 3y agoI wonder if using APIs was more of a first to market move
- mnky9800n 3y agoWhich drama of last week are you referring to? The one about the openai guy saying it's all just the data set? Or something else?
- quchen 3y agoTheir CEO was fired, hired by Microsoft, took a bunch of people with him, and is now back at the company
- giardini 3y agoDid the "bunch of people" also return (from Microsoft to OpenAI)?
- whatever_yeah 3y agoI must have missed the "dataset" news you're referring to, could you elucidate?
- adamnemecek 3y agoAll machine learning is Hopf convolution, analogous to renormalization. This should come as no surprise, renormalization can be modeled via the Ising model which itself is closely related to Hopfield networks which are recurrent networks.
- dbmikus 3y agoDon't know any of these terms, but you gave me some interesting topics to google about. Thanks!
- calf 3y agoThat's an interesting point, are there any resources to learn about this? I have a CS background, in that we generally only cover 1st-year physics and very little theoretical math beyond linear algebra, etc.
- adamnemecek 3y agoThere is some yeah, there is a recent book on deep learning via renormalization https://arxiv.org/pdf/2106.10165.pdf https://arxiv.org/pdf/2106.10165.pdf. I have a discord https://discord.cofunctional.ai https://discord.cofunctional.ai
- dchuk 3y agoHopefully not a dumb question: how do I buy a physical copy?
- rossant 3y agoIt'll be published in a few days: https://mitpress.mit.edu/9780262048644/understanding-deep-learning/ https://mitpress.mit.edu/9780262048644/understanding-deep-le...
- oakejp12 3y agoThe PDF figures for 'Why does deep learning work' seem to point to 'Deep learning and ethics' and vice versa.
- TrackerFF 3y agoReading through it, and it def looks accessible.
- ksvarma 3y agoSimply great work and making it freely available is outstanding!!
- contrarian1234 3y agoIt's very hard to judge a book like this... (based on a table of contents?) Who is the author ? Have they published anything else highly rated ? Are there good reviews from people that know what they're talking about? Are there good reviews from students that don't know anything ?
- siddbudd 3y agoSome Google-fu for you: based on a table of contents? You can download the draft of Chapters 1-21 (500+ pages) from the linked site. Who is the author ? Simon J. D. Prince is Honorary Professor of Computer Science at the University of Bath and author of Computer Vision: Models, Learning and Inference. A research scientist specializing in artificial intelligence and deep learning, he has led teams of research scientists in academia and industry at Anthropics Technologies Ltd, Borealis AI, and elsewhere. Have they published anything else highly rated ? Author of >50 peer reviewed publications in top tier conferences (CVPR, ICCV, SIGGRAPH etc.) https://scholar.google.com/citations?user=fjm67xYAAAAJ&hl=en https://scholar.google.com/citations?user=fjm67xYAAAAJ&hl=en Are there good reviews [...] The book has not been published, this is literally a free draft that you are looking at. The book is listed on Amazon as a pre-order for 85USD.
- arman_hkh 3y agoMarcus Hutter on his [Marcus' AI Recommendation Page]: "Prince (2023) is the (only excellent) textbook on deep learning."
- xcv123 3y ago> based on a table of contents? The entire pdf is available as a free download on that page. First link at the top. https://github.com/udlbook/udlbook/releases/download/v1.16/UnderstandingDeepLearning_24_11_23_C.pdf https://github.com/udlbook/udlbook/releases/download/v1.16/U...
- komatsu 3y agoI can highly recommend the author. His last book "Computer Vision: Models, Learning, and Inference" is very readable, approaches the matter from unorthodox viewpoints + includes lot of excellent figures supporting the text. I'm buying this on paper!
- Slix 3y agoIf I start now and start reading up on AI, will I become anything close to an expert? I'm worried that I'm starting a journey that requires a Master's or PhD.
- ocharles 3y agoVery hard to answer without knowing what your goal is. Do you want to be a practitioner of DL, or do you want to be a researcher?
- mavelikara 3y agoNot the OP, but I’d like to hear you answer and reasoning for the “practitioner of DL” case.
- Chirono 3y agoFrom reading this book you’d have a very good grasp of the underlying theory, much more than many ML engineers. But you’d be missing out on the practical lessons, all the little tips and intuitions you need to be able to get systems working in practice. I think this just takes time and it’s as much an art as it is a science.
- crimsoneer 3y agoYou probably won't become an expert, but I'm not clear why you'd want to!
- strikelaserclaw 3y agoThe only guide post to use in this world of ever increasing information to learn is to ask yourself "do i find learning this stuff enjoyable?", questions like "can i become an expert" are vague and not good guideposts.
- martingoodson 3y agoMost comments here are in one of two camps: 1) you don't need to know any of this stuff, you can make AI systems without this knowledge, or 2) you need this foundational knowledge to really understand what's going on. Both perspectives are correct. The field is bifurcating into two different skill sets: ML engineer and ML scientist (or researcher). It's great to have both types on a team. The scientists will be too slow; the engineers will bound ahead trying out various APIs and open-source models. But when they hit a roadblock or need to adapt an algorithm many engineers will stumble. They need an R&D mindset that is quite alien to many of them. This is when an AI scientists become essential.
- braza 3y ago> But when they hit a roadblock or need to adapt an algorithm many engineers will stumble. My experience is the other way around. People underestimate how powerful building systems is and how most of the problems worth solving are boring and require out-of-the-box techniques. During the last decade, I was in some teams and I noticed the same pattern: The company has some extra budget and "believes" that their problem is exceptional. Then goes and hires some PhDs Data scientists with some publications but only know R and are fresh from some Python bootcamps. After 3 months, or this new team no much was done, tons of Jupyter notebooks around but no code in production, and some of them did not even have an environment to do experimentation. The business problem is still not solved. The company realizes that having a lot of Data Scientists not not so many Data/ML Enginers means that they are (a) blocked to do pushing something to production or (b) are creating a death star of data pipelines + algorithms + infra (spending 70% more of resources due to lack of knowledge on Python). The project gets delayed. Some people become impatient. Now you have a solid USD 2.5 million/year team that is not capable of delivering a proof of concept due to the fact that people cannot do the serving via Batch or via REST API. The company lost momentum, competitors moved fast. They released an imperfect solution, but a solution ahead, and they have users on it and they are enhancing. Frustration kicks in, and PMs and Eng Managers fight about accountability. VP of Product and Engineering wants heads in a silver plate. Some PhDs get fired and go to be teachers in some local university. Fin.
- gardenhedge 3y ago
- Webster09 3y ago[flagged]