54 ms·
Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory
- sage76 3y agoDo people really finish these books cover to cover? I have been working on Bishop's PRML, and it is extremely time consuming to really finish the book, and do all the exercises. I saw a blog from a guy who did the same, and it took him 1500+ hours. Not a single person in my masters program finished any of these books. They just did the courses and googled whatever else was necessary.
- dachworker 3y agoIs anyone using any of this math? My guess is no. At best it provides "moral support" for deep learning researchers who want to feel reassured that what they are attempting to do is not impossible. Glad to be proven wrong, though.
- nerdponx 3y agoDescribing it as "moral support" really sells it short. Imagine computer science without sorting algorithms, search algorithms, etc that have been proven correct and have known proven properties. This math serves the same purpose as CS theory. So yes, if you're just fitting a model from a library like Keras, you're not really "using" the math. If you're working with data sets below a certain size, problems below a certain level of complexity, and models that have been deployed for many years and have well studied properties, you can do a lot with only a cursory understanding of the math, much like you can write perfectly functional web apps in Python or Java without really understanding how the language runtime works at a deep level. But if you don't actually know how it works, you're going to get stuck pretty badly if you encounter a situation that isn't already baked into a library. If you want to see what happens when you don't know the underlying math, look at the current generation of "data science" graduates, who don't know their math or statistics fundamentals. There are plenty of issues on the hiring side of course, but ultimately the reason those kids aren't getting jobs is that they don't actually know what they're doing, because they were never forced to learn this stuff.
- nephanth 3y agoAccording to the abstract it covers different ANN architectures, optimization algorithms, probably backpropagation.. so um yes? That is stuff anyoke in machine learning uses everyday?
- danielmarkbruce 3y agoSome people like to think and communicate in dense math notation. So, yes.
- godelski 3y agoThere's something I tell my students. You don't need math to make good models, but you do need to know math to know why your models are wrong. So yes, math is needed. If you don't have math you're going to hoodwink yourself into thinking you can get to AGI by scale alone. You'll just use transformers everywhere because that's what everyone else does and you'll get confused between activation functions. You'll make models and models that work, but there's a big difference in working models and knowing where to expect your models to fail and understanding their limitations. I feel a lot of people just look at test set results and expect that to mean that the model isn't overfitting. (not to mention tuning hps based on test set results)
- famouswaffles 3y ago>If you don't have math you're going to hoodwink yourself into thinking you can get to AGI by scale alone. There are many researchers who "have math" and still believe this. Appeal to Authority is a fallacy at the best of times but it's usually a convincing one. Not so much when the authority hasn't formed consensus on the appeal.
- light_hue_1 3y agoOh sure. I say the same to my students. But the particular spin on this book makes it look to non-experts that this is the math you need to do something useful with deep learning. And that's just not true. Certainly you need to understand what you're optimizing, how your optimizer works, what your objective function is doing, etc. But the vast majority of people don't need to know about theoretical approximation results for problems that they will never actually encounter in real life, etc. For example, I have never used used anything like "6.1.3 Lyapunov-type stability for GD optimization" in a decade of ML research. I'm sure people do! But not on the kinds of problems I work on. Just look at the comments here. People are complaining about the lack of context, but this is fine for the audience the book is aimed at. It's just the average HN reader. I think it would be better if the authors chose a different title. As it stands, non-experts will be attracted and then be put off, and experts will think the book is likely to be too generic.
- 3y ago
- fastneutron 3y agoIn the latter part of the book that covers PINNs and other PDE methods, it helps to frame these using the same kind of functional analysis that is used to develop more traditional numerical methods. In this case, it provides a way for practitioners to verify the physical consistency between the various methods.
- aquafox 3y ago[flagged]
- deleted 3y ago[deleted]
- nerdponx 3y agoIt looks to me like most of the space is taken up with a plot of the sine function and the python code to generate the plot. Maybe it's a little fluffy, but it might be good for somebody self-taught, or a young person learning all of this stuff for the first time and wanting a quick reference.
- godelski 3y agoThere's a lot to critique but this is a really weird one (page 49 if anyone is following). The whole thing is 5 sentences and all the space is because a diagram and code block. The 5 sentences should be the thing to complain about
- joe_the_user 3y agoI only skimmed but I get the impression that sort of thing is common in the text. I think it's got the problem that deep learning "isn't really math" - in the sense that deep learning using indeed very elaborate computational structures that can be specified mathematically but it doesn't prove theorems about them - not theorems that characterize what's happening. The theorems are just hints about what might be happening. The key deep learning knowledge is in papers that basically only show that X approach works best on Y (plus maybe some suggestive theorem) - for example Attention Is All You Need.
- godelski 3y ago> The theorems are just hints about what might be happening. Isn't this true everywhere? Certainly it is just, in the words of Asimov, the relativity of wrongness. I mean even physics is "just a hint" despite being an incredibly strong one. I think maybe a lot of people might not agree but I think a lot of people aren't as aware of all the research that still goes on in every day physics. Like studying ocean waves/currents, wind, explosions, materials, and so much more that is not quantum or relativity. But quantum and relativity get far more attention, so perception bias. > The key deep learning knowledge is in papers that basically only show that X approach works best on Y (plus maybe some suggestive theorem) I very much disagree. Those are certainly the most visible, but not the most foundational. Actually I believe this approach is holding us back, and diffusion is my best example of this. Big steps in diffusion and GANs were made around the same time, but GANs were easier to implement and less resource heavy. Sohl-Dickstein certainly is a key player despite Ho being more well known. Same with Aapo Hyvarinen. I think we got too captivated by GANs that it made it harder to publish anything else. I've had some experience with this personally, where I've given up trying to publish in Normalizing Flows because reviewers will ask why my works are not better than GANs (or now diffusion) despite being better than other Flows or even got this on a distillation based paper (multiple times before we abandoned it). If there's too heavy of concern on metrics (not using as guides/hints, but as targets) then how can other things advance in a normal way? You'd have to take leaps and bounds instead of incrementalism (which we've established is fine for popular paths. Former GANs, now diffusion). Leaps and bounds because the community size is exceptionally disproportionate and that is far more time and research being put into one than another. I'd argue that we have pretty good evidence to believe a hypothesis that counterfactually diffusion would have emerged as a strong player sooner if this weren't how we measured publication criteria (SOTA chasing). I believe this problem has only become worse. But this is how technology always advances, it isn't one technology getting better and better, but we see the composition of different technologies. Almost always where the replacement starts out as significantly worse than the existing status quo. So I'd argue we're leaving a lot of good work on the table by doing this. Certainly we have enough people working in ML that we can adequately do both, which is certainly much more optimal. You need both, but problem is we just compare <new thing, or not as established thing> to <current popular and SOTA thing> as if benchmarks are the only component of the story here.
- reqo 3y agoIs it common to publish books directly to ArXiv, especially books that have just been released?
- godelski 3y agoFirst time I've seen one of these books where I wished there was more words and less math. Usually it is quite the opposite. But this book seems written as if they wanted to avoid natural language at all costs.
- deleted 3y ago[deleted]
- sarahwoods 3y ago[dead]
- axpy906 3y agoThis is in Tensorflow. Would rather see a numpy version or something along those lines so that students can better understand what each step looks like in code. I concur on the comments noting lack of explanation for the notation/lemmas/proof.
- deleted 3y ago[deleted]
- godelski 3y agoI second this. Numpy would be the way to go, so students can switch to JAX or PyTorch trivially. Or they could use a mix, starting with numpy, build the layer from scratch, then hand over the abstraction. Pyro would be really good for this too
- _giorgio_ 3y agoTensorflow? LOL what is this, the year 2010?
- CamperBob2 3y agoMost of the examples I saw used Pytorch. (Which is still a step or two removed from the actual machinery, of course.)
- HybridCurve 3y agoAs someone who has a deeper knowledge of programming rather than math, I find the mathematical notation here to be harder to understand than the code (even in a programming language I do not know). Does anyone with a stronger mathematical background here find it easier to understand the math as written more easily than the source code?
- joshuanapoli 3y agoMathematical notation usually has a problem with preferring single-letter names. We usually prefer to avoid highly abbreviated identifier names in software, because they make the program harder to read. But they’re common in Math, and I think that it makes for a lot of work jumping back and forth to remind oneself what each symbol means when trying to make sense of a statement.
- tsimionescu 3y agoI think the main difference is that in programming you typically use names from your domain, like "request" or "student". But math objects are all very abstract, they don't denote any domain. For example, if I have a triangle and I want to name its vertexes so I can refer to them later, what would be a good name? Should I call them vertexA, vertexB, and vertexC just so it's not a single letter?
- aabajian 3y agoAll three authors are PhDs or PhD-candidates in mathematics. The notation is extremely dense. I'm curious who their target audience of "students and scientists" are for this book.
- angra_mainyu 3y agoI had a bunch of classes in undergrad (physics) that had basically the same notation and style.
- tnecniv 3y agoLikely graduate students with a very theoretical interest. Some theoretically-oriented scientists and engineers are also basically applied mathematicians. It is presumably targeted at people that want to further develop the theoretical aspects of learning, as opposed to applied practitioners
- HighFreqAsuka 3y agoI've seen quite a few of these books attempting to explain deep learning from a mathematical perspective and it always surprises me. Deep learning is clearly an empirical science for the time being, and very little theoretical work that has been so impactful that I would think to include it in a book. Of the such books I've seen, this one seems like actively the worst one. A significant amount of space is dedicated to proving lemmas that provide no additional understanding and are only loosely related to deep learning. And a significant chunk of the code I see is just the plotting code, which I don't even understand why you'd include. I'm confident that very few people will ever read significant chunks of this. I think the best textbooks are still Deep Learning by Goodfellow etal and the more modern Understanding Deep Learning (https://udlbook.github.io/udlbook/ https://udlbook.github.io/udlbook/).
- blauditore 3y agoI think the mathematical background starts making sense once you get a good understanding of the topic, and then people make the wrong assumption that understanding the math will help learning the overall topic, but it that's usually pretty hard. Rather than trying to form an ituition based on the theory, it's often easier to understand the technicalities after getting an intuition. This is generally true in exact sciences, especially mathematics. That's why examples are helpful.
- danielmarkbruce 3y agoUDL has some dense math notation in it. Math isn't just about proofs. It's a way to communicate. There are several different ways to communicate how a neural net functions. One is with pictures. One is with some code. One is with words. One is with some quite dense math notation.
- HighFreqAsuka 3y agoI agree with that, I think UDL uses the necessary amount of math to communicate the ideas correctly. That is obviously a good thing. What it does not do is pretend to be presenting a mathematical theory of deep learning. Basically UDL is exactly how I think current textbooks should be presented.
- ottaborra 3y agoThis makes me wonder. Is deep learning as a field an empirical science purely because everyone is afraid of the math? It has the richness of modern day physics but for some reason most the practioners seem to want to keep thinking of it as the wild west
- HighFreqAsuka 3y agoNo, there are many very mathematically inclined deep learning researchers. It's an empirical science because the mathematical tools we possess are not sufficient to describe the phenomena we observe and make predictions under one unified theory. Being an empirical science does not mean that the field is a "wild west". Deep learning models are subjectable to repeatable controlled experiments, from which you can improve your understanding of what will happen in most cases. Good practitioners know this.
- ottaborra 3y agoThe main point you're making is fair The only gripe I have is > Being an empirical science does not mean that the field is a "wild west" I think what you meant to say is: "Being an empirical science does not <b>necessarily</b> mean that the field is a \"wild west\"" you clearly haven't seen the social sciences > Good practitioners know this sure? Edit: Removed unnecessary portions that wouldn't have continued the conversation in any meaningful way
- bawolff 3y agoI think the necessarily is clearly implied from context.
- trhway 3y ago>It's an empirical science because the mathematical tools we possess are not sufficient to describe the phenomena we observe and make predictions under one unified theory. To me the deep learning is actually itself a [long-awaited] tool (which has well established, and simple at that, math underneath - gradient based optimization, vector space representation and compression) to make a good progress toward mathematical foundations of the empirical science of cognition. In the 90-ies there were works showing that for example Gabors in the first layer of the biological visual cortex are optimal for the feature based image recognition that we have. And as it happens in the DL visual NNs the convolution kernels in the first layers also converge to the Gabor-like. I see [signs of] similar convergence in the other layers (and all those semantically meaningful vector operations in the embedding space in LLMs are also very telling). Proving optimality or similar is much harder there, yet to me those "repeatable controlled experiments" (i.e. stable convergence) provide strong indication that it will be the case (as something does drive that convergence, and when there is such a drive in dynamic systems, you naturally end asymptotically up ("attracted") near something either fixed or periodic), and that would be a (or even "the") math foundation for understanding of cognition (dis-convergence from the real biological cognition, ie. emergence of completely different, yet comparable, type of cognition would also be great, if not even the much greater result) .
- runsWphotons 3y agoI like this book and everyone complaining about the math and math notation is a silly goose.
- deleted 3y ago[deleted]
- programjames 3y agoOh my, that was the first thing that I came to complain about here. The type setting is pretty awful, it makes it very difficult to read.
- c7b 3y agoSeems like a good collection of standard ML techniques, introduced with a fairly unified mathematical notation and quite a few proofs. Quite the Herculean effort (600 pages!). It just seems to me like they're putting the emphasis on the stuff that is more straightforward to formalize rather than the stuff that would be interesting to understand. Look eg at the SGD chapter. I picked this because I think optimization is one of the areas where mathematicians actually can and do make impactful contributions to ML. But then look at the chapter in the book: most of the proofs are fairly elementary (like bias-variance decompositions or Jensen inequalities), some more interesting theorems (on convergence) are cited from the literature and do not build on the lemmata, and the sub-chapters on the actually interesting methods like ADAM,... are completely free of proofs or theory. It seems to me that after reading the chapter, a reader will have a good understanding of modern SGD methods and how we got there, but they won't necessarily be much wiser about why those methods work, other than having a good intuition confirmed by numerical experiments. If that's the outcome, then I wonder what the fuss proving all the basic stuff was all for. Wouldn't it be more useful to dedicate the space to convergence proofs for ADAM (which do exist) rather than showing lots of stuff like E(XY) = E(X)E(Y) for independent random variables? That's just one chapter, I may not be doing them full justice here, although I did read through a few others as well. I first got this impression from the ANN chapter, which is ripe with long proofs for rather basic and uninteresting stuff, and from the physics-informed neural networks paper (which I actually find really nice, although it suffers a bit from the same problem as the SGD chapter). I don't want to be too critical here, it is nice in general to move towards a more rigorous and unified exposition of ML methods, and their approach should extend to the more technical results as well, just questioning where they drew the line of what to include and what not.
- modeless 3y ago> they won't necessarily be much wiser about why those methods work, other than having a good intuition confirmed by numerical experiments. This is the state of the field as a whole, isn't it? > Wouldn't it be more useful to dedicate the space to convergence proofs for ADAM (which do exist) Convergence proofs don't really explain why Adam tends to work better than other methods. It's hard to blame them for not being able to explain things that, currently, nobody understands. But I guess it kind of undermines the idea of a theory-heavy approach to teaching if the theory we have can't predict the things that are actually important.
- h423oi4j234 3y ago[flagged]
- Koshkin 3y agoI find that programming languages is a compromise between the need to communicate between a human and the (dumb) machine on the one hand, and the need to make this communication more or less readable by humans. Mathematical notation was invented to calculate (automate reasoning) and as a way of communication between (smart) humans.
- deleted 3y ago[deleted]
- sealWithIt 3y agoLooks like exactly what I was looking for :p
- majikaja 3y agoI don't think the content of the comments in this thread is limited to ML. I think there is lot of applied math research out there (almost all of it?) that hardly anyone outside of academia actually reads. I think there's some useful stuff but my impression is that research papers are mostly dead ends so I stick to graduate textbooks. Maybe other people have other approaches? I'm not a math researcher so I don't need to be at the cutting edge.
- deleted 3y ago[deleted]
- vivzkestrel 3y ago[flagged]
- chongli 3y agoI tried to look at your screenshot and holy hell what happened to imgur? That site used to be great for sharing images. Now it is enshittified to the max. I couldn’t even zoom in to look at the equations without some random animated GIFs popping up over the entire screen. I even use an ad blocker!
- CamperBob2 3y agoMake the robot overlords work for you. ChatGPT 4's data analysis plugin takes a pretty impressive stab at explaining it: https://i.imgur.com/DCAnoCu.png https://i.imgur.com/DCAnoCu.png It will also make an attempt at turning the expressions into Python. It bombed out at first but caught itself and retried without any additional prompting: https://i.imgur.com/nzywKkb.png https://i.imgur.com/nzywKkb.png Not being familiar with SymPy, though, and not having time to think it through myself, this might be a bunch of hallucinated gobbledygook. Caveat lector.
- vivzkestrel 3y agoas a guy not familiar with this math, i am not sure if i should rely on Chat GPT or not. I use GPT only for tasks I am familiar with
- CamperBob2 3y agoYou definitely shouldn't rely on it. But you should use it.
- currymj 3y agoif you have no mathematical background at all this isn’t the book for you i think. that is not really advanced mathematics although a little notationally dense. there are many good materials such as the fantastic fast.ai course that don’t require such mathematical background. if you are motivated to learn about ML, then studying the topic can gradually be a route in to more mathematical knowledge so that equations like this would not seem intimidating.
- montyanderson 3y agofor those who want some maths-heavy stuff for deep learning, check francois fluret's book https://fleuret.org/francois/lbdl.html https://fleuret.org/francois/lbdl.html. the pdf is free but the print is so cute.
- 3abiton 3y agoWhat makes it stand out compared to OP post?
- CamperBob2 3y agoHas anyone figured out a way to print the Fleuret book on A4 paper? Every other page ends up upside down when I've tried it, which is problematic with a duplexer.
- ajb 3y agoYou can probably use pdftk to rotate every other page somehow.
- zolbrek 3y agoOff topic but I'm replying here since the medieval cat thread is closed for comments. I found my copy of Catwatching and Desmond Morris does indeed claim on page 12 that cats were persecuted in the Middle Ages. "These good times for cats were not to last, however. In the Middle Ages the feline population of Europe was to experience several centuries of torture, torment, and death at the instigation of the Christian church. Because they had been involved in earlier pagan rituals, cats were proclaimed evil creatures, the agents of Satan and familiars of witches. Christians everywhere were urged to inflict as much pain and suffering on them as possible. The sacred had become the dammed. Cats were publicly burned alive on Christian feast days. Hundreds of thousands of them were flayed, crucified, beaten, roasted, and thrown from the tops of church towers at the urging of the priesthood, as part of a vicious purge against the supposed enemies of Christ."
- ajb 3y ago
- up2isomorphism 3y ago[flagged]
- j2kun 3y ago> in couple of weeks you can have deeper understanding on this topic than most of the practitioners out there This book is just over two months old... Are you saying _you_ did this since Oct 31?
- chongli 3y agoPeople tend to hate things they don't understand more than the things that are wrong. The comments under reflect to some degree about this observation. I have noticed that most HN discussions on math topics seem to devolve into complaints about notation. There seems to be a fairly large contingent of people from the programming side of things who don't have a formal background in mathematics. I get the feeling that it isn't just frustration from a lack of understanding. If the topic was from some other technical field like organic chemistry or medicine I don't think you'd see this kind of response. I believe where the frustration comes from is adjacency: people have an expectation that they ought to be able to understand it because programming (via computer science) is adjacent to math. This expectation combined with a lack of understanding is what leads to cognitive dissonance. And that cognitive dissonance is what leads to the complaints about notation. The notation isn't the problem. Every field has its own notation, jargon, and conventions. Mathematics notation is very simple and terse but the underlying concepts can be very abstract. Understanding comes from a lot of mathematical practice, not a glossary of terms. To paraphrase Euclid: "there is no royal road to mathematics (geometry)."
- tnecniv 3y agoNot just understanding, but how to parse it. For example, I can read a theorem that introduces a bunch of variables in the first sentence that don’t get used until a few lines later and say “yeah those are probably just some constants they’ll use in an upcoming equation for an upper bound or something.” Before I was familiar with higher math, I’d say “what the hell are these where did they come from?” After reading enough books and papers, you can see a few steps ahead a lot of the time due to past experience (similar to what veteran chess players do). I can also jump to the middle of the document and see some notation that was defined earlier and guess what it is. Different authors might denote the space of continuous functions in a few different ways but they tend to be very similar for common objects. That said, there’s a lot of poorly written theory publications out there in my opinion. A big sin to me is that a lot of them will do stuff without explaining in simple English beforehand how they’re going to do it, why they’re doing it, and why it is important. The first item you can get if you’re going through the math line by line at least, but that can be arduous and often I want to understand the big picture before I dig into the details. It doesn’t take much — just a couple of sentences before or after the result can go a long way.
- two_in_one 3y agoIt's hard to call comprehensive. Transformers - one page. A picture would be nice. No "prompt engineering", no "double deep". In fact words "prompt" and "double" aren't used at all. "Recognition" is used only once outside of bibliography just for reference. Looks like theory will not catch up with practice any time soon. With looming singularity it's bit worrying.
- tnecniv 3y agoRight in the title it explains it’s a book on theory. “Prompt engineering” doesn’t really fit in any theoretical framework I’m aware of and, while I also like graphics, most theory publications are light on them. You might be looking for a different kind of book, which is fine, but I think the content matches the title. Also, a book on theory is going to lag quite a bit in terms of topics. The general process is that people discover something new and interesting empirically and publish articles on it. Other people develop theory explaining why that things work and publish articles on it. Once the theory gets crystallized, big ideas get distilled into a book.
- nephanth 3y agoIt's an introductory book though? I don't think it aims at being comprehensive That said, i do agree that more on transformers would be nice since they're becoming quite central in every field of machine learning. Prompt engineering is extremely new, vastly empirical, and theory on is still only beginning (though i do remember seeing some nice papers passing). It would probably be a mistake to include it in an introductory book I have never heard of "double deep", what is that?
- fourier456 3y agoThis just isn’t really a good book. One sign; it’s an arXiv book.
- deleted 3y ago[deleted]