9 ms·

Confession as an AI researcher; seeking advice

346 points by subroutine 9y ago

Blazespinnaker 9y ago
A technique I have found that works well is read a lot of paper and their citations, but don't dive deep. Each paper usually provides some easy to grasp insight (far too little per paper, but that is elitism for you) that you can use to get a good picture of the field. Reread papers to grasp more insight. Once you have a good overall picture, find some area/problem that really interests you, you like the math, and isn't covered well and bone up on the math techniques. Do your research and present. Showing up at thesis defenses is good too. Learn a lot from the back and forth with advisors. The key is to understand at a high level why the different math techniques are being used without actually understanding all the details. This won't be sufficient for your own work,but at least you'll have a good idea how your part fits in the scheme of things.
miloshadzic 9y ago
Most-upvoted reply is excellent.
- aashu_dwivedi 9y ago
  you should link to the reply instead, because the most upvoted reply might change.
  - gbuk2013 9y ago
    https://www.reddit.com/r/MachineLearning/comments/73n9pm/d_confession_as_an_ai_researcher_seeking_advice/ https://www.reddit.com/r/MachineLearning/comments/73n9pm/d_c...
    - deleted 9y ago
      [deleted]
- jasode 9y ago
  The top-voted reply by TillWinter may be excellent but it actually doesn't answer the question posed by the OP. TillWinter's reply is mostly about the "divide-and-conquer" approach to learning. Yes, efficient study habits are good to know but that type of answer can be generically applied to any learning endeavor. The reddit OP is asking about specific resources that help with intuition. In contrast, many math teaching materials work from an analytical and symbolic approach (e.g. axioms and properties.) Unfortunately, most people can't truly learn intuition that way and they end up in mindless "plug & chug" to pass the test.
vermontdevil 9y ago
The explanation by TillWinter about systematic task assessment is quite impressive https://www.reddit.com/r/MachineLearning/comments/73n9pm/comment/dnrsmh9 https://www.reddit.com/r/MachineLearning/comments/73n9pm/com...
pierre_d528 9y ago
"“I understood nothing, but it was really fascinating,” he said. So Scholze worked backward, figuring out what he needed to learn to make sense of the proof. “To this day, that’s to a large extent how I learn,” he said. “I never really learned the basic things like linear algebra, actually — I only assimilated it through learning some other stuff.”" https://www.quantamagazine.org/peter-scholze-and-the-future-of-arithmetic-geometry-20160628 https://www.quantamagazine.org/peter-scholze-and-the-future-...
- frostwhale 9y ago
  Yeah. Reading the post I see a guy overwhelmed by a bunch of equations and numbers. Which isn't to say he shouldn't learn them, but math is always far more intimidating when you don't understand it than other subjects.
  - pierre_d528 9y ago
    > "intimidating" Exactly. /speculation There is a point where one starts to see "behind" the symbols. It's a strange sensation, as if one could understand the ideas in a non-verbal way. The symbols become optional. Intimidation crawls back before curiosity at this point. An amazing book on the subject is: "Hadamard - The psychology of invention in the mathematical field" /speculation
    - SomeStupidPoint 9y ago
      What took me a long time -- and is still a skill I'm developing -- is to both verify and "read" the math at the same time, to see the proof and the story at the same time. At one level, you're observing a technical construction and trying to ensure that it's (mostly) sound; but at another level, you're trying to understand the broader picture of how it fits in, what the builder was trying to accomplish or what perspective of the world they're trying to share. Mathematics is -- like any language -- just the articulation of an experience, of an insight, of an understanding. As you get further into mathematics (and possess more technical skills of your own), it becomes more important to see "Oh, he's trying to apply the machinery of homotopy to type theories as a means of discussing equivalence" than it is to get bogged down in the technical details. Often, the details are wrong in the first draft, but in a fixable way. (This is extremely common in major proofs.) > There is a point where one starts to see "behind" the symbols. It's a strange sensation, as if one could understand the ideas in a non-verbal way I think at some point, you have to compile mathematics to non-verbal ideas for computational reasons -- your verbal processing skills are simply too slow and too simple compared to other systems. Your visual and motor systems are way more powerful and (in the case of motor systems) operate in high dimensions. Much like GPUs in computers, if you can find a representation of a problem that works on a specialized system, you can often get a big computational boost; in mathematics, we have to push our understanding of self and experience to the limits to find more efficient representations of ideas, so we can operate on more interesting or complex ones. I think most mathematicians work in extremely personal, non-portable internal representations, and then use the symbols as a way to create an external representation that the other mathematicians can compile into their own internal representations. If you see mathematics as extremely high level code meant to be compiled to equivalent internal representations on thousands of slightly different compilers, I think the language starts to make more sense -- it's meant to be a reverse compilation target for machine code that's been under revision for ~3000 years, so of course it looks a little funky. Ed: I will say this -- One thing I've noticed as I've gotten older is that we do a really poor job of teaching students the story of mathematics -- the human motivations, the community, the long standing projects (some have gone on for hundreds of years; some are still ongoing). I sincerely believe that for young kids (less than, say 10), it would be better for their development to teach skills 4 days a week and simply tell them part of the story on the 5th. It would make mathematics much more relatable and understandable.
empath75 9y ago
You don’t need to understand everything in every field and I doubt very many people do. When Einstein was working on general relativity, he had a lot of help from friends and colleagues who pointed him towards the math he needed. He didn’t learn differential geometry until he was already deep into general relativity. Find a level of abstraction that you’re comfortable with and learn to be okay with black boxes at the lower level, and only dig into those boxes when what’s inside them actually matters.
- joshvm 9y ago
  I think an important lesson for any grad student is to learn to read through the bullshit in papers and try and understand what the authors actually did. It helps a lot that in CS you can often see the code that the authors published along with the paper. Just staring at formulae doesn't mean much, because for all you know the author just hammed up the equations to get their paper into a top conference. That's not to say that the equations are excessive, or the authors are being misleading, but I think there is definitely an expectation in some fields that putting equations in makes your paper look clever even if they're broadly unecessary. It's also wildly different depending on the field. If you look at variational methods in computer vision, images are [continuous] mappings from some domain onto the reals (I : Ω->R3 for colour). Does that change the fact that an image in memory is just a bunch of numbers in a grid? Not really, but it's bloody confusing the first time you see it. This doesn't help with understanding the maths, but at some point you have to give up and say "This guy proved it, and someone else peer reviewed it, so I can use it to solve my problem". It's perfectly OK to stand on other people's work and still make creative contributions to your field, that's the point of research.
  - mljoe 9y ago
    >I think an important lesson for any grad student is to learn to read through the bullshit in papers and try and understand what the authors actually did. We actively work to make our writing hard to understand in this field. I do this all the time myself. I don't really need this complex looking equation to make my point. But if I don't have it in there a reviewer will think my writing is not academic enough. So there you have it. Once you go in realizing this is the case everywhere, it becomes a lot easier to understand academic papers.
indescions_2017 9y ago
I think this is fairly common first-year grad student emotional response. And quite frankly, it is the job of your mentor and department to ensure you receive sufficient training for an academic or research career. Modern AI is evolving rapidly but there is a foundation upon which everyone draws upon. The Sutton and Barto book is one such foundational text. Find a collaborator in the Math department to work with. And participate daily in stack overflow forums for math and stats, such as Cross Validated. I can also recommend CASI by Efron and Hastie. Deep historical understanding of where we are today in probabilistic inference. https://web.stanford.edu/%7Ehastie/CASI/ https://web.stanford.edu/%7Ehastie/CASI/
nonbel 9y ago
From one of the comments: >"Professional heavy math people are those who said in the 60s that the perceptron's limitations proved all AI was impossible. And in the 90s that one hidden layer was all you needed, deep learning was useless." Can anyone provide the citations for this? I was aware of the latter but not the first one. You can find people still repeating the one layer stuff up to a few years ago just by reading stackexchange.
- mannykannot 9y ago
  I wonder if the author is referring to the oft-repeated claim that Minsky's and Papert's proof that a perceptron cannot learn the Xor function had a chilling effect on research into neural networks generally, even though Minsky and Papert themselves had shown that multi-layer networks were capable of doing so [1][2]. I realize that even this alleged misunderstanding is not the same as a claim that AI is impossible. The closest attempt of a mathematical proof of the impossibility of AI that I am aware of is the Lucas-Penrose argument from Gödel's first incompleteness theorem [3]. [1] https://en.wikipedia.org/wiki/Perceptrons_(book) https://en.wikipedia.org/wiki/Perceptrons_(book) [2] Minsky M. L. and Papert S. A. 1969. Perceptrons. Cambridge, MA: MIT Press. [3] http://www.iep.utm.edu/lp-argue/ http://www.iep.utm.edu/lp-argue/
  - nonbel 9y ago
    Thanks, the first ref looks like it may be the one.
- marcosdumay 9y ago
  Besides, heavy math people were also those who proved that recurrent multi-level neural networks were Turing complete and pushed the field up from the ground. But sorry, I don't have citations. It's stuff I've read a long time ago, in random books picked at library shelves by looking at the covers :)
ssanders82 9y ago
Currently getting my Masters in AI. I'll be honest, I can understand the concepts when presented to me, but the mathematical proofs are beyond me. I've learned to be ok with that. There's just not enough time in a 2 year program to teach myself the underlying vagaries of everything I encounter.
archagon 9y ago
I like the comment that likens this sort of deeply-linked knowledge to a DAG. In my own (limited) experience, once I’ve mentally found the DAG where every node either references some other node or baseline knowledge, the learning task almost immediately switches from daunting to routine. Just work on understanding each node in the dependency chain until you get to the one you seek!
ivan_ah 9y ago
Learning all the advanced math within a few years is a hopeless endeavour. It would take decades of hard work because there is so much out there, and we need to know all of it if you want to make progress (e.g. pull out a fancy-name theorem out of nowhere to solve some practical problem). I find a better approach is to focus on a few basic ideas you need specifically for your work and digging deep in there. Nobody can be expert-level in everything, but you can be expert level in your specific domain of research. Also for ML stuff, it's hard to overemphasize the importance of understanding linear algebra really well. Here is a excerpt of a book I wrote on LA which should get you started on learning this fascinating swiss-army-knife-like subject: https://minireference.com/static/excerpts/noBSguide2LA_preview.pdf https://minireference.com/static/excerpts/noBSguide2LA_previ...
- gtani 9y ago
  What you say is correct, but that reddit thread concerns (I think) a narrow class of devs of which there are many here: US CS majors who graduate with only a few semesters of calc and discrete and are underequipped to bootstrap themselves if they need to understand, say, topology. But they have to gain some understanding, so i put a few book lists in that reddit thread, took about ... 5 minutes. BTW I really like what i've seen of your guidebooks, or whatever you call them (neo-textbooks?).
- vladislav 9y ago
  "Learning all the advanced math within a few years is a hopeless endeavour." Not in my experience. It's possible to get the equivalent of a bachelors and masters in math within two years (which is enough to overcome the issues listed in the post), but it's all you'll be doing for that period of time. Well worth it imo.
  - sangnoir 9y ago
    > It's possible to get the equivalent of a bachelors and masters in math within two years (which is enough to overcome the issues listed in the post), but it's all you'll be doing for that period of time. About that - I don't think OP (or their research group) can afford to hold off publishing and attending ML conferences for 2 whole years while brushing up on math.
internetman55 9y ago
Do you really not need familiarity with the relevant math to be admitted to AI doctoral programs? I wouldn't have thought that was the case.
jokoon 9y ago
I always had the same feeling. I'm not bad at math in general (I'm not well versed in high level maths either), but as a developer, trying to jump in the ML field seems really impossible. One would think that he could teach himself ML algorithms, but you ALWAYS end up reading math notation instead of pseudo code. To be honest the 3blue1brown videos seems really wonderful at explaining what is going on without going too deep, as the math in ML lecture seems to be trying to prove everything, and is always trying to teach using math notation all the time. I guess this is happening because most of ML is mostly coming from research since it's all new, so it's being taught mostly by people who can grok the math, meaning mathematicians, it's not taught by people who are programmers. This really shows how much math should keep being math, and not leak into fields where practice matters more. Programming languages and pseudo code are not for nothing. Computers don't talk math. So as years go by, ML will be taught more as a practice subject rather than a theory one, and things will get better. I think it's just a matter of how it's being taught, because reading code will always make more sense than reading high level math. Videos and oral explanations also help a lot.
- DLTarasi 9y ago
  You might appreciate the Deep Learning for Coders courses from Fast.ai. It's basically ML as a practice subject rather than theory as you suggested. I felt similar to you when I first started learning ML but their code first approach really helped it click for me on an intuitive level. Then you can go back and dig into the maths behind it.
afpx 9y ago
How does one expect to do ‘AI research’ without much of a background in math? Machine learning is pretty much all math. Researchers are generally expected to be experts in their field. The people writing the papers on arxiv likely spent most of their lives learning about machine learning and mathematics. Unfortunately, there’s not an easy path to become an expert. One just has to dig in and learn from the ground up. Edit: The good news is that it’s never been as easy to learn math than it is now. When I was an undergrad in math, there were almost no resources available to learn the intuitions behind the math. One just has to keep doing proofs and exercises over and hope that it would ‘click’ at some point. But, sometimes that wouldn’t happen until many years later. Nowadays, one can watch YouTube videos where experts describe the intuition behind the math. It’s awesome.
- joshvm 9y ago
  Absolutely you can do 'AI Research' without a degree in maths. Sure you need a grounding in linear algebra, stats, probability and calculus, but not much more than a CS or physics degree will teach you. That stuff is indeed learned easily, and it's not what the Reddit user is worried about. That also ignores applications of machine learning, which is also a massive (and lucrative) field. But because it's a trendy field, I think there is an obsession with people needing to understand everything theoretical that comes out for fear of missing the boat. Some of the really interesting papers that have come out over the last few years - for example artistic style transfer and Faster R-CNN - have hardly any maths in. You can count the equations on one hand in both those papers. No doubt the authors know their stuff, but how readable are those papers compared to e.g. a 100-page proof? Which did I learn more from? They're a combination of two things: intuitive network architecture and a clever loss function. The first thing is a combination of intuition and programming, the second involves a little maths, but nothing outrageous. [1] https://arxiv.org/abs/1508.06576 https://arxiv.org/abs/1508.06576 [2] https://arxiv.org/abs/1506.01497 https://arxiv.org/abs/1506.01497
  - afpx 9y ago
    You're right - I see now when he says 'AI Research' he really means doing applied ML. And, that can certainly be done without knowing everything. If the goal is to make a lot of money doing applied ML, then become a consultant and aim to know 10% more than the customers. If the goal is to create models that are relatively effective, then read tutorials, play with data, experiment, and iterate. But, if the goal is to create very effective models and be able to actually explain why they work (which I think is what many companies want), then one has to understand the math. That is, showing some interesting relationships, trends, predictions, or inferences on a data analysis portal or consumer web site is one thing. But, using ML to dispense medication, regulate a medical device, drive a power plant, or identify criminal suspects - those may require different skills. (BTW I don't mean to disparage the middle group, as that's largely what I do. But, luckily I have people in the latter group who can validate what I'm doing.)
mi_lk 9y ago
A little off-topic, but hope this would be an another reminder that despite having huge success/hype these days, deep learning/machine learning are just another tools for solving problems, and you can't go really far if you just treat it like a magic framework (i.e. import tensorflow as tf) and fail to understand its underlying principles.
tzahola 9y ago
I think he should just go on with his research and don’t bother with understanding every obscure reference in papers. One can grasp the core ideas surprisingly well even when skipping over proofs and formulas. And when the time comes to write his own papers, he should remember to intentionally make it harder to read for outsiders. E.g. instead of writing “I calculated the total error by summing the per-neuron errors”, one should write “the loss function utilized an integral over the output lattice using a discretized method by Newton et. al.”, or some other bullshit.
- gfodor 9y ago
  As an amateur who has jumped in and out of learning basic ML over the years, it has been interesting to see the web of terminology expand to the point where your post is no longer satire. Writing a dictionary or annotating papers to decipher ML-speak to basic-math speak would be a pretty worthwhile endeavor for someone (I see glimmers of this in the work being done by the folks at fast.ai) and in general would probably not remove much real information.
amelius 9y ago
Perhaps the author should write a ML tool to help sift through all the material ;)
hackernewsacct 9y ago
Starting from pre-calculus what areas of mathematics (with book recommendations) should one study rigorously to have the foundations to pursue a PhD in Machine Learning?
akhilcacharya 9y ago
How does one get into an ML PhD like this? I was under the impression it was impossible if you’re not a math double major.
- jefft255 9y ago
  I am a math double major doing a research masters in ML/CV right now, and I know plenty of pure CS majors who are doing just fine. The math that 95% of ML scientists use is not that hard to grasp. Sure, when they encounter functional analysis stuff, they start to cry inside a little, but that doesn't happen very often.
  - akhilcacharya 9y ago
    Interesting. I’m interested in a related MS/PhD with a similar math background as OP and assumed I was disqualified.
ohazi 9y ago
From TillWinter's response: > Also: doing the master is to understand that you don't know anything, and doing your doctorate is to learn the others know nothing as well. I had always heard variations on the first part -- that going to a good school was supposed to humble you by showing you how much you don't actually know. Never heard the second part. That's great.
vladislav 9y ago
Based on your post, I highly recommend taking a year or two off to focus on math only. You can get the equivalent of a bachelors and masters in pure math in just two years (if that's all you're doing with your time), and it would be enough to fix all the issues you're experiencing. Just take the pure math courses instead of computational math, as abstract and difficult as possible, it will generalize much better :). I got into math for exactly the same reasons while doing research in computer vision as an undergrad, and taking the requisite time off to learn advanced math (actually going overboard on it) has been an incredible boon to my AI career.
imh 9y ago
When I left school, I found myself in a similar boat, and decided to set a goal of getting myself the knowledge equivalent of an undergrad degree in math. I already had a physics degree under my belt, so it wasn't as long of a path as it might be for others, but over a bunch of years of self study it paid off. When it comes to your career a few years is nothing. The strong foundation pays regular dividends because learning things that use it comes so much more quickly. It's a huge, slow, painful investment, but totally doable and with tremendous ROI if you want to work with stats/ML/optimmization/really any numerical computing for a living. The reason I recommend this route is that most of the more advanced math books you will encounter will assume this stuff as the readers' common knowledge. Having that foundation, the majority of the literature is already tailored to you!
stochastic_monk 9y ago
Being a grad student, this person is at a perfect place to build up their math background. Any school almost certainly offers the following: 1. Convex Optimization -- not all problems are convex, but solutions for nonconvex problems end up primarily using convex methods with slight adaptations. 2. Stochastic Optimization -- ML is pretty much all stochastic optimization. No surprise there. 3. Statistical/Theoretical Machine Learning -- courses built around concentration bounds, PAC learnability, and the Valiant/Vapnik school of thought. This gives you what you need to talk about generalizability and sample complexity. 4. Numerical Linear Algebra -- being smart about linear algebra is most of efficient machine learning. Knowing which kinds of factorizations help you solve problems efficiently. Can you do a Gram-Schmidt factorization? Cholesky decomposition? LU factorization? When do these things fail? When do you benefit from sparse representations? 5. Graphical Models -- Markov chains, Markov fields, causal relationships, HMMs, factor graphs, forward-backward algorithm, sum-product algorithms. If you're in school, take advantage of the fact that you're in school. Once you have a grasp on these things (and you'll have to catch up on real analysis, matrix calculus, and a few other fields of math), you'll be able to start reasoning about ways to improve existing methods or come up with your own. I think a lot of it is just developing mathematical maturity to give you a vocabulary to think about things with.
- LolWolf 9y ago
  This would’ve literally been my answer. Brilliant. I can’t recommend this answer enough.
- kxyvr 9y ago
  Alright, so I do research in and produce optimization tools professionally. From my biased point of view, someone is better off learning generic optimization theory and algorithms rather than specialized versions like convex or stochastic optimization. Generally speaking, generic nonlinear optimization methods form the foundation for everything and then there's a series of tricks to specialize the algorithm. Very specifically, there are two books that I think provide a good foundation. First, is Convex Functional Analysis by Kurdila and Zabarankin. Not many people know about this book, but essentially it provides a self-contained background to prove the Generalized Weirstrass Theorem, which details the conditions necessary for the existence and/or uniqueness of an optimization problem. This is important because even convex problems don't necessarily have a minimum. For example, min exp(-x) doesn't have one, but it does have an infimum. The background necessary to understand this book is real analysis and as a quick aside I think Rudin's Principles of Mathematical Analysis is the best for this. Second, is Nocedal and Wright's Numerical Optimization book. It provides a good overview of the powerful algorithms in optimization that we should be using. Now, it's weakness is that it often cheats and uses a stronger constraint qualification than we're afforded in practice. Candidly, I find that the derivative of the constraints will not remain full rank and we will likely violate the LICQ. Further, it covers a number of algorithms that really shouldn't be used in practice, ever. That said, it does cover the good algorithms and it generally has the best presentation out of the other books. Sadly, I don't know of any killer books for numerical linear algebra. And, yes, I've read cover to cover things like Saad's Iterative Methods for Sparse Linear Systems, Trefethen and Bau's Numerical Linear Algebra, and Golub and van Loan's Matrix Computations. They're valuable and well-written, but don't quite cover what I end up having to do to make linear algebra work on my solvers. Anyway, this is all biased and opinion, so take what you will. If someone else has some of their favorite references for optimization or numerical linear algebra, I'd love to hear.
punnerud 9y ago
Sound like a need for a “imposters handbook” for Mathematics, just like there is for Computer Science: https://bigmachine.io/products/the-imposters-handbook https://bigmachine.io/products/the-imposters-handbook
mathgenius 9y ago
One thing that hasn't been mentioned: learning mathematics from talking to another human can be 10-100 times faster than getting it from books. Another thing: mathematics is huge and seems to accommodate all personality types. Pick something that turns you on and grow outwards from there. The folks on reddit seem to be obsessed with Rudin, and that's good stuff, but there's so many other roads to follow. And I'm so impressed by how much better the comments are here than on reddit! Good job HN, you rock.
Blazespinnaker 9y ago
This all does bring up a good point. Why not create a publicly accessible 'math tree' that people can use to learn about any kind of math. If there is a symbol or step they don't understand they should be able to follow it all the way down to basic counting.
arkh 9y ago
> the “utility density” of reading those 1000-page textbooks is very low. A lot of pages are not relevant, but I don’t have an efficient way to sift them out. I understand that some knowledge might be useful some day, but the reward is too sparse to justify my attention budget. The vicious cycle kicks in again. That is their main problem. All those useless pages are what becomes useful later. And we find the same kind of attitude everywhere in tech: why read a full RFC when you can assume shit and get done with a 2 paragraphs tutorial?
cosmic_ape 9y ago
That person approaches the issue as his personal problem. However, there are likely many other students around him with the same problem. It is a problem of the whole field recently. One solution would be simply to a arrange a local seminar, and understand a couple of papers in full detail. It would help to invite a couple of mathematically aware students, from mathematics, physics, or the part of cs faculty where they prove stuff. They should be able to explain and answer questions immediately, which is way more effective than reading whole books or taking courses. Those can be read for details later. If the papers for the seminar are deep learning papers, part of the outcome of the seminar is likely to be an understanding that the authors of these papers do not necessarily understand the mathematics themselves.
graycat 9y ago
I'll try to reply to the frustrations of the author of the OP. I'll give some high level views and also outline the math topics mentioned, high school through much of a Ph.D. in parts of applied math. I respond in three parts: Part I I have a good pure/applied math Ph.D. and work in applied math and computing; while I call my work applied math and not artificial intelligence (AI), machine learning (ML), or computer science, it appears from the OP that there is significant overlap between my background and work and what the OP is concerned about. The Reddit post by the guy in Germany was terrific although easy to parody as a big feature of old German culture! :-)! That post is maybe a bit over organized. I've posted too often that the best future for computer science was pure/applied math, e.g., that someone seriously interested in the future of computing should as an undergraduate just major in pure math with some applied math and, essentially f'get about anything specifically about computer science. Or, for the essential computer science, write some code in some common procedural programming language for some simple exercises, check out at the library D. Knuth's The Art of Computer Programming, Volume 3, Sorting and Searching, learn about big-O notation, the heap data structure and heap sort, as an exercise program, say, a priority queue based on the heap data structure, learn about the Gleason bound and how heap sort achieves it so is in an important sense the fastest possible sort algorithm, as a side exercise look at AVL trees, and call computer science done for an undergraduate! This is partly a joke but not entirely. Well, it appears that the OP has started to discover some of why I've said such things about math. This role for math is just a special case of the old standard situation that, in nearly all fields, the best work mathematizes the field as in, e.g., mathematical physics. Indeed, there is the old joke that good coverage of the math needed for theoretical physics is so much about just the math that can do the physics just in the footnotes. It is a standard situation that nearly everyone in the STEM fields is convinced that they need to know more math. As I read papers by computer science professors, I tend to agree! Here I'll try to help the person with their lament in the OP: (1) Start at about 50,000 feet up and begin to identify in what fields, broadly on what problems, you want to work. Remember: One of the keys to success is good, early work in problem selection. (2) If you want to work in AI, I suggest you try to regard the current headline topics in AL/ML as nearly irrelevant. Sure, for political reasons, might have to keep up, and maybe there are some current, hot applications, but I don't see that work -- first-cut, ballpark, basically empirical curve fitting to huge quantities of data -- as much of a start on AI. E.g., at one time I was hired to work in an AI project, specifically expert systems. My first reaction was that expert systems -- rule based programming, working memory and the RETE algorithm, rule firing conflict resolution -- were junk and, in particular, nothing like a good start on AI, programming style, or anything else. After 20 years my opinion has changed: Expert systems were worse than junk, as a style of programming, for much of anything, and in particular on anything like AI. Maybe I got some revenge: We were trying to use expert systems for the monitoring and management of large server farms and networks. For the monitoring, for essentially health and wellness, they were using essentially just thresholds set by hand on single variables one at a time. My view was, wrap that data and processing in all the rules possible, and still the results won't be very good. Sure, for monitoring there are two ways to be wrong, (A) a false alarm and (B) a missed detection. So, right, we're forced into statistical hypothesis testing with (i) a null hypothesis that the system is healthy, (ii) a false alarm is Type I error, (iii) a missed detection is Type II error. Quickly decide that need some statistical hypothesis tests, with the null hypothesis that the system is healthy, that are both multidimensional (treat several variables jointly) distribution-free (make no assumptions about probability distributions) and find that apparently there were none such in the literature. So, I invented a large class of such tests that, really, totally knocked the socks off expert systems for much of monitoring. And what I did was just some applied math and applied probability -- some group theory, some probability theory based on measure theory, some measure preserving as in ergodic theory, etc. So, yes, I could say that my monitoring took in training data, did some machine learning, was better at its monitoring than humans so was some artificial intelligence, was computer science, etc., but I didn't: I just called my work some applied math. E.g., I got real hypothesis tests where false alarm rate was adjustable and known in advance and some useful best possible results on detection rate. So, as I wrote in the paper, take away the mathematical assumptions, derivations, theorems and proofs, forget about false alarm rate and detection rate, and just do the specified data manipulations and call the work AI. So, such real problems and applied things are one approach to computer science, AI, ML, etc. Then, sure, for such work as in what I did for detection, need, really, the Ph.D. coursework in pure and applied math and some ability to do publishable theorems and proofs in math -- for both, net, need a pure/applied math Ph.D. So, back to AI and mostly setting aside the current headlines: For AI, I'd suggest that from 50,000 feet up start by watching https://www.youtube.com/watch?v=vc8MddDFRw4 https://www.youtube.com/watch?v=vc8MddDFRw4 with a mommy kitty cat (domestic short hair tabby) and 6 or so of her kittens, maybe only 2 days old, and how they learn. Their learning is astoundingly fast. E.g., for how they learn to use their hind legs, can see fairly directly some of just how they do that. In just that one video clip, in real time of likely just a week or so, those kittens go from nearly helpless balls of fur to young kitty cats. Easy to guess that in two months they will be safely 40 feet up a tree catching something or other, effortlessly doing gymnastic feats that would shame Olympic athletes, etc. Astounding. Okay for AI from 50,000 feet up, start by trying to guess how those kittens learn. Maybe there will be some math in that, maybe not. Then if you can implement your guess in software, if the software appears from good tests in practice to learn some things well, and if your guess is fairly general, then maybe you have some progress on AI. Here f'get about my view and confirm with some AI profs who would have to review your work, hire you for a research slot, give you a research grant, etc. So, for some research, (i) could do some math as I did for that monitoring or (ii) try to do some real AI, e.g., by starting by watching those kittens learn where maybe, eventually there will be some math. For both those research directions, notice that need (i) some overview of the real problem, (ii) some intuitive insights, (iii) some good, new ideas. Maybe the ideas will be mathematical or use math and maybe not. E.g., might go a long way on what those kittens are doing before use much math. (3) But for the math as mentioned by the OP, I'll try to give an outline: (i) Start with the real numbers, e.g., as learned likely well enough by the 9th grade. Then, sure, learn about the complex numbers. So, net, have a high school major in math, e.g., everything short of calculus. (ii) Learn college calculus well. At least in part, you can do well alone: Due to some circumstances beyond my control, for freshman calculus I just got a good book, studied, worked the exercises, and learned. Then at a college with a good math department, I started on sophomore calculus. You can do such stuff yourself. (iii) It would be good to take a course in abstract algebra, especially one where nearly all the exercises are proofs. So, learn about sets, functions, groups, rings, fields, more about the real and complex numbers, maybe some basic number theory, the greatest common divisor and least common multiple algorithms, and some about vector spaces. Might touch on cryptography and coding theory. Really the more important, maybe main, value of the course is just learning how to write proofs. The math there is nearly all just so childishly simple that it's easy to learn to write very highly precise proofs -- crucial stuff if later want to publish theorems and proofs. Blunt fact of life, politically incorrect observation: Without such training in writing theorems and proofs, and, really, just in math notation and how to do math derivations, tough ever to learn how. So, can find chaired professors of computer science at top computer science departments in top US research universities who, however, fumble terribly with just standard math notation and, especially, with how to write theorems and proofs.
ryanx435 9y ago
The depth and illegibility of the field he is describing make me believe that we are much further away from general purpose ai than I previously thought
amai 9y ago
Remember: "Complex models are rarely useful (unless for those writing their dissertations)." (V.I.Arnold)
rdlecler1 9y ago
As Alan Kay noted, the right point of view can add 80 IQ. I was in a quantitatively heavy field and always felt out classed by those with strong physics and maths backgrounds. Nevertheless I published to papers in Nature journals and overturned about 10 years of high profile research, not because I was smarter but because I spent more time trying to find the right perspective and when I found anomalies instead of brushing over it, confident in my own intelligence, I instead drilled down until I found the root of the problem — something that everyone else had overlooked. You don’t need to be a classical genius to make a contribution but you probably do have to be tenacious.
agentofoblivion 9y ago
I think the real issue is that the OP has a mistaken expectation that they should understand everything. For instance, the group that wrote the Wasserstein GAN paper are surely those that think night and day about distance metrics. And they might be totally lost reading a paper about some energy based method that relies on concepts from physics. The point is that researchers have their little niche and they try to make contributions in areas adjacent to it. It's unrealistic to think everyone publishing papers understand all the other papers, particularly in such a cross-disciplinary field like ML. There's also a big gap between a researcher deep in their career and a student fresh out of a masters program. It's also hard to transition from someone who's used to reading and understanding textbooks to someone who's often reading really technical research and understanding very little of it at first. You just have to push through and have confidence that you'll eventually learn enough to make a contribution. That's what it means to "become an expert"--you start off as not being an expert and then beat your head against the wall for a few years until you bootstrap your way out of it. And if you want to do it in a reasonable amount of time, you should probably choose something you have some of the fundamentals for.
tabtab 9y ago
Perhaps you can focus on improving the usability of AI tool-sets to a wider market rather than focus on finding the Next Big Magic Equation. Example: https://github.com/RowColz/AI https://github.com/RowColz/AI An AI expert(s) may do the initial setup, but factor tables allow more "typical" office workers to tune and prune the results.
togelius 9y ago
I am an AI researcher and faculty member at a large and famous university. I probably know less math than that Reddit poster. Math is important if you are specifically interested in the math of AI. If you are interested in inventing algorithms and solutions you mostly don't need the math.

346 points by subroutine 9y ago

A technique I have found that works well is read a lot of paper and their citations, but don't dive deep. Each paper usually provides some easy to grasp insight (far too little per paper, but that is elitism for you) that you can use to get a good picture of the field. Reread papers to grasp more insight. Once you have a good overall picture, find some area/problem that really interests you, you like the math, and isn't covered well and bone up on the math techniques. Do your research and present. Showing up at thesis defenses is good too. Learn a lot from the back and forth with advisors. The key is to understand at a high level why the different math techniques are being used without actually understanding all the details. This won't be sufficient for your own work,but at least you'll have a good idea how your part fits in the scheme of things.

miloshadzic 9y ago

Most-upvoted reply is excellent.

aashu_dwivedi 9y ago
you should link to the reply instead, because the most upvoted reply might change.
- gbuk2013 9y ago
  https://www.reddit.com/r/MachineLearning/comments/73n9pm/d_confession_as_an_ai_researcher_seeking_advice/ https://www.reddit.com/r/MachineLearning/comments/73n9pm/d_c...
  - deleted 9y ago
    [deleted]
jasode 9y ago
The top-voted reply by TillWinter may be excellent but it actually doesn't answer the question posed by the OP. TillWinter's reply is mostly about the "divide-and-conquer" approach to learning. Yes, efficient study habits are good to know but that type of answer can be generically applied to any learning endeavor. The reddit OP is asking about specific resources that help with intuition. In contrast, many math teaching materials work from an analytical and symbolic approach (e.g. axioms and properties.) Unfortunately, most people can't truly learn intuition that way and they end up in mindless "plug & chug" to pass the test.

vermontdevil 9y ago

The explanation by TillWinter about systematic task assessment is quite impressive https://www.reddit.com/r/MachineLearning/comments/73n9pm/comment/dnrsmh9 https://www.reddit.com/r/MachineLearning/comments/73n9pm/com...

pierre_d528 9y ago

"“I understood nothing, but it was really fascinating,” he said. So Scholze worked backward, figuring out what he needed to learn to make sense of the proof. “To this day, that’s to a large extent how I learn,” he said. “I never really learned the basic things like linear algebra, actually — I only assimilated it through learning some other stuff.”" https://www.quantamagazine.org/peter-scholze-and-the-future-of-arithmetic-geometry-20160628 https://www.quantamagazine.org/peter-scholze-and-the-future-...

frostwhale 9y ago
Yeah. Reading the post I see a guy overwhelmed by a bunch of equations and numbers. Which isn't to say he shouldn't learn them, but math is always far more intimidating when you don't understand it than other subjects.
- pierre_d528 9y ago
  > "intimidating" Exactly. /speculation There is a point where one starts to see "behind" the symbols. It's a strange sensation, as if one could understand the ideas in a non-verbal way. The symbols become optional. Intimidation crawls back before curiosity at this point. An amazing book on the subject is: "Hadamard - The psychology of invention in the mathematical field" /speculation
  - SomeStupidPoint 9y ago
    What took me a long time -- and is still a skill I'm developing -- is to both verify and "read" the math at the same time, to see the proof and the story at the same time. At one level, you're observing a technical construction and trying to ensure that it's (mostly) sound; but at another level, you're trying to understand the broader picture of how it fits in, what the builder was trying to accomplish or what perspective of the world they're trying to share. Mathematics is -- like any language -- just the articulation of an experience, of an insight, of an understanding. As you get further into mathematics (and possess more technical skills of your own), it becomes more important to see "Oh, he's trying to apply the machinery of homotopy to type theories as a means of discussing equivalence" than it is to get bogged down in the technical details. Often, the details are wrong in the first draft, but in a fixable way. (This is extremely common in major proofs.) > There is a point where one starts to see "behind" the symbols. It's a strange sensation, as if one could understand the ideas in a non-verbal way I think at some point, you have to compile mathematics to non-verbal ideas for computational reasons -- your verbal processing skills are simply too slow and too simple compared to other systems. Your visual and motor systems are way more powerful and (in the case of motor systems) operate in high dimensions. Much like GPUs in computers, if you can find a representation of a problem that works on a specialized system, you can often get a big computational boost; in mathematics, we have to push our understanding of self and experience to the limits to find more efficient representations of ideas, so we can operate on more interesting or complex ones. I think most mathematicians work in extremely personal, non-portable internal representations, and then use the symbols as a way to create an external representation that the other mathematicians can compile into their own internal representations. If you see mathematics as extremely high level code meant to be compiled to equivalent internal representations on thousands of slightly different compilers, I think the language starts to make more sense -- it's meant to be a reverse compilation target for machine code that's been under revision for ~3000 years, so of course it looks a little funky. Ed: I will say this -- One thing I've noticed as I've gotten older is that we do a really poor job of teaching students the story of mathematics -- the human motivations, the community, the long standing projects (some have gone on for hundreds of years; some are still ongoing). I sincerely believe that for young kids (less than, say 10), it would be better for their development to teach skills 4 days a week and simply tell them part of the story on the 5th. It would make mathematics much more relatable and understandable.

empath75 9y ago

You don’t need to understand everything in every field and I doubt very many people do. When Einstein was working on general relativity, he had a lot of help from friends and colleagues who pointed him towards the math he needed. He didn’t learn differential geometry until he was already deep into general relativity. Find a level of abstraction that you’re comfortable with and learn to be okay with black boxes at the lower level, and only dig into those boxes when what’s inside them actually matters.

joshvm 9y ago
I think an important lesson for any grad student is to learn to read through the bullshit in papers and try and understand what the authors actually did. It helps a lot that in CS you can often see the code that the authors published along with the paper. Just staring at formulae doesn't mean much, because for all you know the author just hammed up the equations to get their paper into a top conference. That's not to say that the equations are excessive, or the authors are being misleading, but I think there is definitely an expectation in some fields that putting equations in makes your paper look clever even if they're broadly unecessary. It's also wildly different depending on the field. If you look at variational methods in computer vision, images are [continuous] mappings from some domain onto the reals (I : Ω->R3 for colour). Does that change the fact that an image in memory is just a bunch of numbers in a grid? Not really, but it's bloody confusing the first time you see it. This doesn't help with understanding the maths, but at some point you have to give up and say "This guy proved it, and someone else peer reviewed it, so I can use it to solve my problem". It's perfectly OK to stand on other people's work and still make creative contributions to your field, that's the point of research.
- mljoe 9y ago
  >I think an important lesson for any grad student is to learn to read through the bullshit in papers and try and understand what the authors actually did. We actively work to make our writing hard to understand in this field. I do this all the time myself. I don't really need this complex looking equation to make my point. But if I don't have it in there a reviewer will think my writing is not academic enough. So there you have it. Once you go in realizing this is the case everywhere, it becomes a lot easier to understand academic papers.

indescions_2017 9y ago

I think this is fairly common first-year grad student emotional response. And quite frankly, it is the job of your mentor and department to ensure you receive sufficient training for an academic or research career. Modern AI is evolving rapidly but there is a foundation upon which everyone draws upon. The Sutton and Barto book is one such foundational text. Find a collaborator in the Math department to work with. And participate daily in stack overflow forums for math and stats, such as Cross Validated. I can also recommend CASI by Efron and Hastie. Deep historical understanding of where we are today in probabilistic inference. https://web.stanford.edu/%7Ehastie/CASI/ https://web.stanford.edu/%7Ehastie/CASI/

nonbel 9y ago

From one of the comments: >"Professional heavy math people are those who said in the 60s that the perceptron's limitations proved all AI was impossible. And in the 90s that one hidden layer was all you needed, deep learning was useless." Can anyone provide the citations for this? I was aware of the latter but not the first one. You can find people still repeating the one layer stuff up to a few years ago just by reading stackexchange.

mannykannot 9y ago
I wonder if the author is referring to the oft-repeated claim that Minsky's and Papert's proof that a perceptron cannot learn the Xor function had a chilling effect on research into neural networks generally, even though Minsky and Papert themselves had shown that multi-layer networks were capable of doing so [1][2]. I realize that even this alleged misunderstanding is not the same as a claim that AI is impossible. The closest attempt of a mathematical proof of the impossibility of AI that I am aware of is the Lucas-Penrose argument from Gödel's first incompleteness theorem [3]. [1] https://en.wikipedia.org/wiki/Perceptrons_(book) https://en.wikipedia.org/wiki/Perceptrons_(book) [2] Minsky M. L. and Papert S. A. 1969. Perceptrons. Cambridge, MA: MIT Press. [3] http://www.iep.utm.edu/lp-argue/ http://www.iep.utm.edu/lp-argue/
- nonbel 9y ago
  Thanks, the first ref looks like it may be the one.
marcosdumay 9y ago
Besides, heavy math people were also those who proved that recurrent multi-level neural networks were Turing complete and pushed the field up from the ground. But sorry, I don't have citations. It's stuff I've read a long time ago, in random books picked at library shelves by looking at the covers :)

ssanders82 9y ago

Currently getting my Masters in AI. I'll be honest, I can understand the concepts when presented to me, but the mathematical proofs are beyond me. I've learned to be ok with that. There's just not enough time in a 2 year program to teach myself the underlying vagaries of everything I encounter.

archagon 9y ago

I like the comment that likens this sort of deeply-linked knowledge to a DAG. In my own (limited) experience, once I’ve mentally found the DAG where every node either references some other node or baseline knowledge, the learning task almost immediately switches from daunting to routine. Just work on understanding each node in the dependency chain until you get to the one you seek!

ivan_ah 9y ago

Learning all the advanced math within a few years is a hopeless endeavour. It would take decades of hard work because there is so much out there, and we need to know all of it if you want to make progress (e.g. pull out a fancy-name theorem out of nowhere to solve some practical problem). I find a better approach is to focus on a few basic ideas you need specifically for your work and digging deep in there. Nobody can be expert-level in everything, but you can be expert level in your specific domain of research. Also for ML stuff, it's hard to overemphasize the importance of understanding linear algebra really well. Here is a excerpt of a book I wrote on LA which should get you started on learning this fascinating swiss-army-knife-like subject: https://minireference.com/static/excerpts/noBSguide2LA_preview.pdf https://minireference.com/static/excerpts/noBSguide2LA_previ...

gtani 9y ago
What you say is correct, but that reddit thread concerns (I think) a narrow class of devs of which there are many here: US CS majors who graduate with only a few semesters of calc and discrete and are underequipped to bootstrap themselves if they need to understand, say, topology. But they have to gain some understanding, so i put a few book lists in that reddit thread, took about ... 5 minutes. BTW I really like what i've seen of your guidebooks, or whatever you call them (neo-textbooks?).
vladislav 9y ago
"Learning all the advanced math within a few years is a hopeless endeavour." Not in my experience. It's possible to get the equivalent of a bachelors and masters in math within two years (which is enough to overcome the issues listed in the post), but it's all you'll be doing for that period of time. Well worth it imo.
- sangnoir 9y ago
  > It's possible to get the equivalent of a bachelors and masters in math within two years (which is enough to overcome the issues listed in the post), but it's all you'll be doing for that period of time. About that - I don't think OP (or their research group) can afford to hold off publishing and attending ML conferences for 2 whole years while brushing up on math.

internetman55 9y ago

Do you really not need familiarity with the relevant math to be admitted to AI doctoral programs? I wouldn't have thought that was the case.

jokoon 9y ago

I always had the same feeling. I'm not bad at math in general (I'm not well versed in high level maths either), but as a developer, trying to jump in the ML field seems really impossible. One would think that he could teach himself ML algorithms, but you ALWAYS end up reading math notation instead of pseudo code. To be honest the 3blue1brown videos seems really wonderful at explaining what is going on without going too deep, as the math in ML lecture seems to be trying to prove everything, and is always trying to teach using math notation all the time. I guess this is happening because most of ML is mostly coming from research since it's all new, so it's being taught mostly by people who can grok the math, meaning mathematicians, it's not taught by people who are programmers. This really shows how much math should keep being math, and not leak into fields where practice matters more. Programming languages and pseudo code are not for nothing. Computers don't talk math. So as years go by, ML will be taught more as a practice subject rather than a theory one, and things will get better. I think it's just a matter of how it's being taught, because reading code will always make more sense than reading high level math. Videos and oral explanations also help a lot.

DLTarasi 9y ago
You might appreciate the Deep Learning for Coders courses from Fast.ai. It's basically ML as a practice subject rather than theory as you suggested. I felt similar to you when I first started learning ML but their code first approach really helped it click for me on an intuitive level. Then you can go back and dig into the maths behind it.

afpx 9y ago

How does one expect to do ‘AI research’ without much of a background in math? Machine learning is pretty much all math. Researchers are generally expected to be experts in their field. The people writing the papers on arxiv likely spent most of their lives learning about machine learning and mathematics. Unfortunately, there’s not an easy path to become an expert. One just has to dig in and learn from the ground up. Edit: The good news is that it’s never been as easy to learn math than it is now. When I was an undergrad in math, there were almost no resources available to learn the intuitions behind the math. One just has to keep doing proofs and exercises over and hope that it would ‘click’ at some point. But, sometimes that wouldn’t happen until many years later. Nowadays, one can watch YouTube videos where experts describe the intuition behind the math. It’s awesome.

joshvm 9y ago
Absolutely you can do 'AI Research' without a degree in maths. Sure you need a grounding in linear algebra, stats, probability and calculus, but not much more than a CS or physics degree will teach you. That stuff is indeed learned easily, and it's not what the Reddit user is worried about. That also ignores applications of machine learning, which is also a massive (and lucrative) field. But because it's a trendy field, I think there is an obsession with people needing to understand everything theoretical that comes out for fear of missing the boat. Some of the really interesting papers that have come out over the last few years - for example artistic style transfer and Faster R-CNN - have hardly any maths in. You can count the equations on one hand in both those papers. No doubt the authors know their stuff, but how readable are those papers compared to e.g. a 100-page proof? Which did I learn more from? They're a combination of two things: intuitive network architecture and a clever loss function. The first thing is a combination of intuition and programming, the second involves a little maths, but nothing outrageous. [1] https://arxiv.org/abs/1508.06576 https://arxiv.org/abs/1508.06576 [2] https://arxiv.org/abs/1506.01497 https://arxiv.org/abs/1506.01497
- afpx 9y ago
  You're right - I see now when he says 'AI Research' he really means doing applied ML. And, that can certainly be done without knowing everything. If the goal is to make a lot of money doing applied ML, then become a consultant and aim to know 10% more than the customers. If the goal is to create models that are relatively effective, then read tutorials, play with data, experiment, and iterate. But, if the goal is to create very effective models and be able to actually explain why they work (which I think is what many companies want), then one has to understand the math. That is, showing some interesting relationships, trends, predictions, or inferences on a data analysis portal or consumer web site is one thing. But, using ML to dispense medication, regulate a medical device, drive a power plant, or identify criminal suspects - those may require different skills. (BTW I don't mean to disparage the middle group, as that's largely what I do. But, luckily I have people in the latter group who can validate what I'm doing.)

mi_lk 9y ago

A little off-topic, but hope this would be an another reminder that despite having huge success/hype these days, deep learning/machine learning are just another tools for solving problems, and you can't go really far if you just treat it like a magic framework (i.e. import tensorflow as tf) and fail to understand its underlying principles.

tzahola 9y ago

I think he should just go on with his research and don’t bother with understanding every obscure reference in papers. One can grasp the core ideas surprisingly well even when skipping over proofs and formulas. And when the time comes to write his own papers, he should remember to intentionally make it harder to read for outsiders. E.g. instead of writing “I calculated the total error by summing the per-neuron errors”, one should write “the loss function utilized an integral over the output lattice using a discretized method by Newton et. al.”, or some other bullshit.

gfodor 9y ago
As an amateur who has jumped in and out of learning basic ML over the years, it has been interesting to see the web of terminology expand to the point where your post is no longer satire. Writing a dictionary or annotating papers to decipher ML-speak to basic-math speak would be a pretty worthwhile endeavor for someone (I see glimmers of this in the work being done by the folks at fast.ai) and in general would probably not remove much real information.

amelius 9y ago

Perhaps the author should write a ML tool to help sift through all the material ;)

hackernewsacct 9y ago

Starting from pre-calculus what areas of mathematics (with book recommendations) should one study rigorously to have the foundations to pursue a PhD in Machine Learning?

akhilcacharya 9y ago

How does one get into an ML PhD like this? I was under the impression it was impossible if you’re not a math double major.

jefft255 9y ago
I am a math double major doing a research masters in ML/CV right now, and I know plenty of pure CS majors who are doing just fine. The math that 95% of ML scientists use is not that hard to grasp. Sure, when they encounter functional analysis stuff, they start to cry inside a little, but that doesn't happen very often.
- akhilcacharya 9y ago
  Interesting. I’m interested in a related MS/PhD with a similar math background as OP and assumed I was disqualified.

ohazi 9y ago

From TillWinter's response: > Also: doing the master is to understand that you don't know anything, and doing your doctorate is to learn the others know nothing as well. I had always heard variations on the first part -- that going to a good school was supposed to humble you by showing you how much you don't actually know. Never heard the second part. That's great.

vladislav 9y ago

Based on your post, I highly recommend taking a year or two off to focus on math only. You can get the equivalent of a bachelors and masters in pure math in just two years (if that's all you're doing with your time), and it would be enough to fix all the issues you're experiencing. Just take the pure math courses instead of computational math, as abstract and difficult as possible, it will generalize much better :). I got into math for exactly the same reasons while doing research in computer vision as an undergrad, and taking the requisite time off to learn advanced math (actually going overboard on it) has been an incredible boon to my AI career.

imh 9y ago

When I left school, I found myself in a similar boat, and decided to set a goal of getting myself the knowledge equivalent of an undergrad degree in math. I already had a physics degree under my belt, so it wasn't as long of a path as it might be for others, but over a bunch of years of self study it paid off. When it comes to your career a few years is nothing. The strong foundation pays regular dividends because learning things that use it comes so much more quickly. It's a huge, slow, painful investment, but totally doable and with tremendous ROI if you want to work with stats/ML/optimmization/really any numerical computing for a living. The reason I recommend this route is that most of the more advanced math books you will encounter will assume this stuff as the readers' common knowledge. Having that foundation, the majority of the literature is already tailored to you!

stochastic_monk 9y ago

Being a grad student, this person is at a perfect place to build up their math background. Any school almost certainly offers the following: 1. Convex Optimization -- not all problems are convex, but solutions for nonconvex problems end up primarily using convex methods with slight adaptations. 2. Stochastic Optimization -- ML is pretty much all stochastic optimization. No surprise there. 3. Statistical/Theoretical Machine Learning -- courses built around concentration bounds, PAC learnability, and the Valiant/Vapnik school of thought. This gives you what you need to talk about generalizability and sample complexity. 4. Numerical Linear Algebra -- being smart about linear algebra is most of efficient machine learning. Knowing which kinds of factorizations help you solve problems efficiently. Can you do a Gram-Schmidt factorization? Cholesky decomposition? LU factorization? When do these things fail? When do you benefit from sparse representations? 5. Graphical Models -- Markov chains, Markov fields, causal relationships, HMMs, factor graphs, forward-backward algorithm, sum-product algorithms. If you're in school, take advantage of the fact that you're in school. Once you have a grasp on these things (and you'll have to catch up on real analysis, matrix calculus, and a few other fields of math), you'll be able to start reasoning about ways to improve existing methods or come up with your own. I think a lot of it is just developing mathematical maturity to give you a vocabulary to think about things with.

LolWolf 9y ago
This would’ve literally been my answer. Brilliant. I can’t recommend this answer enough.
kxyvr 9y ago
Alright, so I do research in and produce optimization tools professionally. From my biased point of view, someone is better off learning generic optimization theory and algorithms rather than specialized versions like convex or stochastic optimization. Generally speaking, generic nonlinear optimization methods form the foundation for everything and then there's a series of tricks to specialize the algorithm. Very specifically, there are two books that I think provide a good foundation. First, is Convex Functional Analysis by Kurdila and Zabarankin. Not many people know about this book, but essentially it provides a self-contained background to prove the Generalized Weirstrass Theorem, which details the conditions necessary for the existence and/or uniqueness of an optimization problem. This is important because even convex problems don't necessarily have a minimum. For example, min exp(-x) doesn't have one, but it does have an infimum. The background necessary to understand this book is real analysis and as a quick aside I think Rudin's Principles of Mathematical Analysis is the best for this. Second, is Nocedal and Wright's Numerical Optimization book. It provides a good overview of the powerful algorithms in optimization that we should be using. Now, it's weakness is that it often cheats and uses a stronger constraint qualification than we're afforded in practice. Candidly, I find that the derivative of the constraints will not remain full rank and we will likely violate the LICQ. Further, it covers a number of algorithms that really shouldn't be used in practice, ever. That said, it does cover the good algorithms and it generally has the best presentation out of the other books. Sadly, I don't know of any killer books for numerical linear algebra. And, yes, I've read cover to cover things like Saad's Iterative Methods for Sparse Linear Systems, Trefethen and Bau's Numerical Linear Algebra, and Golub and van Loan's Matrix Computations. They're valuable and well-written, but don't quite cover what I end up having to do to make linear algebra work on my solvers. Anyway, this is all biased and opinion, so take what you will. If someone else has some of their favorite references for optimization or numerical linear algebra, I'd love to hear.

punnerud 9y ago

Sound like a need for a “imposters handbook” for Mathematics, just like there is for Computer Science: https://bigmachine.io/products/the-imposters-handbook https://bigmachine.io/products/the-imposters-handbook

mathgenius 9y ago

One thing that hasn't been mentioned: learning mathematics from talking to another human can be 10-100 times faster than getting it from books. Another thing: mathematics is huge and seems to accommodate all personality types. Pick something that turns you on and grow outwards from there. The folks on reddit seem to be obsessed with Rudin, and that's good stuff, but there's so many other roads to follow. And I'm so impressed by how much better the comments are here than on reddit! Good job HN, you rock.

Blazespinnaker 9y ago

This all does bring up a good point. Why not create a publicly accessible 'math tree' that people can use to learn about any kind of math. If there is a symbol or step they don't understand they should be able to follow it all the way down to basic counting.

arkh 9y ago

> the “utility density” of reading those 1000-page textbooks is very low. A lot of pages are not relevant, but I don’t have an efficient way to sift them out. I understand that some knowledge might be useful some day, but the reward is too sparse to justify my attention budget. The vicious cycle kicks in again. That is their main problem. All those useless pages are what becomes useful later. And we find the same kind of attitude everywhere in tech: why read a full RFC when you can assume shit and get done with a 2 paragraphs tutorial?

cosmic_ape 9y ago

That person approaches the issue as his personal problem. However, there are likely many other students around him with the same problem. It is a problem of the whole field recently. One solution would be simply to a arrange a local seminar, and understand a couple of papers in full detail. It would help to invite a couple of mathematically aware students, from mathematics, physics, or the part of cs faculty where they prove stuff. They should be able to explain and answer questions immediately, which is way more effective than reading whole books or taking courses. Those can be read for details later. If the papers for the seminar are deep learning papers, part of the outcome of the seminar is likely to be an understanding that the authors of these papers do not necessarily understand the mathematics themselves.

graycat 9y ago

I'll try to reply to the frustrations of the author of the OP. I'll give some high level views and also outline the math topics mentioned, high school through much of a Ph.D. in parts of applied math. I respond in three parts: Part I I have a good pure/applied math Ph.D. and work in applied math and computing; while I call my work applied math and not artificial intelligence (AI), machine learning (ML), or computer science, it appears from the OP that there is significant overlap between my background and work and what the OP is concerned about. The Reddit post by the guy in Germany was terrific although easy to parody as a big feature of old German culture! :-)! That post is maybe a bit over organized. I've posted too often that the best future for computer science was pure/applied math, e.g., that someone seriously interested in the future of computing should as an undergraduate just major in pure math with some applied math and, essentially f'get about anything specifically about computer science. Or, for the essential computer science, write some code in some common procedural programming language for some simple exercises, check out at the library D. Knuth's The Art of Computer Programming, Volume 3, Sorting and Searching, learn about big-O notation, the heap data structure and heap sort, as an exercise program, say, a priority queue based on the heap data structure, learn about the Gleason bound and how heap sort achieves it so is in an important sense the fastest possible sort algorithm, as a side exercise look at AVL trees, and call computer science done for an undergraduate! This is partly a joke but not entirely. Well, it appears that the OP has started to discover some of why I've said such things about math. This role for math is just a special case of the old standard situation that, in nearly all fields, the best work mathematizes the field as in, e.g., mathematical physics. Indeed, there is the old joke that good coverage of the math needed for theoretical physics is so much about just the math that can do the physics just in the footnotes. It is a standard situation that nearly everyone in the STEM fields is convinced that they need to know more math. As I read papers by computer science professors, I tend to agree! Here I'll try to help the person with their lament in the OP: (1) Start at about 50,000 feet up and begin to identify in what fields, broadly on what problems, you want to work. Remember: One of the keys to success is good, early work in problem selection. (2) If you want to work in AI, I suggest you try to regard the current headline topics in AL/ML as nearly irrelevant. Sure, for political reasons, might have to keep up, and maybe there are some current, hot applications, but I don't see that work -- first-cut, ballpark, basically empirical curve fitting to huge quantities of data -- as much of a start on AI. E.g., at one time I was hired to work in an AI project, specifically expert systems. My first reaction was that expert systems -- rule based programming, working memory and the RETE algorithm, rule firing conflict resolution -- were junk and, in particular, nothing like a good start on AI, programming style, or anything else. After 20 years my opinion has changed: Expert systems were worse than junk, as a style of programming, for much of anything, and in particular on anything like AI. Maybe I got some revenge: We were trying to use expert systems for the monitoring and management of large server farms and networks. For the monitoring, for essentially health and wellness, they were using essentially just thresholds set by hand on single variables one at a time. My view was, wrap that data and processing in all the rules possible, and still the results won't be very good. Sure, for monitoring there are two ways to be wrong, (A) a false alarm and (B) a missed detection. So, right, we're forced into statistical hypothesis testing with (i) a null hypothesis that the system is healthy, (ii) a false alarm is Type I error, (iii) a missed detection is Type II error. Quickly decide that need some statistical hypothesis tests, with the null hypothesis that the system is healthy, that are both multidimensional (treat several variables jointly) distribution-free (make no assumptions about probability distributions) and find that apparently there were none such in the literature. So, I invented a large class of such tests that, really, totally knocked the socks off expert systems for much of monitoring. And what I did was just some applied math and applied probability -- some group theory, some probability theory based on measure theory, some measure preserving as in ergodic theory, etc. So, yes, I could say that my monitoring took in training data, did some machine learning, was better at its monitoring than humans so was some artificial intelligence, was computer science, etc., but I didn't: I just called my work some applied math. E.g., I got real hypothesis tests where false alarm rate was adjustable and known in advance and some useful best possible results on detection rate. So, as I wrote in the paper, take away the mathematical assumptions, derivations, theorems and proofs, forget about false alarm rate and detection rate, and just do the specified data manipulations and call the work AI. So, such real problems and applied things are one approach to computer science, AI, ML, etc. Then, sure, for such work as in what I did for detection, need, really, the Ph.D. coursework in pure and applied math and some ability to do publishable theorems and proofs in math -- for both, net, need a pure/applied math Ph.D. So, back to AI and mostly setting aside the current headlines: For AI, I'd suggest that from 50,000 feet up start by watching https://www.youtube.com/watch?v=vc8MddDFRw4 https://www.youtube.com/watch?v=vc8MddDFRw4 with a mommy kitty cat (domestic short hair tabby) and 6 or so of her kittens, maybe only 2 days old, and how they learn. Their learning is astoundingly fast. E.g., for how they learn to use their hind legs, can see fairly directly some of just how they do that. In just that one video clip, in real time of likely just a week or so, those kittens go from nearly helpless balls of fur to young kitty cats. Easy to guess that in two months they will be safely 40 feet up a tree catching something or other, effortlessly doing gymnastic feats that would shame Olympic athletes, etc. Astounding. Okay for AI from 50,000 feet up, start by trying to guess how those kittens learn. Maybe there will be some math in that, maybe not. Then if you can implement your guess in software, if the software appears from good tests in practice to learn some things well, and if your guess is fairly general, then maybe you have some progress on AI. Here f'get about my view and confirm with some AI profs who would have to review your work, hire you for a research slot, give you a research grant, etc. So, for some research, (i) could do some math as I did for that monitoring or (ii) try to do some real AI, e.g., by starting by watching those kittens learn where maybe, eventually there will be some math. For both those research directions, notice that need (i) some overview of the real problem, (ii) some intuitive insights, (iii) some good, new ideas. Maybe the ideas will be mathematical or use math and maybe not. E.g., might go a long way on what those kittens are doing before use much math. (3) But for the math as mentioned by the OP, I'll try to give an outline: (i) Start with the real numbers, e.g., as learned likely well enough by the 9th grade. Then, sure, learn about the complex numbers. So, net, have a high school major in math, e.g., everything short of calculus. (ii) Learn college calculus well. At least in part, you can do well alone: Due to some circumstances beyond my control, for freshman calculus I just got a good book, studied, worked the exercises, and learned. Then at a college with a good math department, I started on sophomore calculus. You can do such stuff yourself. (iii) It would be good to take a course in abstract algebra, especially one where nearly all the exercises are proofs. So, learn about sets, functions, groups, rings, fields, more about the real and complex numbers, maybe some basic number theory, the greatest common divisor and least common multiple algorithms, and some about vector spaces. Might touch on cryptography and coding theory. Really the more important, maybe main, value of the course is just learning how to write proofs. The math there is nearly all just so childishly simple that it's easy to learn to write very highly precise proofs -- crucial stuff if later want to publish theorems and proofs. Blunt fact of life, politically incorrect observation: Without such training in writing theorems and proofs, and, really, just in math notation and how to do math derivations, tough ever to learn how. So, can find chaired professors of computer science at top computer science departments in top US research universities who, however, fumble terribly with just standard math notation and, especially, with how to write theorems and proofs.

ryanx435 9y ago

The depth and illegibility of the field he is describing make me believe that we are much further away from general purpose ai than I previously thought

amai 9y ago

Remember: "Complex models are rarely useful (unless for those writing their dissertations)." (V.I.Arnold)

rdlecler1 9y ago

As Alan Kay noted, the right point of view can add 80 IQ. I was in a quantitatively heavy field and always felt out classed by those with strong physics and maths backgrounds. Nevertheless I published to papers in Nature journals and overturned about 10 years of high profile research, not because I was smarter but because I spent more time trying to find the right perspective and when I found anomalies instead of brushing over it, confident in my own intelligence, I instead drilled down until I found the root of the problem — something that everyone else had overlooked. You don’t need to be a classical genius to make a contribution but you probably do have to be tenacious.

agentofoblivion 9y ago

I think the real issue is that the OP has a mistaken expectation that they should understand everything. For instance, the group that wrote the Wasserstein GAN paper are surely those that think night and day about distance metrics. And they might be totally lost reading a paper about some energy based method that relies on concepts from physics. The point is that researchers have their little niche and they try to make contributions in areas adjacent to it. It's unrealistic to think everyone publishing papers understand all the other papers, particularly in such a cross-disciplinary field like ML. There's also a big gap between a researcher deep in their career and a student fresh out of a masters program. It's also hard to transition from someone who's used to reading and understanding textbooks to someone who's often reading really technical research and understanding very little of it at first. You just have to push through and have confidence that you'll eventually learn enough to make a contribution. That's what it means to "become an expert"--you start off as not being an expert and then beat your head against the wall for a few years until you bootstrap your way out of it. And if you want to do it in a reasonable amount of time, you should probably choose something you have some of the fundamentals for.

tabtab 9y ago

Perhaps you can focus on improving the usability of AI tool-sets to a wider market rather than focus on finding the Next Big Magic Equation. Example: https://github.com/RowColz/AI https://github.com/RowColz/AI An AI expert(s) may do the initial setup, but factor tables allow more "typical" office workers to tune and prune the results.

togelius 9y ago

I am an AI researcher and faculty member at a large and famous university. I probably know less math than that Reddit poster. Math is important if you are specifically interested in the math of AI. If you are interested in inventing algorithms and solutions you mostly don't need the math.