9 ms·
Ilya Sutskever: “If you learn all of these, you’ll know 90% of what matters”
- Woodi 2y agoOn the related subject: any way to filter out anything AI/LLM related from HN ?
- isaacfung 2y agoWrite a chrome extension that prompts a LLM/uses a custom trained (BERT based) text classification model to filter the posts. https://github.com/thomasj02/AiFilter https://github.com/thomasj02/AiFilter https://github.com/GoogleChrome/chrome-extensions-samples https://github.com/GoogleChrome/chrome-extensions-samples https://huggingface.co/docs/transformers/en/tasks/sequence_classification https://huggingface.co/docs/transformers/en/tasks/sequence_c...
- Woodi 2y agoO, or maybe plugin with regexes will do for 90% use cases ? ;)
- KolenCh 2y agoI created a Zotero group: https://www.zotero.org/groups/5541371/ilya_sutskever/library/ https://www.zotero.org/groups/5541371/ilya_sutskever/library... This should be useful if you use Zotero. For pre-prints that have been published, they are replaced by the published version. Except for a few blog posts, and the Stanford course, and a copyrighted book, you should be able to obtained the PDF through Zotero automatically. Interesting facts: published date ranges from 1993 (the only item from the 90s, by his advisor) to 2020, types varies from 1 course, 1 dissertation, 10 preprints, (1 is a preprint of a chapter of a book, some are presented somewhere), 11 conference papers, 1 book, to 3 blog posts.
- rrr_oh_man 2y agoSource?
- wartijn_ 2y agoI don’t think John Carmack will gain much knowledge by studying a game engine.
- pmayrgundter 2y agoIs this product spam? List looks legit, but clicking on one launched some popover that disappeared and then no paper
- latexr 2y agoThis is just someone’s bookmarks folder. Where’s the proof these were recommended by Ilya? This looks like a subtle way to advertise a VC funded web browser.
- Tenoke 2y agoCarmack has more or less confirmed it. https://twitter.com/ID_AA_Carmack/status/1622673143469858816 https://twitter.com/ID_AA_Carmack/status/1622673143469858816
- dennis_moore 2y ago> A canonical list of references from a leading figure would be appreciated by many. That confirms the opposite, no?
- latexr 2y agoThat confirms the existence of a list, there’s zero indication it’s this list.
- Tenoke 2y agoHe posted it when this list was getting viral and acknowledged it without saying it's not the real one. I read it as a tacit confirmation at the time.
- notfried 2y agoApparently, this is the list Ilya gave to John Carmack more than a year ago [1], which was shared by an OpenAI employee 10 days ago [2]. [1] https://dallasinnovates.com/exclusive-qa-john-carmacks-different-path-to-artificial-general-intelligence/ https://dallasinnovates.com/exclusive-qa-john-carmacks-diffe... [2] https://twitter.com/keshavchan/status/1787861946173186062 https://twitter.com/keshavchan/status/1787861946173186062
- aragonite 2y agoSupposedly someone who used to work at openai found a partial version of the list from onboarding: https://twitter.com/andrew_n_carr/status/1752526711311507526 https://twitter.com/andrew_n_carr/status/1752526711311507526
- dennis_moore 2y agoI failed to find any evidence that this person is an OpenAI employee, but please correct me if I'm wrong.
- is_true 2y agoa backup just in case https://nlp.seas.harvard.edu/annotated-transformer/ https://nlp.seas.harvard.edu/annotated-transformer/ https://scottaaronson.blog/?p=762 https://scottaaronson.blog/?p=762 https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://arxiv.org/pdf/1409.2329.pdf https://arxiv.org/pdf/1409.2329.pdf https://www.cs.toronto.edu/~hinton/absps/colt93.pdf https://www.cs.toronto.edu/~hinton/absps/colt93.pdf https://arxiv.org/pdf/1506.03134.pdf https://arxiv.org/pdf/1506.03134.pdf https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf https://proceedings.neurips.cc/paper_files/paper/2012/file/c... https://arxiv.org/pdf/1511.06391.pdf https://arxiv.org/pdf/1511.06391.pdf https://arxiv.org/pdf/1811.06965.pdf https://arxiv.org/pdf/1811.06965.pdf https://arxiv.org/pdf/1512.03385.pdf https://arxiv.org/pdf/1512.03385.pdf https://arxiv.org/pdf/1511.07122.pdf https://arxiv.org/pdf/1511.07122.pdf https://arxiv.org/pdf/1704.01212.pdf https://arxiv.org/pdf/1704.01212.pdf https://arxiv.org/pdf/1706.03762.pdf https://arxiv.org/pdf/1706.03762.pdf https://arxiv.org/pdf/1409.0473.pdf https://arxiv.org/pdf/1409.0473.pdf https://arxiv.org/pdf/1603.05027.pdf https://arxiv.org/pdf/1603.05027.pdf https://arxiv.org/pdf/1706.01427.pdf https://arxiv.org/pdf/1706.01427.pdf https://arxiv.org/pdf/1611.02731.pdf https://arxiv.org/pdf/1611.02731.pdf https://arxiv.org/pdf/1806.01822.pdf https://arxiv.org/pdf/1806.01822.pdf https://arxiv.org/pdf/1405.6903.pdf https://arxiv.org/pdf/1405.6903.pdf https://arxiv.org/pdf/1410.5401.pdf https://arxiv.org/pdf/1410.5401.pdf https://arxiv.org/pdf/1512.02595.pdf https://arxiv.org/pdf/1512.02595.pdf https://arxiv.org/pdf/2001.08361.pdf https://arxiv.org/pdf/2001.08361.pdf https://arxiv.org/pdf/math/0406077.pdf https://arxiv.org/pdf/math/0406077.pdf https://www.vetta.org/documents/Machine_Super_Intelligence.pdf https://www.vetta.org/documents/Machine_Super_Intelligence.p... https://www.lirmm.fr/~ashen/kolmbook-eng-scan.pdf https://www.lirmm.fr/~ashen/kolmbook-eng-scan.pdf https://cs231n.github.io/ https://cs231n.github.io/
- snowfield 2y agoJust learn 30 papers
- r0ckarong 2y agoOr keep doom scrolling some other shit. Your choice. At least this is relevant information for people who are actually interested.
- password54321 2y agoYou mean for those who will just add it into their bookmark pile. You could have started yesterday and built a NN from scratch if you were truly interested.
- Ringz 2y agoReading 30 studies does not mean "learned and understood". Especially if you start from zero.
- beepboopboop 2y agoDisagree, I’ve already changed my title to “ML professional” on linked in. Aside from the odd hallucination, I’m indistinguishable from an expert to the average hiring team.
- seanhunter 2y agoYou have to use "Certified ML professional" or noone is going to take you seriously.
- rawgabbit 2y agoI changed my title to “OpenAI Certified Professional” and 10 recruiters have already reached out!
- barrenko 2y agoPeople will conveniently disregard the fact that it's meant for Carmack, so not an aspiring coder.
- taraparo 2y agoWould be nice if someone put those into an llm and generate a single coherent document with all the essential knowledge.
- CamperBob2 2y agoThat's... pretty much what an LLM is. Dive in, and when you get stuck, ask questions in Claude 3 or ChatGPT4.
- mkl 2y agoThere are a few past discussions, going back to October 2022. I wonder how out of date the list is now. Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack? https://news.ycombinator.com/item?id=34641359 https://news.ycombinator.com/item?id=34641359 396 points by alan-stark on Feb 3, 2023 | 137 comments -- John Carmack’s ‘Different Path’ to Artificial General Intelligence (dallasinnovates.com) https://news.ycombinator.com/item?id=34637650 https://news.ycombinator.com/item?id=34637650 378 points by cmdr2 on Feb 3, 2023 | 495 comments -- Ask HN: Where can I find an updated AI reading list? https://news.ycombinator.com/item?id=33166948 https://news.ycombinator.com/item?id=33166948 3 points by costant on Oct 11, 2022 | 6 comments
- klunger 2y agoThe headline belies the scope of this task. If you click in to see what these are: - one is a full course in convolutional neural networks https://cs231n.github.io/ https://cs231n.github.io/ - one is a 500 page text book https://www.lirmm.fr/~ashen/kolmbook-eng-scan.pdf https://www.lirmm.fr/~ashen/kolmbook-eng-scan.pdf - another one is an 80 page text book (?) https://arxiv.org/pdf/math/0406077 https://arxiv.org/pdf/math/0406077 and so on. I would guess it is possible to go through this list if you make it your full time job for a year or so. That might be worth it depending on your priorities. Regardless, it bears mentioning.
- admissionsguy 2y ago> If you make it your full time job for a year or so Sounds like a good MSc program
- sheepdestroyer 2y agoReportedly, Carmack did it an a week by locking himself in an hotel room
- jb1991 2y agoAh, the freedoms of not having any other responsibilities.
- soulofmischief 2y agoIf you can't take a single week off of work during an entire year in order to advance your career, then you're satisfied with your current one. There is no need to direct negativity toward others for making time to learn. Carmack is in this position today because of such work ethic, not in spite of it.
- brayhite 2y agoWhat a strangely narrow way of looking at this. Work isn’t the only responsibility. What is the correlation between taking a week away from responsibilities - work, family, etc. - and job satisfaction?
- throwaway71271 2y agoSome time ago I also copied the list at https://punkx.org/jackdoe/30.html https://punkx.org/jackdoe/30.html so its easier to wget (e.g. get all the arxiv pdfs) wget -A .pdf -r -H -D arxiv.org -nd https://punkx.org/jackdoe/30.html A friend and me used pdfunite to join them in one pdf and then printed them in a local printing shop in good a4 paper https://www.printenbind.nl/ https://www.printenbind.nl/ as a spiral book, the end result is very good (also there is space around the pages so that its easy to take notes). Keep in mind there is no confirmation that this is the actual list.
- deleted 2y ago[deleted]
- rossant 2y agoHow many pages is the spiral book?
- throwaway71271 2y ago360 pages, but also added the xLSTM paper https://arxiv.org/pdf/2405.04517 https://arxiv.org/pdf/2405.04517
- msoad 2y agoxLSTM is really really hard to grasp. It's mind boggling how they came up with such a unique architecture. I read the paper multiple times and still not sure if I can say I understood it.
- qwertyforce 2y agoLooks fake
- svet_0 2y ago[flagged]
- ausbah 2y agois there nothing here about attention?
- js8 2y agoThe argument that this is 90% of what matters in ML seems a bit bold. AFAICT it is completely missing reinforcement learning, which has been source of some mindblowing results (from Deepmind) in the past decade. It is also missing other stuff that I as a layman found fascinating (graph neural networks, 1.5 bit networks, ..). Also, I am not sure how much the Kolmogorov complexity actually matters, it's more philosophical than practical. So beware, it is a very opinionated list.
- PartiallyTyped 2y ago> AFAICT it is completely missing reinforcement learning There's no RL in there, nor diffusion. There are many other very interesting papers that I think have lots more to offer — but I just have a master's degree here. I am not a phd let alone a distinguished researcher.
- pmelendez 2y ago>The argument that this is 90% of what matters in ML seems a bit bold. AFAICT it is completely missing reinforcement learning, Wouldn't that account for the other 10%? To me it sounds like the quote is not too far off when you consider that he said "of what matters today". There are many other things missing from that list that might become more relevant tomorrow.
- smeej 2y agoHow long does it take to learn the prerequisites to learn these if you don't already have a CS background? Reading through the list, I can barely parse half the titles due to acronyms and terms I've never seen before. Would I be able to figure them out along the way? Or is there a body of knowledge this is assuming I should already have? (This isn't a criticism of the list, or even the idea of presenting such a list without explaining who the target audience is and isn't. I'm genuinely wondering, if I just start at the top, is there any hope I would figure this out, or do I need to start somewhere else entirely?)
- cwillu 2y agohttps://www.scottaaronson.com/democritus/ https://www.scottaaronson.com/democritus/ (dead tree form: https://www.amazon.ca/Quantum-Computing-since-Democritus-Aaronson/dp/0521199565 https://www.amazon.ca/Quantum-Computing-since-Democritus-Aar...) is a soft'ish introduction to much useful background.
- smeej 2y agoI think I may just need to accept that the work it would take to get to this level would mean a whole life change. It's been 20+ years since I took calculus in high school, and I haven't taken a math or science class since. I reckon that puts me at least five years behind where I'd need to be to start this course and the listed ones. Thanks for the recommendation, and helping me assess whether it's the use I want to make of my time!
- jononor 2y agoDepends on what you mean by learn... And what other relevant background one has outside of CS - programming/math/stats/systems/engineering. Assuming one has a few years of that, and one want to get to the level of being able to apply this to a problem that comes up in a realistic setting (making a product etc). Maybe extreme dedication over many months, or a few years at 20 hours a week? This is assuming that one either is very good at studying alone, or have access to people one can ask alone the way. If one has no relevant background, then I would 5+ years of full time study.
- 2y ago
- _giorgio_ 2y agoI've seen this list many times. It's years old already.
- gessha 2y agoWhat people forget is without a proper, tangible goal, no matter how many core or interesting papers you read, you will still remember nothing. At most you will be able to make armchair ML researcher comments on an online forum.
- GaryNumanVevo 2y agoI have a career in ML research and I use most of my knowledge to be an armchair ML researcher!
- deleted 2y ago[deleted]
- digitalsushi 2y agooh man my cognitive dissonance is crazy cause i am easily in the top 50% percentile for being able to use popular LLMs but that reading list is outside my comprehension abilities. I cannot synthesize a thing I use
- drivebyhooting 2y agoDo you use pencils? I doubt you can synthesize that.
- barfbagginus 2y agoThis has nothing on it about llms, where in context learning, retrieval augmention, tool using, and multi modality is really blazing ground. Ideally we'd have papers about these methods, because we need these tools to implement stuff, and they're changing how we do business. Without in context learning, self critique and tool using, for example, it's easy to think that LLMs are stochastic parrots bound by their training distribution, instead of last mile learners that can adapt to out of distribution contexts. And business applications need RAG to overcome hallucinations. Without these concepts, I don't think people will really know 90% of what matters today, or be ready to take on what matters tomorrow. I could be wrong here, with my focus on so many llm specific topics. Things like in context learning and RAG exist as engineering workarounds for LLM limitations. They are not definite paths to tomorrow's AI, which won't need our cajoling and hand holding. There's a possibility that we'll move past them, so more speculative and theoretical papers are better. But I think there's also a possibility that the AI scaling is done for this AI bubble, and we're stuck with building larger scale systems out of imperfect buggy LLMs for the foreseeable future. In that case, maybe some next generation Zen of integrating LLMs into a working mind eventually emerges as a way forward.. so we should pay attention to engineering efforts around LLMs!
- alabhyajindal 2y agoWhat has John Carmack produced after reading these?
- navs 2y agoGreat way to advertise Arc.