6 ms·
TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
- dzogchen 2y agoFor a model to be ‘fully’ open source you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it. See The Open Source AI Definition from OSI: https://opensource.org/ai https://opensource.org/ai
- szundi 2y agoYes. I cannot comprehend this to this day. A model weights data + runner is how different from a closed source executable? Why do everyone call these open source?
- fhd2 2y agoBecause practically speaking, you can fine tune them I suppose? But that's also true for binaries, games are a good example of where people pushed this quite far. Based on what little experience I have in ML, I'd say it's about the same thing. Whereas an API is more akin to a piece of software you can't tinker with in any way. Guess the bar is just lower in the LLM space :P
- darkwater 2y agoSo a proprietary program with a lot of knobs and configuration files is kind of opensource?
- fhd2 2y agoBy what appears to be the logic for "open source AI", a locally executable proprietary program would be "open source" (because you can meddle with the executable). To me, that's mostly just "not SaaS". But somehow, a different definition appears to have stuck for LLMs than for other types of software.
- PeterStuer 2y agoBecause typically adapting or improving traditional code to your needs is very difficult without access to the source code and build files. For an LLM you can finetune and enhance, distill and embed given just the model weights, the runtime, and a permissive license. Having more is better. Well written detailed model release papers help a lot. Training code and training data are a great bonus. However, I find the purity contest a bit too dismissive of the great contributions to the AI dev ecosystem that Meta and Deepseek have brought us. Without these, there wouldn't be the open ecosystem we have today.
- GeneralMayhem 2y agoIt's not a purity contest, it's a clarity contest. If Meta and Deepseek want to operate the way they have been, where they release baked models and whitepapers, that's fine - and you're right, it's certainly more than they're obligated to release. They just shouldn't be calling it "open source" when the source is literally not open.
- doix 2y agoEh, I can kinda see it. It depends on your definitions of words. People have been muddying the waters with what "open source" means anyway. I have known it to mean code released under an open source license. Other people use it to mean programs where the source is available regardless of license. I would use "source available" to describe that, but some people strongly disagree with my definitions. If I write a program, then obfuscate it and then release the obfuscated code under an open source license, would you consider it open source(I would)? That's kind of the case here, they are releasing the model weights under an open source license. Personally, I think it's fine to shorten it to "open source model" instead of "a model with the weights released under an open source license". What I would object to is releasing model weights under a restrictive license and calling that open source.
- GeneralMayhem 2y ago> If I write a program, then obfuscate it and then release the obfuscated code under an open source license, would you consider it open source(I would)? I wouldn't. Most definitions of open source say something like "in the form used for editing". You can release a built binary under an unrestrictive license, but that does not mean that you've opened the source. It's literally the plain meaning of the words: the source, as in where the thing comes from, needs to be open for it be meaningful.
- sigmoid10 2y agoOpen source becomes really complicated once assets with unclear license are involved in any way. Lots of people for example would say that Jedi Knight 2 is open source because Raven Software released the source code and tools needed to build the game. But that alone doesn't mean you can run it, because you still need to get a hold of all the assets (models, textures, sounds) which may or may not still be property of LucasArts or its successors. Even if you have them, it's actually unclear if it is legal to use them this way. So while there are tons of people working on mods and conversions, noone in their right mind would distribute all the source assets. Much in the same way, no sane company will touch the legal nightmare of releasing LLM training data scraped from public websites. Even releasing the LLM alone might be infringement, there are literally court cases being fought over this right now.
- GeneralMayhem 2y agoGames like that, or the open-source clones of commercial games that require original assets to play (e.g. OpenXCOM), actually give a very clear analogy here: open source does not mean open assets. The software code is under a separate license from the data it processes. Emulators like Dolphin are kind of in this situation too - the program is open, the data it processes is not. And that's fine! It's still valuable to have access to the source code, even if the "batteries" aren't included. Of course, if you really want to call it an open source model you should include the source for the data scraping/cleaning stages too; then the only thing missing would be the compute time and risk of acquiring dubiously-legal inputs. I personally prefer a taxonomy like: * Open weights: you can download the artifact and run it locally, not just use it through an application like chatgpt or an API. * Open source: the code that created the artifact is provided in the same format that the authors used to work on it. * Open data: the dataset that the source code was used on is available for download. All three of those could be individually licensed or released, for 8 possible combinations. In the analogy to games, they would correspond to the licenses on the retail binary, the source code of the game, and the original uncompressed art assets or Blender projects, respectively.
- sigmoid10 2y agoIf it has already been established that open source doesn't mean open assets, why would we change that now? After all, training data is literally nothing but assets - except that you don't need them to run the application. So in that sense open LLMs are more open than these games.
- benterix 2y agoBecause Meta called them in this way to differentiate themselves from "Open"AI and everyone else followed the suit.
- deleted 2y ago[deleted]
- moffkalast 2y agoBecause marketing (open source is a buzzword after all), and the media just repeats what they read in press releases verbatim. But most people working with the models themselves call them open-weight, except for some occasional exception like OLMo that publishes the dataset and training scripts and is actually open source.
- dartos 2y agoBecause Meta called llama that shortly after it got leaked and it stuck. The AI crowd doesn’t care much for licenses anyway.
- rettichschnidi 2y agoSadly, the OSAID also does not require training data to be available. :(
- a2128 2y agoIs it reasonable to expect companies to redistribute 100TB of copyrighted content they used for their LLM, just on the off-chance someone has a few million laying around and wants to reproduce the model from scratch?
- anotherhue 2y agoThe RL-only (no SFT) approaches might remove that issue. Problem sets should be smaller (and mechanically creatable) than the entire western corpus.
- puapuapuq 2y agoFor open source model claiming SOTA performance, we could at least check for data leak from its training data.
- emseetech 2y agoWould a reference file with filename, size, source and checksum count towards the OSI definition?
- atq2119 2y agoPerhaps that's not reasonable to expect, but Meta apparently kind of did it anyway, if not in a way that helps reproduce their LLM: https://arstechnica.com/tech-policy/2025/02/meta-torrented-over-81-7tb-of-pirated-books-to-train-ai-authors-say/ https://arstechnica.com/tech-policy/2025/02/meta-torrented-o...
- moffkalast 2y agoActually they did, the entire 15T tokens that were supposedly used for training the llama-3 base models are up on HF as a dataset: https://huggingface.co/datasets/HuggingFaceFW/fineweb https://huggingface.co/datasets/HuggingFaceFW/fineweb It's just not literally labelled so because of obvious reasons.
- TeMPOraL 2y agoRedistribute? No. Itemize and link to? Yes. With LLMs, the list doesn't even have to be kept up to date, nor the links alive (though publishing content hashes would go a long way here). It's not like you can get an identical copy of a model built anyway, there's too much randomness at every stage in the process. But, as long as the details of cleanup and training are also open, a list of training material used would suffice - people would fetch parts of it, substitute other parts with equivalents that are open/unlicensed/available, add new sources of their own, and the resulting model should have similar characteristics to the OG one who we could, now, call "open source".
- HumanOstrich 2y agoThat is incorrect -- you do not have to provide the full training data to meet the requirements. I recommend reading the actual Open Source AI Definition[1] and the FAQ[2]. There's also the whitepaper[3] that goes into much more detail about the state of affairs. [1]: https://opensource.org/ai/open-source-ai-definition https://opensource.org/ai/open-source-ai-definition [2]: https://hackmd.io/@opensourceinitiative/osaid-faq#What-is-the-role-of-training-data-in-the-Open-Source-AI-Definition https://hackmd.io/@opensourceinitiative/osaid-faq#What-is-th... [3]: https://opensource.org/wp-content/uploads/2025/02/2025-OSI-DataGovernanceOSAI-final-v5.pdf https://opensource.org/wp-content/uploads/2025/02/2025-OSI-D...
- amelius 2y agoWe need a new definition then.
- deleted 2y ago[deleted]
- Salgat 2y agoFYI their open source ai definition released with a lot of controversy, unsurprisingly because it had heavy contribution from corporations with their own interests. It's best to ignore it for now until the wider community has decided on an appropriate open source definition.
- deleted 2y ago[deleted]
- 2-3-7-43-1807 2y ago> you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it. the model reveals the architecture which is all you need to use/run/train it.
- anfal 2y agoFully agreed, someone actually mentioned that to me on reddit and I modified the content + added a disclaimer on top of the article. https://www.reddit.com/r/LocalLLaMA/comments/1ilsfb1/comment/mc2f28e/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button https://www.reddit.com/r/LocalLLaMA/comments/1ilsfb1/comment...
- bluelightning2k 2y agoGreat write up of what is presumably a truly great lecture. Debating trying to follow the original now.
- albert_e 2y agoOT: What is a good place to discuss the original video -- once it has dropped out of the HN front-page? I am going through the video myself -- roughly halfway through -- and have a fw things to bring up. Here they are now that we have a fresh opportunity to discuss: 1 - MATH and LLMs I am curious why many of the examples Andrej chose to pose to the LLM were "computational" questions -- for instance "what is 2+2" or some numerical puzzles that needed algebraic thinking and then some addition/subtraction/multiplication (example at 1:50 mins about buying Apples and Oranges). I can understand these abilities of LLMs are becoming powerful and useful too -- but in my mind these are not the "basic" abilities of a next token predictor. I would have appreciated a more clear distinction of prompts that showcase core LLM ability -- to generate text that is acceptable as generally grammatically correct, based in facts and context, without necessarily needing the ability of a working memory / assigning values to algebraic variables / doing arithmetic etc. If there are any good references to discussion on the mathematical abilities of LLMs and the wisdom of trying to make them do math -- versus simply recognizing when a math is needed and generating the necessary python/expressions and let the tools handle it. 2 - META While Andrej briefly acknowledges the "meta" situation where LLMs are being used to create training data for the training of and judge the outputs of newer LLMs ... there is not much discussion on that here. There are just many more examples of how LLMs are used to prepare mitigations for hallucinations by preparing Q&A training sets with "correct" answers etc I am curious to know more about the limitations / perils of using LLMs to train/evaluate other LLMs. I kind of feel that this is a bit like the Manhattan project and atomic weapons -- in that early results and advances are being looped back immediately into the development of more powerful technology. (A smaller fission charge at the core of a larger fusion weapon -- to be very loose with analogies) <I am sure I will have a few more questions as I go through the rets of the video and digest it>
- khazhoux 2y agoI'm still seeking an answer to what DeepSeek really is, especially in the context of their $5M versus ChatGPT's >$1B (source: internet). What did they do versus not do?
- uncomplexity_ 2y agoThis one's for you, from Diana Hu (GP @ YC) https://x.com/sdianahu/status/1887208144025292975 https://x.com/sdianahu/status/1887208144025292975
- fredoliveira 2y agoThere's a great deal of coverage about DeepSeek on Zvi's newsletter here: https://thezvi.substack.com/archive?sort=new https://thezvi.substack.com/archive?sort=new - the first post on R1 is from January 22
- anfal 2y agoHey, Anfal here. I am actually the author of this article. I have had some really good discussion with a few really intelligent fellows I know and they tie very deeply into deepseek more. I'll create a post about it soon.
- est 2y agoI have read many articles about LLMs, and understand how it works in general, but one thing always bothers me: why other models did't work as good as SOTA ones? What's the history and reason behind the current model architecture?
- amelius 2y agoI guess nobody really knows why. Everybody just goes with what works, and tries only small variations. It's a bit like alchemy.
- rob_c 2y agoSimply put no other model has the same number of effective skip connections or passes as much information through the model from input to output. Earlier models had huge bottlenecks in terms of information limits and precision. (Auto encoders Vs uNets for example) And LSTM are still semi unstable. Why the attention design as posited by Google works so well is part the skip forward and part "now we have enough information and processing power to try this". It's well motivated but from a first principles up do we expect this to work well, it's a bit less well understood still. And if you're good at that you'll likely get a job offer very quickly.
- rob_c 2y agoSimply put the stability of attention based models over non attention based ones. Google dropped MHA self attention which was a major idea that they showed to work. OpenAI saw and built an empire on FeedForward attention models which are (compared to most alternatives) super stable at generation. DeepSeek showed evidence it's possible to further push these models and use effectively compression in the model design to pass around sufficient information for training. (Hence the Latent) They also did a lot of other cool stuff, but the main "core of the model" difference is this part... Other than that, the biggest hurdle has been hardware. There's probably no way you could get kit from 2010 without even aes acceleration to evaluate most full-fat mhlffa models let alone train them. There's been a happy convergence of matrix acceleration on GPUs for gaming, graphical and high fidelity stimulation work. This combined with matrix based ML maths combined with high throughput memory advances means we can do what we're doing with llms now. So inevitable outcome or happy convergence? That's for historians to decide imo. I think it's a bit of both.
- deleted 2y ago[deleted]
- miletus 2y agoi saw a good thread today: https://x.com/0xmetaschool/status/1888873661840634111 https://x.com/0xmetaschool/status/1888873661840634111
- EncomLab 2y agoIt would be great if the hardware issues were discussed more - too little is made of the distinction between silicon substrate, fixed threshold, voltage moderated brittle networks of solid-state switches and protein substrate, variable threshold, chemically moderated plastic networks of biological switches. To be clear, neither possesses any magical "woo" outside of physics that gives one or the other some secret magical properties - but these are not arbitrary meaningless distinctions in the way they are often discussed.
- thomasahle 2y agoI find Meta’s approach to hallucinations delightfully counter intuitive. Basically they (and presumably OpenAI and others): - Extract a snippet of training data. - Generate a factual question about it using Llama 3. - Have Llama 3 generate an answer. - Score the response against the original data. - If incorrect, train the model to recognize and refuse incorrect responses. In a way this is obvious in hindsight, but it goes against ML engineers natural tendency when detecting a wrong answer: Teaching the model the right answer. Instead of teaching the model to recognize what it doesn't know, why not teach it using those same examples? Of course the idea is to "connect the unused uncertainty neuron", which makes sense for out-of-context generalization. But we can at least appreciate why this wasn't an obvious thing to do for generation 1 LLMs.
- implmntatio 2y ago> but it goes against ML engineers natural tendency when detecting a wrong answer: Teaching the model the right answer. Hard to buy. If a machine makes a mistake, it's because it was configured wrong or because of wear and tear, solar flares or some quake or some manufacturing defect in a part. If a learning machine makes a mistake, it's because it's learning has not extended it's rule set to cover that matrix/mistake/pattern, yet; and so it includes that mistake/matrix and other mistakes, analyses for patterns and then creates mistakes that fall into that pattern. Later doing that in a rolling release or canine kind of way and even later learning machines will do it all live, synchronous to their concurrent actions. But yeah, thinking about that, I see why ML engineers wouldn't get there from scratch. It's a rhythm, after all, an epiphany about or realization of how ones dog, ones brain works, learned and then coded step by step. And there is, of course the variety of how people learn and "realize". Someone has to show us the work of those savant programmers/engineers I still haven't seen a documentary of.
- 2-3-7-43-1807 2y ago> In a way this is obvious in hindsight, but it goes against ML engineers natural tendency when detecting a wrong answer: Teaching the model the right answer. But the answer space for LLMs is infinite and unbounded. So, no effort will be complete and you will always end up with the question of how to deal with uncertainty. But I admit this is a bit of hindsight 20/20.
- sylware 2y agoIt is sad to see that much attention given to LLM in comparison to the other types of AIs like those doing maths (strapped to a formal solver), folding proteins, etc. We had a talk about those physics AIs using those maths AIs to design hard mathematical models to fit fundamental physics data.
- wolfhumble 2y agoI haven't watched the video, but was wondering about the Tokenization part from the TL;DR: "|" "View" "ing" "Single" Just looking at the text being tokenized in the linked article, it looked like (to me) that the text was: "I View", but the "I" is actually a pipe "|". From Step 3 in the link that @miletus posted in the Hacker News comment: https://x.com/0xmetaschool/status/1888873667624661455 https://x.com/0xmetaschool/status/1888873667624661455 the text that is being tokenized is: |Viewing Single (Post From) . . . The capitals used (View, Single) also makes more sense when seeing this part of the sentence.
- quantumspandex 2y agoAndrej's video is great but the explanation on the RL part is a bit vague to me. How exactly do we train on the right answers? Do we collect the reasoning traces and train on them like supervised learning or do we compute some scores and use them as a loss function ? Isn't the reward then very sparse? What if LLMs can't generate any right answers cause the problems are too hard? Also how can the training of LLMs be parallelized when updating parameters are sequential? Sure we can train on several samples simultaneously, but the parameter updates are with respect to the first step.
- mtkd 2y agoDetails on how DS used GRPO for RL rewards https://medium.com/@sahin.samia/the-math-behind-deepseek-a-deep-dive-into-group-relative-policy-optimization-grpo-8a75007491ba https://medium.com/@sahin.samia/the-math-behind-deepseek-a-d...
- quantumspandex 2y agoThanks!
- epr 2y agohttps://arxiv.org/abs/1707.06347 https://arxiv.org/abs/1707.06347
- quantumspandex 2y agoWill have a look. Thanks!
- fenomas 2y agoAs I understood that part, in RL for LLMs you take questions for which the model already sometimes emits correct answers, and then repeatedly infer while reinforcing the activations the model made during correct responses, which lets it evolve its own ways of more reliably reaching the right answer. (Hence the analogy to training AlphaGo, wherein you take a model that sometimes wins games, and then play a bunch of games while reinforcing the cases where it won, so that it evolves its own ways of winning more often.)
- 9999_points 2y agoIts a shame his LLC in C was just a launch board for his course.
- p0w3n3d 2y agoOn 53 minutes from the original video, he shows how exact is the quotation of an LLM based on the text it was learning from. I wonder how did the bigtech convince the courts that this is not copyright violation (especially when ChatGPT was quoting some GPL code). I can imagine that the same thing would happen opposite, if I trained a model to draw a disney character, and my ass would be sued in a fraction of a second.
- avbanks 2y agoThis is still being litigated I believe.
- sambull 2y agoIt's a interesting question, one I wonder now that our federal data is being exfiltrated to AI companies. If they train their models on the data; how does the law tell them to 'unlearn'? Destroying the copies they took will be what the courts ordered, but the data will still be there.
- fenomas 2y agoNote that he's inferring from a base model there, which are fairly capable of regurgitating their (highly-weighted) inputs since they do nothing but predict pre-training tokens. For instruct services like ChatGPT, if they regurgitate something I'd think it would more likely be their fine-tuning data, which is usually owned by the provider (and also kept secret).
- p0w3n3d 2y agowhat I mean is if we can describe an LLM as a lossy compression (which are words spoken by Andrej), we could define what was done during inferring as uncompressing the compressed data, and at this moment shit would hit the fan.
- deleted 2y ago[deleted]