4 ms·

Completely agreed. The point is the knowledge, the learning and the journey. If a kid has a passion for building or toying with LLMs, then of course, by all mea

by kalms 1mo ago

Completely agreed. The point is the knowledge, the learning and the journey. If a kid has a passion for building or toying with LLMs, then of course, by all means, please start tearing them apart or even build and train your own model. You'll learn a ton, even if you won't necessarily end up using it here and now. The learning experience will compound and of course that will be useful. The above is, after all, the whole genesis of the word 'hacker'. We should celebrate that.

rdedev 1mo ago
How exactly does one go about "tinkering" with an LLM? Any architectural change you introduce needs fine tuning. That needs data and compute I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasonable time with that much compute.
- alexgartrell 1mo ago
  https://thinkingmachines.ai/tinker/ https://thinkingmachines.ai/tinker/ https://github.com/thinking-machines-lab/tinker-cookbook https://github.com/thinking-machines-lab/tinker-cookbook (As a TML person, I'm obviously biased, but I couldn't resist because of "tinkering"). TBF it's hard to imagine a real architecture change that wouldn't require a ton of compute, but you could certainly fine tune and play with different recipes, loss functions, etc. And Claude can carry you a lot of the way through doing this. One fun task is to invent a tool and then train a small model to use it. You could export that small model and run it locally for free forever to do your thing. I think this is what a lot of Software Engineering will look like later. There are a lot of other high level abstractions here to look at. Prime Intellect has one. The other thing to play with is self-hosting small models, but IMO most of the interesting stuff is actually related to multi-gpu or multi-node inference so there's not necessarily a ton to learn here.
- danielmarkbruce 1mo ago
  You can build a model with 100m parameters..... the first L in LLM is not meaningful anymore. So, you build one from scratch.
- musebox35 1mo ago
  This is one question that is not just for kids but for all "gpu poor" adults as well. One concept that I found useful is "toy models": small compute models that replicate a certain, not very well explained phenomenon that the large models have. Of course, you can't have a toy model for everything, some behaviors are only visible at scale. But, there is a bunch of things like in-context learning that you can play with without burning a ton of compute. The best analogy is strip mining (big labs) vs cave exploration (solitary/small teams). I think this is how science progresses at the boundaries by smart/curious/hardworking individuals because depth is a requisite for finding the right questions and then the answer. It is not for everyone and it does not always work. But you learn a ton even if it doesn't pan out to be a big breakthrough.
- canyon289 1mo ago
  A lot of different ways. Every guide on this guidebook runs on a ~$500 gpu or a couple of dollars on a cloud instance with a bigger GPU https://ravinkumar.com/GenAiGuidebook/book_intro.html https://ravinkumar.com/GenAiGuidebook/book_intro.html This guidebook covers pretraining, post training (SFT, RL) and a couple other topics. And others authors have also written books that fit on single node reasonable hardware. If you want to start with a pretrained base I built Gemma 270m and released it last year. This fits on a raspberry pi. https://developers.googleblog.com/en/introducing-gemma-3-270m/ https://developers.googleblog.com/en/introducing-gemma-3-270... The fundamentals of AI don't require industrial amounts of large scale. Think of it like this, when I was learning how a plane worked when I was a kid I didn't build a 747 at home, I started with scale sized model planes. Same idea here. And FWIW I'm a staff researcher at Deepmind (opinions here are my own) so I want to specifically encourage all people out there, you can learn a lot about how these LLMs work at home, for (mostly free), using resources like colabs or spot pricing on accelerator providers. There's many great resources out there and I encourage anyone willing to learn to go for it!

by kalms 1mo ago

rdedev 1mo ago

How exactly does one go about "tinkering" with an LLM? Any architectural change you introduce needs fine tuning. That needs data and compute I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasonable time with that much compute.

alexgartrell 1mo ago
https://thinkingmachines.ai/tinker/ https://thinkingmachines.ai/tinker/ https://github.com/thinking-machines-lab/tinker-cookbook https://github.com/thinking-machines-lab/tinker-cookbook (As a TML person, I'm obviously biased, but I couldn't resist because of "tinkering"). TBF it's hard to imagine a real architecture change that wouldn't require a ton of compute, but you could certainly fine tune and play with different recipes, loss functions, etc. And Claude can carry you a lot of the way through doing this. One fun task is to invent a tool and then train a small model to use it. You could export that small model and run it locally for free forever to do your thing. I think this is what a lot of Software Engineering will look like later. There are a lot of other high level abstractions here to look at. Prime Intellect has one. The other thing to play with is self-hosting small models, but IMO most of the interesting stuff is actually related to multi-gpu or multi-node inference so there's not necessarily a ton to learn here.
danielmarkbruce 1mo ago
You can build a model with 100m parameters..... the first L in LLM is not meaningful anymore. So, you build one from scratch.
musebox35 1mo ago
This is one question that is not just for kids but for all "gpu poor" adults as well. One concept that I found useful is "toy models": small compute models that replicate a certain, not very well explained phenomenon that the large models have. Of course, you can't have a toy model for everything, some behaviors are only visible at scale. But, there is a bunch of things like in-context learning that you can play with without burning a ton of compute. The best analogy is strip mining (big labs) vs cave exploration (solitary/small teams). I think this is how science progresses at the boundaries by smart/curious/hardworking individuals because depth is a requisite for finding the right questions and then the answer. It is not for everyone and it does not always work. But you learn a ton even if it doesn't pan out to be a big breakthrough.
canyon289 1mo ago
A lot of different ways. Every guide on this guidebook runs on a ~$500 gpu or a couple of dollars on a cloud instance with a bigger GPU https://ravinkumar.com/GenAiGuidebook/book_intro.html https://ravinkumar.com/GenAiGuidebook/book_intro.html This guidebook covers pretraining, post training (SFT, RL) and a couple other topics. And others authors have also written books that fit on single node reasonable hardware. If you want to start with a pretrained base I built Gemma 270m and released it last year. This fits on a raspberry pi. https://developers.googleblog.com/en/introducing-gemma-3-270m/ https://developers.googleblog.com/en/introducing-gemma-3-270... The fundamentals of AI don't require industrial amounts of large scale. Think of it like this, when I was learning how a plane worked when I was a kid I didn't build a 747 at home, I started with scale sized model planes. Same idea here. And FWIW I'm a staff researcher at Deepmind (opinions here are my own) so I want to specifically encourage all people out there, you can learn a lot about how these LLMs work at home, for (mostly free), using resources like colabs or spot pricing on accelerator providers. There's many great resources out there and I encourage anyone willing to learn to go for it!