4 ms·
Can someone help me understand what I’m looking at here? This repository allows me to train a specific model on a specific data set, and finally test the result
by c0wb0yc0d3r 2y ago
Can someone help me understand what I’m looking at here? This repository allows me to train a specific model on a specific data set, and finally test the result? Is that correct?
I am interested in how large and small language models are trained, but as someone who has little knowledge in this world I find it hard to cut through the noise to find useful information.
Really I’m looking for an open source project that helps a person gain this knowledge. Something like a docker container that encapsulates all the dependencies. When training it will use any available gpu or tell me why my gpu can’t be used and then fall back to cpu. Then had a simple interface to test the training results. Finally you can easily pull back the curtain to understand the process in better detail and maybe even adapt it to different model to experiment.
Does something like that exist?
- timnetworks 2y agoAs opposed to inference (like generating text and images), training requires some more math (fp16 or bf16) and a single CPU generally won't cut it. The prepare/train/generate instructions in the github linked are pretty much it for the 'how' of training a model. You give it a task and it does it for 1 billion trillion epochs and saves the changes incrementally (or not). Training a LoRA for an image model may be more approachable, there's more blog entries etc on this, and the process is largely similar, except you're doing it for a single slice instead of the whole network. [edit] I'm also learning so correct me if I'm off, hn!
- sva_ 2y ago> You give it a task and it does it for 1 billion trillion epochs and saves the changes incrementally (or not). Somewhat confusingly, big LLM are most just trained for 1 epoch afaik.
- SJC_Hacker 2y agoDo you have a good theoretical foundation in ML ? You will also need some linear algebra. If not would invest the time in a decent course, there are plenty online, even offline if you are close enough to where its offered. I took one from Andrew Ng on Coursera years ago, which used Matlab. There are probably much better, more up-to-date options now, especially now that LLMs are very in-vogue. The fundamentals such as gradient descent, ANNs and back-propagation however, is still relevant, and hasn't changed much. Trying to understand what code is doing without that foundation will be an exercise in futility.
- c0wb0yc0d3r 2y agoI don’t have a solid ML foundation, and it’s been a decade or more since I’ve worked with linear algebra. For now I think that might be too deep for what I’m after. I’m at the beach looking out at the vast ocean that is machine learning and LLMs.
- barrenko 2y agoYou're probably having the right hunch, it takes a crapload of time, especially if you want to implement and not just "get an intuition".
- MacTea 2y agohttps://course.fast.ai/ https://course.fast.ai/ is the best. From their site: " A free course designed for people with some coding experience, who want to learn how to apply deep learning and machine learning to practical problems. "
- c0wb0yc0d3r 2y agoThis is at the top of my lunch time learning list. Not quite what I’ve been envisioning but it’s in the right direction. Thanks!
- TheTaytay 2y agoIf you are looking for something that actually explains most concepts behind it, Karpathy’s series will teach you what you want to know. (Mentioned elsewhere) If you are looking for command line tools to fine tune and evaluate models on known datasets, this article is a good take! https://www.philschmid.de/fine-tune-llms-in-2025 https://www.philschmid.de/fine-tune-llms-in-2025