5 ms·
LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
- Vineeth147 3mo ago[flagged]
- stephantul 3mo agoSadly 100% generated. I think the idea is interesting though, although I wonder if training time for LoRA is such a bottleneck to deserve its own, extremely narrowly scoped, leaderboard. Maybe if it was more tasks or more models we could hope that it transfers? With a single task, and a single model, I’d be afraid of this overfitting pretty heavily. For NanoGPT, I think the idea always was that the ideas can be transferred to much larger models, or serve as stepping stones for investigations on larger models.
- Vineeth147 3mo agoThat's fair on both points. Much of this was built with AI, but the runs and numbers are real. They are also reproducible, so I would prefer to be judged on that. And yes, using a single model and task can lead to overfitting. The plan is to add more tracks, including bigger models and other tasks, so a technique only matters if it transfers. Right now, it's just the initial track, so your concern is valid. Thanks for the feedback.
- Aerialoo 3mo agoAI generates or not - irrelevant if the substance is high quality. Always good to see projects. You should consider hosting this leaderboard on hugging face, you will get a lot more AI enthusiasts and practitioners there.
- Vineeth147 2mo ago[dead]
- Gisbitus 3mo agoI understand the feedback in the second paragraph, however I do not understand why we're judging projects by whether they've been AI generated or not. Have we stopped treating software as a black box? This behavior will only lead to devs moving away from OSS to avoid the AI stigma.
- stephantul 3mo agoNot the software, the whole thing. As an author: show me why you thought this was interesting and why you’re doing it, and why you think it’s relevant. What does it build towards? What does climbing this leaderboard mean to me? Absent those things, this is just some thing my opus could generate as well.
- Vineeth147 3mo agoI fine-tune small models with limited resources, but it's hard to tell which speedup claims are real. Each method, like DoRA, rsLoRA, Unsloth, or packing tricks, uses different models, data, and hardware in their reports. This makes it impossible to really check their claims in practice. The only way I've seen these debates settled is by using a fixed task and having someone act as a referee. That's what nanoGPT's speedrun did for optimizers. The goal is to make a public record that shows which training tricks really save time and which ones don't work when tested again. Each record should explain how the method works, so over time, the leaderboard becomes a kind of lab notebook. As more records are added, the leaderboard serves both as a ranking and as a detailed log of what was tried. I agree, and that's my next step: adding a second track with a different model family and task (SmolLM2 + SQuAD), so we can actually test if these methods transfer instead of just promising they do. Honestly, your opus could probably build the framework too. But a leaderboard is more than just code. It also depends on someone being willing to review submissions for cheating, make decisions when things aren't clear, and keep adding new records even months later. That part can't be automated; it needs to be maintained by people.
- stephantul 3mo agoNice! That sounds a lot more like a mission statement than your actual readme.
- jmward01 3mo agoI think there is real value in going smaller/limiting resources. The trend is 'just make the weights bigger and throw more data at it'. It is a MBA's view of winning. We have a knob, keep turning it. It does work but it may not drive as much creativity as resource limits can drive. It is like urban growth boundaries in city planning. If you aren't allowed to 'just expand' you are forced to build more intelligently inside the city and those creative solutions often lead to major improvements.
- curiouscube 3mo agoYou are right only in so far that it is more economical. But it is not the MBA's view of winning, it's just one potential conclusion you could draw from the bitter lesson of Machine Learning. As long as the need for more intelligence outpaces the economics of using intelligence, you'll get bigger models. This idea that small, fine-tuned models can outperform bigger models capabilities wise is mostly misinformed. They are genuinely good at other metrics, but sadly more actually means better in ML-land (most of the time at least).
- rsfern 3mo agoI don’t think there’s a fundamental reason that performance has to be monotonic in model size or even training FLOPs. At least I don’t think it’s been proved to be so, so I think “misinformed” is a bit premature and sort of makes GP’s point. There’s evidence that model size and representational capacity are not exactly the same, and that scale is maybe more important for learning than it is for representation (past a point). Consider the early work from the current neural scaling paradigm. The Chinchilla scaling study shows that smaller models can match the performance of larger models by training longer. To GP’s point, if everyone is exploiting the scaling lever, few resources are being allocated to finding more efficient training algorithms that could let us work with right-sized models instead of pulling the scaling lever as hard as we can afford to. I’ll end with a dramatic example from my field of materials science (which admittedly might not strictly generalize to LLMs). A lot of the field is pursuing the model scaling strategy, and it’s still paying off. But [0] recently reported competitive accuracy with much smaller models that run faster and can address much larger problems. The model architecture is pretty much the same, but they use a different training strategy and really focus on data quality 0: https://arxiv.org/abs/2504.21286 https://arxiv.org/abs/2504.21286
- monegator 3mo agoFeels like i'm missing the introduction paragraph in the repo. What is LoRA in this context? the communication protocol? Or another term appropriated by LLMs? Why the speedrun?
- gowthamgts12 3mo agoLow Rank Adaptation. Google “Lora llm”
- Schlagbohrer 3mo agoYes but what are LoRAs being tested for / trying to do in this case? I couldn't find any information there about what the actual output goal is for the LoRAs. Speed doing what?
- maleldil 3mo agoHow long it takes to train that particular configuration. The table column is named "train time".
- garimbaboy 3mo agoLoRa (Long Range) is a physical radio communication techniche.[1] LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique for large language models.[2] [1] https://en.wikipedia.org/wiki/LoRa https://en.wikipedia.org/wiki/LoRa [2] https://en.wikipedia.org/wiki/LoRA_(machine_learning) https://en.wikipedia.org/wiki/LoRA_(machine_learning)
- nomel 2mo agoThe author responds with reasoning here: https://news.ycombinator.com/item?id=48975473 https://news.ycombinator.com/item?id=48975473 The AI slop "why" section in the readme is completely useless. I struggle with this too. I start every project doc with a "why" then "overview" section, and AI is still horrendously bad at the "why", filled with vapid corpo/tech startup speak. I usually write the "why" myself, or at least put down the outline to be completed.
- lipenghui222 3mo ago[flagged]
- jkwang 3mo ago[flagged]
- Vineeth147 2mo ago[flagged]
- patrick0d 3mo agoInspired by parameter golf and speedrun approaches I make the case for picking loss functions like a wallclock for LoRA on AI safety targets. The result when I tried it was a functional distillation of an Sparse AutoEncoder into a 5.3MB probe. I have a technical writeup below about it if anyone is interested. https://www.lesswrong.com/posts/PagGF8roBJmjLunsX/competitive-ai-safety-is-the-loss-function-to-make-sure-ai https://www.lesswrong.com/posts/PagGF8roBJmjLunsX/competitiv...
- Vineeth147 2mo ago[dead]
- pigeons 3mo agoI wish there was an acronym for it that didn't collide with the radio tech.
- jorlow 3mo agoHonest question when was the radio tech lora on the hn frontpage? Im slightly shocked this still creates such confusion every time
- walrus01 3mo agoReading the title, for a moment I thought this would be something about a LoRA serial data bridge over RF and an actual physical wall clock, possibly some novel new homebuilt piece of hardware to display perfectly accurate NTP synchronized time or something.
- HumblyTossed 3mo agoSigh... Unfortunate naming.