4 ms·
I wonder if it really needs to be worse. I am playing with the idea of fine tuning a model on my exact stack and coding patterns. I suspect I could get better p
by digitaltrees 5mo ago
I wonder if it really needs to be worse. I am playing with the idea of fine tuning a model on my exact stack and coding patterns. I suspect I could get better performance by training “taste” into a model rather than breadth.
- epicureanideal 5mo agoI also wonder about JS only, Python only, etc models. Maybe the future is a selection of local, specific stack trained models?
- andy_ppp 5mo agoThese models being able to generalise at coding will likely get worse if you remove high quality training data like all of python.
- robrenaud 5mo agoThere is some recent work on modularizing knowledge in LLMs. https://arxiv.org/html/2605.06663v1 https://arxiv.org/html/2605.06663v1 It might be possible to train a big generalist that is a composition of modules, some of which can be dropped dynamically at inference time, depending on the prompt.
- digitaltrees 5mo agoCool. Thanks for sharing. I am thinking about creating a series of smaller models for specific purposes and then orchestrating them so they mirror the human brain which is a bunch of subsystems that give multiple opinions about the same stimulus
- shailendra_sis 5mo agoInteresting direction. I’ve also been thinking about modular / subsystem-based approaches for specialized tasks in small AI systems.
- andy_ppp 5mo agoFine tuning these models (at least with PPO or equivalent) requires even more VRAM than inference does, potentially 2-3 times more.
- jimbokun 5mo agoThat approach has its advantages, but sometimes I want to generate code for a language or kind of project I’m not experienced with using the accepted best practices.