8 ms·

Ideally what I'd like to see is pluggable knowledge bases. So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 1

• Gecko4072 2mo ago
  Sounds like unix philosophy. Or like Neo downloading Kung-Fu. Good either way.
  • goodmythical 2mo ago
    everything is a token
    • jagged-chisel 2mo ago
      There is no token
      • miek 2mo ago
        Form is token, token is emptiness.
  • oblio 2mo ago
    > Sounds like unix philosophy. https://linux.die.net/man/1/ls https://linux.die.net/man/1/ls https://linux.die.net/man/1/find https://linux.die.net/man/1/find https://linux.die.net/man/1/bash https://linux.die.net/man/1/bash
• jvuygbbkuurx 2mo ago
  If that is better (and possible) it will be baked in the tools. But is a model that doesn't know python better or worse at swift is what I wonder.
• locknitpicker 2mo ago
  > So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge. Aren't you describing RAG or even MCP servers? Heck, nowadays you get that also with agent skills and specialized tool calling.
  • kennywinker 2mo ago
    I don't think so? Definitely not MCP, as that pulls info into the context. Unless contexts become REALLY big so that I can add 10B in swift knowledge, that's not gonna help me. Possible RAG? I don't know enough about how that works, but I think that's not quite it either. I don't want to import facts like "the swift standard library contains a reverse array function", i more want to import knowledge - e.g. the parameters used to generate the text to reverse an array in swift. Tool calling wouldn't do it either. You'd have to encode every single possible bit of useful info into the tool call, and the tool response would have to encode every piece as well (variable names, function scopes, types defined in other files, etc). E.g. how does it find a bug, if you have to pass understanding back and forth between the brain that understands debugging and the brain that understands THIS code?
  • everforward 2mo ago
    I don’t think so because those both live in the context window and as such pollute it when they’re not performing optimally. I think having unused or rarely used weights doesn’t influence the results as poorly as RAG injecting irrelevant facts. It sounds to me like some sort of “dynamic MoE” where you can add/create or remove experts on the fly. I think what you’re describing is the closest approximation we reasonably have right now though.
    • locknitpicker 2mo ago
      > I don’t think so because those both live in the context window and as such pollute it when they’re not performing optimally. There is nothing optimal about needing a few billion more parameters to be able to piece together probable answers that can be asserted by querying an oracle. > I think having unused or rarely used weights doesn’t influence the results as poorly as RAG injecting irrelevant facts. Those aren't free. The more parameters you add, the higher the computational cost required to train and prompt a mode. And all for what? To piece together info that you can just query from a data source?
• edgyquant 2mo ago
  This would defeat the AGI narrative/belief that so many building these models have
  • scld 2mo ago
    On the otherhand, your own brain probably doesn't use your eyes to hear.
    • alightsoul 2mo ago
      You could if you do lip reading
    • nvme0n1p1 2mo ago
      You don't think reading lips helps you understand people?
      • hoppp 2mo ago
        That's still not hearing. Hearing has volume, direction, pitch, it's spacial processing etc
        • kennywinker 2mo ago
          Laurel / yanny has entered the chat. Hearing is also psycho, not just acoustic.
          • Izkata 2mo ago
            Brainstorm / Green Needle: https://www.youtube.com/watch?v=1okD66RmktA https://www.youtube.com/watch?v=1okD66RmktA I can change which one I hear on a whim, so much that I can even get Brain Needle and Green Storm out of this video.
    • vasilvv 2mo ago
      Counterintuitively, seeing things can alter the perception of individual sounds you're hearing: https://en.wikipedia.org/wiki/McGurk_effect https://en.wikipedia.org/wiki/McGurk_effect
• kalb_almas 2mo ago
  I would love this but I think the General Reasoning and Make No Mistakes modules would be massive.
  • kennywinker 2mo ago
    I mean, I get pretty damn good general reasoning out of Qwen3.8-27B quantized to 4bits, and that knows Swift, Python, Node, Ruby, Rust, etc. etc. etc. And so far even the biggest model doesn't seem to have a working Make No Mistakes module, so maybe that's not needed
    • dist-epoch 2mo ago
      Qwen3.6-27B (not 3.8) was a much better coder than Gemma4-31B, yet Gemma4-31B was a much better reasoner and general LLM to talk with. Sure, 3.8 maybe it's better now, but an accurate comparison would be with a new Gemma4-31B iteration (that doesn't exist).
      • kennywinker 2mo ago
        I guess the question is how much of the coding skill gap between qwen3.6-27b and gemma4-31b is down to the model architecture, and how much is because of the training data. Assuming you can tweak the training data, regenerate qwen3.6, and get a better coder, then presumably you could have variants - e.g. qwen3.6-swift-27b and qwen3.6-python-27b. Or maybe all coding is too intertwined and you can only get splits like qwen3.6-research-27b and qwen3.6-coding-27b. Which isn't quite my pluggable-models dream, but it's a step closer. But maybe the difference isn't the training data, it's the architecture, in which case pluggable models is probably not possible.
• jeremyloy_wt 2mo ago
  Admittedly I’m pretty ignorant of the details, but I thought this was the mixture of experts architecture
  • vlovich123 2mo ago
    No, that’s a common misconception. MoE despite the name doesn’t have an “expert” slice that’s an expert in any particular thing. It’s just trained models such that a subset of all memory weights is active on a single token. Often, the weights active on the next token are different. It’s used to improve memory bandwidth (throughput) and also to even out load in a distributed serving model - one GPU hosts one “expert” and the conversation is routed to it for the next token. This requires a lot of movement of the KV cache of course so it’s difficult to retain speed across multi node splits - usually it’s a rack of GPUs that you do this for.
  • spmurrayzzz 2mo ago
    MoE's in the abstract often get presented as if theres explicit layers of experts for any given domain of knowledge, like your coding tasks are being routed to coding experts, but it's really not that at all. THe original MoE paper from Noam Shazeer et al. is worth a read on this bit, though the paper is admittedly pretty dense. But TL;DR is that each expert layer is learning highly abstract, localized structural and syntactic patterns in the data to minimize the loss function, and its doing this token-by-token (which in some cases may have some domain clustering, but that's just incidental). When you start batching your queries, even if they all seem like theyre in a single domain, if you visualized the activations you'd notice that most if not all of the network is lighting up on the batched forward pass.
• gopalv 2mo ago
  > I want to click together a model that is laser-focused on what I am doing This is roughly what multi-agent systems are built for. This is possible with models too, but "making one on the fly" is much easier with agent coordination rather than model weights, since they all speak the same language. There is an IBM Mainframe vs Google Distributed system division here. Like Seymour Cray said - two oxen or 1024 chickens. Chickens are harder to harness, so a lot of my work is in sled-dog territory for agent harnesses & command structures.
  • kennywinker 2mo ago
    > This is roughly what multi-agent systems are built for. I think I disagree. For some things, maybe that works - but think of a multi-agent system where one agent understands the code, and passes it off to the reasoning agent to figure out what the bug is. This system is going to suck. Because encoding enough info to figure out what the bug is would just be dumping every single line of the code. So say agent 1 (reasoning) asks agent 2 (swift) to explain what is happening in File.swift. Anything agent 2 passes to agent 1 short of the entire code is a lossy transfer - and then the bug gets missed.
• conartist6 2mo ago
  This is going to be how I write my resume now: how many billion parameters of knowledge I have on each topic
  • kennywinker 2mo ago
    ...kennywinker is a 0.001B MoE model, with 2 active parameters.
• alightsoul 2mo ago
  This is possible today with an agent such as OpenClaw and hermes
  • kennywinker 2mo ago
    Only if you fundamentally misunderstand what I am describing
    • alightsoul 2mo ago
      You cannot have a single purpose LLM. Every single topic contributes to the performance of an LLM on any given area. You cannot have what you want. LLMs do not work like professions, college degrees or people. Using your examples, it is damaging to just know a single programming language, because there are patterns that are more common in say python than in swift, even though the only thing you want is swift, so LLM performance in swift benefits from pythonic patterns. Programming logic is the same in all languages so by having only swift and no other languages, you remove the number of examples the LLM is trained on which degrades performance in Swift which is the only thing you want
      • kennywinker 2mo ago
        > You cannot have a single purpose LLM > so LLM performance in swift benefits from pythonic patterns What I hear you saying is that the best way to make a swift-trained-only LLM smarter is to train it on some python too. And then with an infinite parameter budget, every other programming language or really any other data you train it on makes the model smarter - I accept that premise. But in a fixed parameter budget, what is better? training on 50% Swift + 50% Python, or 50% Swift + 50% Rust. Because if I am doing Swift programming, I want whichever of the two is better for Swift. If I am doing Rust programming, maybe I want the model trained on 50% Python + 50% Rust. Sure, it would be smarter if you tossed in the swift code too - but we have a budget to stick to. Now is it possible to make those pluggable? i.e. can you take a model trained on 50% python, and layer on 50% rust OR swift depending on what language you're using? Probably not right now, but maybe one day?
• hkt 2mo ago
  I strongly suspect this will be the future
  • RealWed6 2mo ago
    The future will be "AI operator license #123..., subclass 'Primitive Coding'", with your TouchID / FaceID enabled only. And I am not being sarcstic.
• pton_xd 2mo ago
  > My model doesn't need to know a single line of python. If I had to guess, the weights necessary to encode "how to program" are much larger than the final step of "output python."
  • echelon 2mo ago
    But the understanding of the language library ecosystem, or even better, your codebase, could let it execute faster and with less context usage.
    • exe34 2mo ago
      That's the bit I feel you can do with ragging - keep a large library of well described solutions and then find what you need from it at runtime.
  • mathisfun123 2mo ago
    <THAT'S A BINGO> ie what everyone asking for this fails to immediately realize.
    • card_zero 2mo ago
      You're a bingo. It's obvious that a model could know no Python, since Python could not exist in the world in the first place.
      • kennywinker 2mo ago
        the best llm for coding is the one that knows every programming language imagined by a caffeine-fuelled comp-sci student at 2am but never built.
        • Retro_Dev 2mo ago
          if neuralink ever becomes a thing, thoughts about programming might be stolen for LLM training data lol
• stanac 2mo ago
  Sounds like MoE but more pluggable. Not sure if anyone is researching something like this. I still think your swift or GIS model will need basic reasoning and coding to work, so it's more like multiple smaller models which you can load as needed, e.g. sub-agents for GIS needs the GIS model. Another approach would be to have basic coding and reasoning model and then load specification for language and libraries into context, it could work for self-hosted models, but I don't want whole specification of the language to be send to API and waste tokens on that.
  • collabs 2mo ago
    My motivation is somehow the model should be able to work without hitting an external resource every single time and somehow have all the knowledge necessary locally so we don't get rate limited but how can you stay up to date with the latest information while still remaining a good boy scout by not reaching out to tool use and scraping every single GitHub repo and issue every single time?
  • Muromec 2mo ago
    I would, once it is possible, specifically unplug the shitcoder expert that sometimes shows up and puts `as unknown as` and bypasses the commit hook that is supposed to enforce it. That should be somewhere near java coder expert in the whatever dimensional space. I can feel it being true.
• jimmaswell 2mo ago
  An LLM works better the more disparate world knowledge it has, even if it's not immediately obvious why it would be relevant. The model finds a structure to the problem you give it in a largely language-agnostic way that benefits from training on every language (these things are direct descendants of Google Translate), and even non-programming knowledge - the structure of your task might resemble an ancient Chinese poem that influences the model's response, for example. That structure is considered a form of compression, as some fascinating and illuminating recent 3blue1brown videos get into - a common pattern in Haskell or FORTRAN and a situation described in an ancient Chinese poem may all compress to something quite similar to your task, thus when the model compresses the idea of your task it immediately draws from those ideas. There are "experts" which do divide parts of the model that are found to activate together for specific tasks, so they can be processed in parallel to join the result at the end, but it's nowhere near the granularity of a SwiftUI expert and a python expert. The difference in those things is so trivial from an abstract point of view that it would make no sense. They would be 99% the same. Distillations also come into this but I'm highly skeptical you could make one guaranteed to only know programming and only in one programming language (especially with as small a sample set as SwiftUI relative to something like C) without its efficacy being hobbled by tunnel vision. Reminiscent of the SpongeBob episode where he empties his mind of everything except fine dining and breathing, then can't remember his name and goes insane. Beyond the basic concepts of general coding and the trivia of syntax, getting anything done requires a large intersection of disparate world knowledge and the ability to apply it to new situations.
  • znnajdla 2mo ago
    Thanks for putting this so well. The mathematical evidence for this “general intelligence underlying everything” is the “universal geometry of embeddings” paper. Fascinating read. Or as the ancient philosophers used to say, the one who knows God knows everything.
    • jan_m_savage 2mo ago
      [dead]
    • 2mo ago
• znnajdla 2mo ago
  This is a fundamental misunderstanding of how LLMs work. You can’t really specialize a model. You specialize the harness. A well-trained general purpose LLM doesn’t need examples in its training data, it can write good code in a new language you invented yesterday with just a spec definition. And it will perform better than a small model trained on lots of examples of your invented language. The reason is because of the “universal geometry of embeddings”, i.e all human languages have the same underlying pattern structure, so any model that is very good in any language is good in all languages. Attempting to specialize a model for a particular purpose often decreases overall performance. Fine-tuning is just a hack to make dumb models more reliable on limited tasks but they become incapable of doing anything else. Unless you are building a factory assembly line where a model is literally doing the same thing over and over, you almost always want a general purpose model over a specialized one.
  • chopete3 2mo ago
    This is so right. We training Whisper Large model on 20,000 audio samples specific to a domain and it ended up reducing the ASR by 5% while improving WER of the finetuned domain by 0.5%. Instead we ended up with no finetuning. We give audio snippet to 2 AsR models, take 3 best transcriptions and ask the LLm to pick the best based on the context. That produced significantly higher accuracy in how an agent understands the users.
    • twalla 2mo ago
      Can you go into more detail on this? I’ve been putting together the data pipeline for an ASR fine tune but your approach sounds more interesting.
      • teravor 2mo ago
        basically he is feeding the same input to multiple models, taking their outputs and dumping it into an LLM to sort out what the actual transcription probably is. expensive but effective.
    • aetherspawn 2mo ago
• saulpw 2mo ago
  The problem with this idea is that knowing Python makes the model a better Swift programmer, as does a higher-number of parameters during training. So you'd be so much better off with a 90B general purpose model trained on everything anyway.
  • kennywinker 2mo ago
    Except assuming a fixed budget of parameters, there is clearly stuff that's better for programming than others. E.g. Qwen 27b is a better coding model than Gemma 4 31B. More params doesn't automatically win. Perhaps what makes a good swift programming model is a ton of python training, so those two things can't be separated - but that doesn't negate the idea of loading a model that's good at the specific task you want - or has specific knowledge of the libraries and tools for the language you're using at the expense of the ones you're not using.
• throwaway1492 2mo ago
  > So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge. My model doesn't need to know a single line of python. Tell me you don’t know how llm work without telling me you don’t know how llm work. That’s not how they work!
  • kennywinker 2mo ago
    I'm not actually describing a specific how, I'm describing a goal. The goal is models that are made for specific work, not general work. My model doesn't need to know a single line of python, but if it's a coding model perhaps giving it some python is the best way to make it smarter. Ok. Great - give it python! But first maybe check if giving it C++ is actually better than giving it python. Because for my purposes it doesn't need to know C++ or Python, and I have a fixed parameter budget. Find me the ratio of Swift:C++:Python that gets me the best Swift coding agent in my parameter budget, and then train that. And then ideally, make it pluggable so I can pick what I want from off the shelf components, but if that's not possible - then just train up as many variants as you can so we can all pick the best variant for our current need.
• dofm 2mo ago
  (Inexpert ramblings follow) Part of the problem of this is likely that the deep meanings of words you might use in chat to describe a business problem or task that you wish to see implemented are essentially inseparable from scenarios in which they are used. Putting aside the bouba/kiki effect and anything like it, complex words only have meanings from usage. That usage is built on grammatical structures that also emerged only from usage. (This is something I was taught as a sort of fact but I gather it was basically abbreviated Wittgenstein? … who I cannot claim to have studied) So what you're looking for is a language model where fundamental word meanings are encoded without the weight of knowledge of where they come from. This is plainly difficult, because complex words are used by extension and analogy, and these days, many are neologisms or portmanteaus, even ephemerally — developed and discarded within a single context. Reasoning about language itself to its full meaning is quite hard. Like my favourite word of the moment: "obscurantist". You see that and you have a glimmer of what it might convey. But why do you? How much of that comes from explicit grammatical knowledge of suffixes, and how much from simple experience of using words like obscured, informant, attendant, dentist, artist? So a language model might be able to deduce what "obscurantist" logically means when applied to a tract or to a person. But without lots of parameters covering its use, could it properly grasp that in some circles it would be pejorative to the point of being deeply offensive? I think the best hope for your pluggable knowledge base idea is model delegation: strong reasoning models that know how to dictate to smaller specialist models and draw conclusions from their responses. I find myself wondering if there's any way that can be done the same way that, say, Gemma 4 12B's integrated vision encoder works — within shared weights, somehow, without them to speak in some intermediate language, like a partitioned brain. But I find it difficult to believe that is pluggable at all.
• nojokepoke 2mo ago
  The LLM is the reasoning engine that uses natural language. You’re describing skills. It’s the natural evolution.
• kelnos 2mo ago
  > I don't want general purpose models. They try to be everything to everyone. I think the vast majority of people do want general purpose models. They want to be able to ask it any question, or ask it to perform any task, and for it to do a decent job at it. I agree that it's really hard (maybe even impossible) to build something that's everything for everyone. But your average (or even above-average) LLM user doesn't want to choose from a catalog to stitch together a model that does just what they need. I do think for certain domains this is useful and will make sense: the model backing a coding harness doesn't need to know about the politics of 400BCE Rome. But I'm skeptical that many software developers will want to do what you propose, picking knowledge bases that are tailored to their current task or project. And at any rate, for web-based chat interfaces, most users just want to type a query and get an answer.
  • kennywinker 2mo ago
    I want to pick it myself because I want to run this stuff locally on hardware I can afford today. But most people seem happy enough using the cloud, where this kind of architecture could be seamless if it existed. I.e. the query to `/chat/completions` contains an extra parameter `domains: ["c++", "swift", "navigation" "gis"]` and then those are the modules that get allocated to this query. When you start a new conversation no domains are set and it hits a generalist model, but the generalist model includes domains so the next query doesn't have to hit the generalist model. The model could even have a tool it could call to rope in new domains if the scope expanded to include other modules.
  • dosisking 2mo ago
    > I think the vast majority of people do want general purpose models. They want to be able to ask it any question, or ask it to perform any task, and for it to do a decent job at it. The vast majority of people listen to the music of Ed Sheeran and think that it is good.
• c7b 2mo ago
  Sounds a bit like 'I want to make horses faster, surely I won't need mechanical engineering knowledge'. We don't know everything that we don't know, so it's hard to say what we don't need to know.
  • kennywinker 2mo ago
    But I am not asking for a faster horse. I am asking for a draft horse instead of a race horse. I know the tasks I have on hand, I know the VRAM and compute budget I have to run them. I am not asking for AGI, I just want something to edit my little text files.
    • c7b 2mo ago
      At a more technical level, what do you suggest? Training a small LLM on Python code exclusively? And then one on general CS/algorithms, which you'll also need? I don't think the current transformer architectures would compose as you suggest.
      • kennywinker 2mo ago
        With what I know about how LLMs work now, I guess I am suggesting more specific variants. Qwen3.8 has a 2.4T version and a 27B version. I understand that to mean that they are the same architecture, just one version has a massive training set and the other has a very small subset. So, it seems very possible that variants of 27B could be generated that tune it for specific things by selecting different training data from the large corpus. One model for Python, another for Swift, another for research, another for creative writing, etc. I think you're right that current architectures don't compose like that - but I feel like that's a result of the focus on MOAR DATA, and a "race for AGI" - if we set those ideas aside, a more composable architecture seems very possible.
        • c7b 2mo ago
          > I understand that to mean that they are the same architecture, just one version has a massive training set and the other has a very small subset. No. It means that the one model has 2.4 trillion parameters while the other has only 27 billion. I don't know the details about their architecture or training, but presumably they used the same or similar training sets for both and a conceptually similar architecture, scaled down. I'd guess they also have some techniques to re-use some of the work done for the big model for the smaller versions (if anyone knows more about this I'd be interested). The architectures cannot be identical by definition because then the parameter count would be the same. Subsetting the data to such narrow fields as you describe could risk losing some edge, there are a lot of emergent capabilities in those models and I don't think that emergence is fully understood yet. There are subject-specific models, but for far broader subject areas than you suggested, like coding or math or prose. I'm sure composability is possible in principle, I'm just sceptical that it'll be a good long-term solution, for my originally stated reason. It's basically just The Bitter Lesson again, we may gain some short-lived edge by putting more domain knowledge into the algorithm, but ultimately (these days often: surprisingly quickly) it'll be outgunned by something that just leverages raw computation better.
• dools 2mo ago
  This just means you have to describe what you want in Swift or whatever. If it doesn’t have the language then it doesn’t have the capability to transform intent into code.
  • kennywinker 2mo ago
    In my hypothetical, 9B of coding and reasoning includes enough language to communicate ideas to the model.
• fennecbutt 2mo ago
  I was hoping model architecture would migrate towards something like this, perhaps it still might. Sparse models seem like they're getting more popular. I'd always thought we'd eventually hotload loras or MoE experts. It would certainly be useful on the robotics/VLA side of things as well; more limited mobile hardware, download and load/unload new skills as needed. Tbf I also don't really care what facts my models have baked in (for llms at least). I care most that the model understands general logic and then general knowledge of some level is secondary. Reason being is that everything is RAG'd in anyway. Models spitting out well established facts is cute but I don't really ever want to rely on say "electronics knowledge" that exists in a tenuous and vague form in the model weights. Humans write books (and datasheets) for a reason. Books are RAG.
• SoftTalker 2mo ago
  If all the model knows is basic coding and Swift, then that's how you'll have to talk to it. At that point you might as well write the code yourself.
• anjel 2mo ago
  I shudder to imagine the monetization schemes of such an architecture though.
• brainless 2mo ago
  This is how my experiments go. And I am sure there are popular agents that do this. How I am trying is to create "Rust Engineer", "Typescript Engineer" or even "Rust Diesel Engineer". I have not tried fine-tuning. I focus on a small model, usually Qwen3.5 9B. I take a bunch of open source repositories and build a KG on it. A small model should be able to enrich your prompt and add technical context. The final, enriched prompt goes to the capable model.
• JV00 2mo ago
  “I know Kung Fu” - Neo (The Matrix, 1999)
• xtiansimon 2mo ago
  I'd [subscribe to] 9B of basic coding and reasoning, add [subscription for] 10B of swift/swiftUI, add [monthly lease of] 5B of GIS/geography knowledge and another [monthly obligation] 5B of frontend app design knowledge. LOL Right now we have the Star Trek computer. A polymath of knowledge. But I’m sure you’re right. It’s the trend of all knowing to move from the bazaars to the temples, and back again.