3 ms·

This doesn't give you the ability to process ideas through the derived new insights, any more than loading the contents of a VLSI program into regular RAM gives

• lxgr 2y ago
  > The linear-algebra primitives used in LLM inference, fundamentally do not have the power to allow an LLM to "emulate" its own internals […] You need host support for that. Neither do biological brains (explicitly), yet we can hypothesize just fine.
  • derefr 2y ago
    You're conflating two steps: 1. hypothesizing — coming up with a novel insight at runtime, that uncovers new parts of the state space the model doesn't currently reach 2. syllogizing — using an insight you've derived at runtime, to reach the new parts of the state space LLMs can do 1, but not 2. (Try it for yourself: get an LLM to prove a trivial novel mathematical theorem [or just describe the theorem to it yourself]; and then ask it to use the theorem to solve a problem. It won't be able to do it. It "understands" the theorem as data; but it doesn't have weights shaped like an emulator that can execute the theorem-modelled-as-data against the context. And, as far as I understand them, current Transformer-ish models cannot "learn" such an emulator as a feature. You need a slightly different architecture for that.) And actually, humans can't really do 2 either! That is: humans can't immediately make use of entirely-novel insights that weren't "trained in", but only just came to them, any more than LLMs can! Instead, for humans, the process we go through is either: • come up with the insight; sleep on it (i.e. do incremental training, converting the data into new weights); use the insight • build up 99% of the weights required for the insight "in the background" over days/months/years without realizing it; make the final single connection to "unlock" the insight; immediately use the insight LLMs don't get to do either of these things. LLMs don't do "memory consolidation"; there is no gradual online/semi-online conversion of "experiences" into weights, i.e. reifying the "code stored as data" into becoming "code" that can be executed as part of the model. With (current) LLMs, there's only the entirely-offline training/fine-tuning/RLHF — at much greater expense and requiring much greater hardware resources than inference does — to produce a new iteration of the model. That's why we're (currently) stuck in a paradigm of throwing prompts at ever-larger GPT base models — rather than just having an arbitrary stateful base-model that you "install" onto a device like you'd install an RDBMS, and then have it "learn on the job" from there.
    • lxgr 2y ago
      > And actually, humans can't really do 2 either! > That is: humans can't immediately make use of entirely-novel insights that weren't "trained in", but only just came to them, any more than LLMs can! Agreed – but I'd argue that they both can, albeit in an extremely clunky way (i.e. very similar to "chain-of-thought" LLMs): Mechanically applying the new insights in a low-efficiency, "emulated" layer. > LLMs don't get to do either of these things. LLMs don't do "memory consolidation"; there is no gradual online/semi-online conversion of "experiences" into weights, i.e. reifying the "code stored as data" into becoming "code" that can be executed as part of the model. At the moment that's definitely their biggest weakness, but one could argue that memory consolidation happens ~once per year, globally, as past interactions with them almost certainly become future training data. In some ways, that's more powerful than a single human gradient descending overnight and in the shower; in others, it's obviously much worse.
      • derefr 2y ago
        > memory consolidation happens ~once per year, globally, as past interactions with them almost certainly become future training data. Yes and no. Humans — organic animal brains in general — have a seemingly highly-clever approach to memory consolidation / online training. The organic process for memory consolidation seems to take advantage of our own analytical capabilities to pick apart and amplify the salient and practically-useful parts of our memories. As such, animal brains seemingly need much less training data to learn a skill than current state-of-the-art model architectures do. You could say that they can leverage an (incremental) training dataset to a much higher degree than current NN training architectures do. (It's sort of the complement to a better compression algorithm having a higher degree of signal-loss-less compression for the same data: animal brains have a higher degree of noise-gain-less data extrapolation over the same data.) If we could figure out what organic brains were doing here, and apply it to the sort of "global fan-in / reduce / fan-out" approach bigcorps use to train new base-model revisions, then I'd agree that the choice of online vs offline training would become just a trade-off of locality vs scale. After all, in the end what you want is to be able to train a model to do something in "soft real time" (taking a few days/weeks to learn, including time for rest); with the model learning to perform the task "on the job" by interacting with the task, like a human/animal would. It doesn't really matter whether the training has to be sent back to the cloud to be incorporated into a new model revision; it just matters that it happens and that you see the results. So if the distributed training architectures used to train GPT base models could "notice insights" in small amounts of RLHF data, and amplify/extrapolate large amounts of new training data from those — then you might expect the model that "learned in the cloud" to "learn on the job" just as well as the model that was local and only thinking about that training data. (But the GPT would have "learned on the job" some million other capabilities at the same time!) But this does start to sound like the kind of thing that violates some property of one of those "can only choose two" triangles. After all, current model training approaches — like everything to do with ML — are taking advantage of SIMD hardware. Which is why training clusters are big GPU clusters. But it seems like the thing our brains do in online training, involves some complex state-driven scheduling of passes over data. In other words, despite the brain itself being massively parallel, it's seemingly doing a lot of things that — if you were programming them — you'd want to push onto a CPU rather than a GPU. (To be specific: it's seemingly doing complex graph-search queries to bias the extrapolation process. And graph search is the antithesis of a horizontally-scalable problem.) And I have a feeling — just an intuition, not with anything concrete to back it up — that trying to scale up that dataset-extrapolation pass, where the framework "explores and connects memories to extract and amplify the insight signals"... would be something that would require most of the resources of the system just to extrapolate out one insight at a time. It wouldn't be able to be done in parallel for millions of capabilities — at least not when the model, and the search-space of incremental memories, are "large" (like a modern GPT model is; and like the RLHF dataset from one day's worth of millions of users' interactions would be.) If this were true, I would think it would be true for the same reasons that you can only vertically scale an IO-bound task so far — a single system only has so much IO bandwidth (e.g. PCIe lanes, DMA channels) available to interact with compute. (Mainframes have a lot of "accelerated" IO, but it's "accelerated" because it's interacting only with things that aren't the system's main compute. The CPU(s) "delegate" handling that IO to cores local to the IO. It's hierarchical NUMA.) When you have IO-bound workloads, you tend to scale horizontally — if you're building a CDN, you don't make big boxes; you just throw more boxes at the problem, where each is separately handling its own requests. And I feel like the translation here, is that if you want this insight-extrapolation, then it's only going to work when it's being done in a partitionable manner — by small models training on small incremental RLHF datasets, of the kind they build up through interaction with a single use-case at a time. And even though you could do that with a global fan-in pass, in the cloud, sharding training across millions of individual small models and assigning each both their own GPU and CPU resources... at that point, why bother? You'd be essentially using your big DC compute-cluster to emulate having millions of regular PCs that have a little GPU and CPU each, each working on their own task in a cache-coherent manner. At that point, just take advantage of the end-user devices that do the inference to also do the training! (Although, maybe you might still want to fan in the much-less-compact post-extrapolated training data that result from these individual on-device models stewing over their own problems; and then use that to train new general pre-trained base models. Or many such models, since you'd probably want smaller, more purpose-specific models in order to make on-device "on the job" training possible in the first place.) --- Funny enough, this could be an interesting hypothesis for the ultimate neurological — rather than biophysiological — role of sleep and dreaming! We do build regular neural connections while awake; but this memory-insight-extrapolation pass specifically, needs to "reserve" the entire brain to do what it does — so it only happens during sleep. It would explain the advice to "sleep on it" to enable lateral thinking. And it could be tested experimentally by measuring how well different animals extrapolate insights from small amounts of training, vs. how long they sleep.