6 ms·

The age of tiny models is just about here. We are finally busting away from the standard transformer block and training. I have a side project that can dramatic

The age of tiny models is just about here. We are finally busting away from the standard transformer block and training. I have a side project that can dramatically shrink networks by a set of techniques I call sacrificial training[1]. I think things like this will finally swing things back to on prem and on device small models that are as capable as the big hosted models are now. [1] https://github.com/jmward01/lmplay/wiki/Sacrificial-Training https://github.com/jmward01/lmplay/wiki/Sacrificial-Training

refulgentis 2y ago
RWKV is def. better than TinyStories 125MB. Unfortunately, I have only seen 3 models, 3B or over, handle RAG. Tested RWKV with a simple in-the-sports-news question and it didn't even get close to approaching the question. And nearly everything was fundamentally incoherent even in its internal reality (ex. Player gets 5000/game and is the first with 1000 in 16 games) (prompt: https://pastebin.com/sCLn5sCJ https://pastebin.com/sCLn5sCJ, response: https://pastebin.com/TqudvDbN https://pastebin.com/TqudvDbN) I don't think there's a position for LLMs that are "just" writers on the market in 2025.
- jasonjmcghee 2y ago
  These tiny models in general have really weird failure modes. I tried the tiny stories prompt about asking mom for a dog who said no, and it output an incredibly dark story about how she asked her dad and they got a dog but it had pancreatic cancer (paraphrasing, it went into detail about the surgery etc.) and then starting writing an informational PSA about who is at risk of pancreatic cancer etc.
  - kube-system 2y ago
    What I find fascinating is how ML models hallucinate in a way that is sometimes reminiscent of a fever dream.
    - deleted 2y ago
      [deleted]
    - ethbr1 2y ago
      It makes sense that the failure modes of language prediction look a lot like ADD.
      - p0w3n3d 2y ago
        It's because they are precisely lacking attention
        jdiff 2y ago
        Don't fall into the trap of applying human psychology to LLMs. Bag-of-chemistry quirks do not translate to matrix-multiplication quirks.
        ethbr1 2y ago
        Why not? In both cases the result is losing the thread of thought.
        deleted 2y ago
        [deleted]
        jdiff 2y ago
        ...No, no that's not how ADHD works. It's difficult to sum up how wrong this is concisely, but I invite you do to some serious research into ADHD, how it functions, and the great variety of ways in which it can present in different people. It's quite a poor analogy.
        ethbr1 2y ago
        I'm aware that anything to do with the brain has a variety of presentations. Could you try to put a couple sentences down on how ADHD is an inapt metaphor for failure modes in this case? It's lazy to claim something is wrong without offering a useful point as to how it's wrong. I trust in your ability to summarize.
  - nerdponx 2y ago
    Lest we forget that this stream-of-consciousness confusion was state of the art just a few years ago. It makes sense if you think about it: a small model's "internal state" isn't rich enough to keep track of whatever it was supposed to be talking about. It makes me think that the reason LLMs need to be so large is that the internal state needs to be bigger than a typical human "idea", whatever that might mean.
    - acchow 2y ago
      The way we do LLMs now is that the program and the data are one and the same. The program mutates itself as it "executes". This is probably also how the brain works since there is no hard separation between "memory" neurons and "data processing" neurons. (biology has no hard separation in general).
  - seattleeng 2y ago
    As is usually the case, check the data! A lot of the dataset used has fairly morbid scenarios, so the model is working as expected. All the data was synthetically created with GPT4
- jmward01 2y ago
  I plan on checking out RWKV and seeing if I can add my sacrifical training techniques to it this weekend. There is a reason quantization works, it is because models are very badly trained right now. I think we can get really good performance on .1b and 1b models which opens up the world to fine-tuning again. I was playing with fine-tuning llama 7b and 13b a while back but the HW/SW stack made it so unwieldy and the ROI was terrible compared to just adjusting prompts on gpt-4o-mini and the like. I have hope that we are about to see single GPU, very simple, fine-tuning again as models shrink and GPUs grow.
  - jmward01 2y ago
    I doubt anyone is still looking at this thread but I did actually start playing with RWKV by adding sacrificial training techniques to it and the results look promising, at least for early training.
- daxfohl 2y ago
  Would there be any way to distribute RAG across multiple smaller models? Rather than one giant model handling your entire document base, have it be more of a tree where the top level classifies the docs into top-level categories and sends it to submodels to subclassify, etc? (Doesn't have to be 1:1 classification). And same for q/a search? These could all presumably be the same physical instance, just each query would use a different system prompt and perhaps different embeddings. (I'm guessing; I don't actually know how RAG works). So, a little slower and clunkier, but presumably way more efficient. And match could be anywhere between horrible to better-than-one-large-model. This would be more like how businesses organize docs. Or maybe there's no real benefit to this, and each subclassifier would require just as big of a model as if you were to throw all docs into a single model anyway. I assume it's probably been tried before.
  - refulgentis 2y ago
    TL;DR: It's a very interesting line of thought that as late as Q2 2024, there were a couple thought leaders who pushed the idea we'd have, like 16 specialized local models. I could see that in the very long term, but as it stands, it works the way you intuited: 2 turkeys don't make an eagle, i.e. there's some critical size where its speaking coherently, and its at least an OOM bigger than it needs to be in order to be interesting for products fwiw RAG for me in this case is: - user asks q. - llm generates search queries. - search api returns urls. - web view downloads urls. - app turns html to text. - local embedding model turns text into chunks. - app decides, based on "character" limit configured by user, how many chunks to send. - LLM gets all the chunks, instructions + original question, and answers. It's incredibly interesting how many models fail this simple test, there's been multiple Google releases in the last year that just couldn't handle it. - Some of it is basic too small to be coherent, bigcos don't make that mistake though. - There's another critical threshold where the model doesn't wander off doing the traditional LLM task of completing rather than answering. What I mean is, throwing in 6 pages worth of retrieved webpages will cause some models to just start rambling like its writing more web pages, i.e. they're not able to "identify the context" of the web page snippets, and they ignore the instructions.
  - groby_b 2y ago
    There's just been a twitter post by Omar Khattab (@lateinteraction) on encoding documents into a scoring function instead of a simple vector for the work on ColBERT - and maybe at some point using a DNN as scoring function. So, yes, maybe there's a way to "distribute" RAG. (I still wonder if that isn't just MoE taken to its logical conclusion) So, dig for ColBERT papers, might be helpful. (I wish I had the time to do that)
- raegis 2y ago
  > Unfortunately, I have only seen 3 models, 3B or over, handle RAG. What's the unit "B" in "3B"? I can search for acronyms like "RAG" just fine, but you experts aren't making it easy for us beginners :) Edit: Apologies, this is obvious. My brain needed a reboot for the new year.
  - cauliflower2718 2y ago
    You can ask an LLM exactly this question and it will tell you. (The answer is billions of parameters)
    - SketchySeaBeast 2y ago
      But what if they want to know they are finding the correct answer?
      - elliotto 2y ago
        Asking anonymous people on a forum would be much better.
        SketchySeaBeast 2y ago
        At least a forum with domain-specific knowledge.
        gpm 2y ago
        And people to go "no, that's wrong" if someone posts something that's wrong.
        greesil 2y ago
        No, any answer will do.
      - cauliflower2718 2y ago
        I think basic definitions for LLMs are solidly within the bounds of what we would expect e.g. chatgpt to be competent at. The task (defining terms) is simple and the specific content (basic LLM stuff) is easy to check by anyone who works on the LLM. I agree with the general sentiment that we should not just blindly trust LLMs though.
- a1o 2y ago
  What is tiny and what is big? Can I have a model that is like 100MB in weights and run with llama.cpp in my MacBook M2?
  - refulgentis 2y ago
    Yeah, absolutely -- you'll probably pull 100+ token/s. Here's a good range of model sizes that run just fine with llama.cpp on mac: https://huggingface.co/telosnex/fllama/tree/main https://huggingface.co/telosnex/fllama/tree/main. I recommend trying the Telosnex* app, it uses llama.cpp and abstracts over LLMs so you can i.e. switch between local/servers at will. The important part for you is its free, accelerated on macOS, and very easy to use local LLMs with (Settings > AI > LLM > On Device, tap Get) Prepare to be underwhelmed, slightly: its only when you start hitting 3B that its coherent, anything under that will feel more like a markov chain than an LLM. Depending on how geeked out you'll be to have it running locally, you might have fun with that Telosnex can run local models on every platform, i.e. you can run local models on iOS/Android/web too. * because it's mine :3 It is quietly released currently. I want to get one more major update before widely announcing it in Jan 2025
    - qskousen 2y ago
      Sorry to side track, but question about Telosnex - would you consider a Linux release with something other than Snap? Maybe Flatpak or appimage?
      - refulgentis 2y ago
        If its a (mostly) CI-able process, I'm totally open to it --- I looked into "What should I do besides Snap?" about 4 months ago; got quickly overwhelmed, because I don't have enough knowledge to understand what's fringe vs. common. I'll definitely take a look at Flatpak again in the next month, 30 second Google says its possible (h/t /u/ damiano-ferrari at https://www.reddit.com/r/FlutterDev/comments/z35gdo/can_you_distribute_a_flutter_linux_app_as_a/ https://www.reddit.com/r/FlutterDev/comments/z35gdo/can_you_...) (thanks for your interest btw, been working on this for ~year and this is my first outside feature request :) may there be many more)
- wolfgangK 2y ago
  «Unfortunately, I have only seen 3 models, 3B or over, handle RAG.» I would love to know which are these 3 models, especially if they can perform grounded RAG. If you have models (and their grounded RAG prompt formats) to share, I'm very interested ! Thx.
mr_toad 2y ago
> small models that are as capable as the big hosted model Free lunch?

refulgentis 2y ago

RWKV is def. better than TinyStories 125MB. Unfortunately, I have only seen 3 models, 3B or over, handle RAG. Tested RWKV with a simple in-the-sports-news question and it didn't even get close to approaching the question. And nearly everything was fundamentally incoherent even in its internal reality (ex. Player gets 5000/game and is the first with 1000 in 16 games) (prompt: https://pastebin.com/sCLn5sCJ https://pastebin.com/sCLn5sCJ, response: https://pastebin.com/TqudvDbN https://pastebin.com/TqudvDbN) I don't think there's a position for LLMs that are "just" writers on the market in 2025.

jasonjmcghee 2y ago
These tiny models in general have really weird failure modes. I tried the tiny stories prompt about asking mom for a dog who said no, and it output an incredibly dark story about how she asked her dad and they got a dog but it had pancreatic cancer (paraphrasing, it went into detail about the surgery etc.) and then starting writing an informational PSA about who is at risk of pancreatic cancer etc.
- kube-system 2y ago
  What I find fascinating is how ML models hallucinate in a way that is sometimes reminiscent of a fever dream.
  - deleted 2y ago
    [deleted]
  - ethbr1 2y ago
    It makes sense that the failure modes of language prediction look a lot like ADD.
    - p0w3n3d 2y ago
      It's because they are precisely lacking attention
      - jdiff 2y ago
        Don't fall into the trap of applying human psychology to LLMs. Bag-of-chemistry quirks do not translate to matrix-multiplication quirks.
        ethbr1 2y ago
        Why not? In both cases the result is losing the thread of thought.
        deleted 2y ago
        [deleted]
        jdiff 2y ago
        ...No, no that's not how ADHD works. It's difficult to sum up how wrong this is concisely, but I invite you do to some serious research into ADHD, how it functions, and the great variety of ways in which it can present in different people. It's quite a poor analogy.
        ethbr1 2y ago
        I'm aware that anything to do with the brain has a variety of presentations. Could you try to put a couple sentences down on how ADHD is an inapt metaphor for failure modes in this case? It's lazy to claim something is wrong without offering a useful point as to how it's wrong. I trust in your ability to summarize.
- nerdponx 2y ago
  Lest we forget that this stream-of-consciousness confusion was state of the art just a few years ago. It makes sense if you think about it: a small model's "internal state" isn't rich enough to keep track of whatever it was supposed to be talking about. It makes me think that the reason LLMs need to be so large is that the internal state needs to be bigger than a typical human "idea", whatever that might mean.
  - acchow 2y ago
    The way we do LLMs now is that the program and the data are one and the same. The program mutates itself as it "executes". This is probably also how the brain works since there is no hard separation between "memory" neurons and "data processing" neurons. (biology has no hard separation in general).
- seattleeng 2y ago
  As is usually the case, check the data! A lot of the dataset used has fairly morbid scenarios, so the model is working as expected. All the data was synthetically created with GPT4
jmward01 2y ago
I plan on checking out RWKV and seeing if I can add my sacrifical training techniques to it this weekend. There is a reason quantization works, it is because models are very badly trained right now. I think we can get really good performance on .1b and 1b models which opens up the world to fine-tuning again. I was playing with fine-tuning llama 7b and 13b a while back but the HW/SW stack made it so unwieldy and the ROI was terrible compared to just adjusting prompts on gpt-4o-mini and the like. I have hope that we are about to see single GPU, very simple, fine-tuning again as models shrink and GPUs grow.
- jmward01 2y ago
  I doubt anyone is still looking at this thread but I did actually start playing with RWKV by adding sacrificial training techniques to it and the results look promising, at least for early training.
daxfohl 2y ago
Would there be any way to distribute RAG across multiple smaller models? Rather than one giant model handling your entire document base, have it be more of a tree where the top level classifies the docs into top-level categories and sends it to submodels to subclassify, etc? (Doesn't have to be 1:1 classification). And same for q/a search? These could all presumably be the same physical instance, just each query would use a different system prompt and perhaps different embeddings. (I'm guessing; I don't actually know how RAG works). So, a little slower and clunkier, but presumably way more efficient. And match could be anywhere between horrible to better-than-one-large-model. This would be more like how businesses organize docs. Or maybe there's no real benefit to this, and each subclassifier would require just as big of a model as if you were to throw all docs into a single model anyway. I assume it's probably been tried before.
- refulgentis 2y ago
  TL;DR: It's a very interesting line of thought that as late as Q2 2024, there were a couple thought leaders who pushed the idea we'd have, like 16 specialized local models. I could see that in the very long term, but as it stands, it works the way you intuited: 2 turkeys don't make an eagle, i.e. there's some critical size where its speaking coherently, and its at least an OOM bigger than it needs to be in order to be interesting for products fwiw RAG for me in this case is: - user asks q. - llm generates search queries. - search api returns urls. - web view downloads urls. - app turns html to text. - local embedding model turns text into chunks. - app decides, based on "character" limit configured by user, how many chunks to send. - LLM gets all the chunks, instructions + original question, and answers. It's incredibly interesting how many models fail this simple test, there's been multiple Google releases in the last year that just couldn't handle it. - Some of it is basic too small to be coherent, bigcos don't make that mistake though. - There's another critical threshold where the model doesn't wander off doing the traditional LLM task of completing rather than answering. What I mean is, throwing in 6 pages worth of retrieved webpages will cause some models to just start rambling like its writing more web pages, i.e. they're not able to "identify the context" of the web page snippets, and they ignore the instructions.
- groby_b 2y ago
  There's just been a twitter post by Omar Khattab (@lateinteraction) on encoding documents into a scoring function instead of a simple vector for the work on ColBERT - and maybe at some point using a DNN as scoring function. So, yes, maybe there's a way to "distribute" RAG. (I still wonder if that isn't just MoE taken to its logical conclusion) So, dig for ColBERT papers, might be helpful. (I wish I had the time to do that)
raegis 2y ago
> Unfortunately, I have only seen 3 models, 3B or over, handle RAG. What's the unit "B" in "3B"? I can search for acronyms like "RAG" just fine, but you experts aren't making it easy for us beginners :) Edit: Apologies, this is obvious. My brain needed a reboot for the new year.
- cauliflower2718 2y ago
  You can ask an LLM exactly this question and it will tell you. (The answer is billions of parameters)
  - SketchySeaBeast 2y ago
    But what if they want to know they are finding the correct answer?
    - elliotto 2y ago
      Asking anonymous people on a forum would be much better.
      - SketchySeaBeast 2y ago
        At least a forum with domain-specific knowledge.
        gpm 2y ago
        And people to go "no, that's wrong" if someone posts something that's wrong.
      - greesil 2y ago
        No, any answer will do.
    - cauliflower2718 2y ago
      I think basic definitions for LLMs are solidly within the bounds of what we would expect e.g. chatgpt to be competent at. The task (defining terms) is simple and the specific content (basic LLM stuff) is easy to check by anyone who works on the LLM. I agree with the general sentiment that we should not just blindly trust LLMs though.
a1o 2y ago
What is tiny and what is big? Can I have a model that is like 100MB in weights and run with llama.cpp in my MacBook M2?
- refulgentis 2y ago
  Yeah, absolutely -- you'll probably pull 100+ token/s. Here's a good range of model sizes that run just fine with llama.cpp on mac: https://huggingface.co/telosnex/fllama/tree/main https://huggingface.co/telosnex/fllama/tree/main. I recommend trying the Telosnex* app, it uses llama.cpp and abstracts over LLMs so you can i.e. switch between local/servers at will. The important part for you is its free, accelerated on macOS, and very easy to use local LLMs with (Settings > AI > LLM > On Device, tap Get) Prepare to be underwhelmed, slightly: its only when you start hitting 3B that its coherent, anything under that will feel more like a markov chain than an LLM. Depending on how geeked out you'll be to have it running locally, you might have fun with that Telosnex can run local models on every platform, i.e. you can run local models on iOS/Android/web too. * because it's mine :3 It is quietly released currently. I want to get one more major update before widely announcing it in Jan 2025
  - qskousen 2y ago
    Sorry to side track, but question about Telosnex - would you consider a Linux release with something other than Snap? Maybe Flatpak or appimage?
    - refulgentis 2y ago
      If its a (mostly) CI-able process, I'm totally open to it --- I looked into "What should I do besides Snap?" about 4 months ago; got quickly overwhelmed, because I don't have enough knowledge to understand what's fringe vs. common. I'll definitely take a look at Flatpak again in the next month, 30 second Google says its possible (h/t /u/ damiano-ferrari at https://www.reddit.com/r/FlutterDev/comments/z35gdo/can_you_distribute_a_flutter_linux_app_as_a/ https://www.reddit.com/r/FlutterDev/comments/z35gdo/can_you_...) (thanks for your interest btw, been working on this for ~year and this is my first outside feature request :) may there be many more)
wolfgangK 2y ago
«Unfortunately, I have only seen 3 models, 3B or over, handle RAG.» I would love to know which are these 3 models, especially if they can perform grounded RAG. If you have models (and their grounded RAG prompt formats) to share, I'm very interested ! Thx.

mr_toad 2y ago

> small models that are as capable as the big hosted model Free lunch?