5 ms·

How would we do that? There's no historic precedent for that. Its fundamentally an information theory thing: what's the max amount of intelligence you can get o

How would we do that? There's no historic precedent for that. Its fundamentally an information theory thing: what's the max amount of intelligence you can get out of 1 KB/MB/GB? There has to be a limit and I'm not convinced that it's far off.

yieldcrv 3mo ago
It’s not just about small models, that’s only one part of evolution Some groups are baking models into silicone, Deepmind has an example, it gets 18,000 tokens/sec on Llama 3.1, not sure about parameter size
- intrasight 3mo ago
  I think this is the future - at least it will be for on-device models. Apple, for instance, will "bake silicon" once a year for their current model, and use that chip in all their devices.
- fooker 3mo ago
  > Some groups are baking models into silicone While some other groups are baking silicone into models :)
  - yieldcrv 3mo ago
    Those are doctors fooker! but yes misplaced e
londons_explore 3mo ago
It doesn't matter. However much intelligence you can squeeze into 1 GB, people will always want more. Standard Def TV was plenty for 50 years. But when more was on offer, everyone went for it, and now you can't even buy a 480p TV.
- t-sauer 3mo ago
  Your example also counters your point: A lot of people do not care about 4k, and even less do care about 8k or above. We reached a point where most people are completely happy with the quality they get.
  - phoghed 3mo ago
    Likely constrained by the fact people are mostly watching low bitrate Netflix streams.
  - lelanthran 3mo ago
    > Your example also counters your point: A lot of people do not care about 4k, and even less do care about 8k or above. We reached a point where most people are completely happy with the quality they get. And when (not if) we get to the 4k (or 8k) equivalent of LLMs, they'll just be baked into hardware and we'll have near instant responses while running locally. There is no future where OpenAI, Anthropic, etc survive with their current business model; at some point we will hit a point where training a new model is done only every 5 years or so, and in that scenario we aren't going to be running models of pricey server GPUs with latencies measured in seconds and full responses measured in minutes. We'll be running locally with sub-millisecond latencies and responses measured in milliseconds. There is no way for any big company to compete with the current business plan of selling inference or subscriptions.
    - andrekandre 3mo ago
      > a point where training a new model is done only every 5 years or so not to take away from your point, but 5 years is a long time to be out of date (esp with programming languages)... or maybe im misinterpreting?
  - seb1204 3mo ago
    I dare say many are watching on handheld devices and not big screens. At least a sizeable percentage I would say. My guess.
- 3mo ago
esterna 3mo ago
Once I start a conversation about Postgres query plans, maybe 95% of the knowledge will be almost certainly not needed, and for an inference provider there will be many more concurrent queries with reasonably large overlap. Maybe future architectures will be able to take advantage of this so not all parameters are needed in memory for almost all instances. If we are talking about all knowledge, then I agree that the compression ratio is very impressive already.
- NaiveBayesian 3mo ago
  Mixture of Experts is already used by pretty much all modern LLMs to address exactly this phenomenon. Hopefully, future models can be trained to be even more aware of external knowledge, accessible through web search / RAG / whatever it will be then, and might not need to internalize much knowledge at all.
  - bbatha 3mo ago
    > future models can be trained to be even more aware of external knowledge Then you need longer contexts, which is proving to a much more stubborn problem than general knowledge compression.
  - tacitusarc 3mo ago
    Note that MoE is a sparsification mechanism and doesn’t actually have to do with expertise or what would commonly be considered areas of expertise.
  - xyzsparetimexyz 3mo ago
    I suppose the right approach would be to run queries like the ones you want your model to be able to do, sort weights/experts by usage frequency and then reareange the weight so that all but the most frequently used can stay on disk. Tricky though, it could be that a e.g. database question uses 90% of the network at some point or another Edit: I guess at the moment this is just having an LRU cache of experts
anon373839 3mo ago
So far, frontier capability keeps getting shrunk to fit consumer level hardware. The lag time being over a year, though, precludes this from being called SOTA by the time it arrives. There must be a limit, I agree, but there have been no signs of approaching it yet. The most recent cohort of small models have shown the biggest leap in capability so far.
lelanthran 3mo ago
> How would we do that? There's no historic precedent for that. Its fundamentally an information theory thing: what's the max amount of intelligence you can get out of 1 KB/MB/GB? There has to be a limit and I'm not convinced that it's far off. Sure, it might be very near using the current approach, but... it might also be might be very far off because we are using the wrong approach. I mean, look at the max amount of intelligence you can get out of a human brain powered by two bananas...
- leoc 3mo ago
  Though even if there are big further breakthroughs to be made here, they won’t necessarily be made soon, and if they aren’t made in the next ~10 years then they won’t really affect the outcome of chip investments made now or in the immediate future.
xscott 3mo ago
All the terms are squishy, but being sloppy about it, I think there's intelligence that needs to be in the model, and knowledge that could live in a database. Right now, models are memorizing a lot of stuff they don't need to. We know how to index lots of information on (comparatively slow) SSDs or across a network. See Karpathy's "Cognitive Core" idea. (I don't have a good link)
- t-3 3mo ago
  I was recently skimming through some 20-30 old books about Information Retrieval (https://en.wikipedia.org/wiki/Information_retrieval https://en.wikipedia.org/wiki/Information_retrieval), and it was immediately quite striking to me that LLMs are the culmination of the integration and development of many ideas in IR that were impractical or theoretical back then. Decoupling the database from the search interface would make the relationship between modern AI and IR even clearer.
petesergeant 3mo ago
> How would we do that? There's no historic precedent for that. Its fundamentally an information theory thing Maybe? Human science history is absolutely littered with examples of things that were “constrained” by a fundamental law … until they weren’t, because we’d misunderstood or misapplied the law.
odyssey7 3mo ago
Historical precedent: Quickselect, or any other discovery of a surprising algorithmic speed-up. LLMs haven’t been on the scene for very long. There’s still lots of room for efficiency discoveries. Plus, I keep hearing quantum is going to be a big deal in the next few years.
- _glass 3mo ago
  Super interesting, when Quantum computing would enable really large models, or much larger contexts. But QRAM is even more behind than pure fault-tolerant gates.
EarthMephit 3mo ago
I haven't tried it, but colibri is meant to allow running GLM 5.2 the 744B model in 32GB by using prefetches and streaming from fast SSDs. https://github.com/JustVugg/colibri https://github.com/JustVugg/colibri
- kcb 3mo ago
  Terrible performance on $20,000 worth of GPUs.
bebe8393jrir 3mo ago
[flagged]
fooker 3mo ago
I'm claiming that the max amount of intelligence you get out of a 100GB model is unlikely to be that much lower than what you can get out of a 1TB model.
- xyzsparetimexyz 3mo ago
  That depends on whether you can separate intelligence from memorization. I think we can agree that theres more than 1 TB of memorizable information in the world, even when highly compressed.
  - fooker 3mo ago
    Sure, agreed. The question then is, how much memorization is really needed for intelligence if you can query structured information?
deleted 3mo ago
[deleted]
drdrek 3mo ago
But why do you assume that you need unlimited amount of intelligence to perform work? If you can cut out just the parts that are required for doing SDR work or Development, etc you can get performance beyond what we have with generalized intelligence. Even on the physical aspect think about it, if you need to do less networking between GPUs and can do more in each single chip, you can use more "tokens" for reasoning without increasing time/cost.

yieldcrv 3mo ago

It’s not just about small models, that’s only one part of evolution Some groups are baking models into silicone, Deepmind has an example, it gets 18,000 tokens/sec on Llama 3.1, not sure about parameter size

intrasight 3mo ago
I think this is the future - at least it will be for on-device models. Apple, for instance, will "bake silicon" once a year for their current model, and use that chip in all their devices.
fooker 3mo ago
> Some groups are baking models into silicone While some other groups are baking silicone into models :)
- yieldcrv 3mo ago
  Those are doctors fooker! but yes misplaced e

londons_explore 3mo ago

It doesn't matter. However much intelligence you can squeeze into 1 GB, people will always want more. Standard Def TV was plenty for 50 years. But when more was on offer, everyone went for it, and now you can't even buy a 480p TV.

t-sauer 3mo ago
Your example also counters your point: A lot of people do not care about 4k, and even less do care about 8k or above. We reached a point where most people are completely happy with the quality they get.
- phoghed 3mo ago
  Likely constrained by the fact people are mostly watching low bitrate Netflix streams.
- lelanthran 3mo ago
  > Your example also counters your point: A lot of people do not care about 4k, and even less do care about 8k or above. We reached a point where most people are completely happy with the quality they get. And when (not if) we get to the 4k (or 8k) equivalent of LLMs, they'll just be baked into hardware and we'll have near instant responses while running locally. There is no future where OpenAI, Anthropic, etc survive with their current business model; at some point we will hit a point where training a new model is done only every 5 years or so, and in that scenario we aren't going to be running models of pricey server GPUs with latencies measured in seconds and full responses measured in minutes. We'll be running locally with sub-millisecond latencies and responses measured in milliseconds. There is no way for any big company to compete with the current business plan of selling inference or subscriptions.
  - andrekandre 3mo ago
    > a point where training a new model is done only every 5 years or so not to take away from your point, but 5 years is a long time to be out of date (esp with programming languages)... or maybe im misinterpreting?
- seb1204 3mo ago
  I dare say many are watching on handheld devices and not big screens. At least a sizeable percentage I would say. My guess.
3mo ago

esterna 3mo ago

Once I start a conversation about Postgres query plans, maybe 95% of the knowledge will be almost certainly not needed, and for an inference provider there will be many more concurrent queries with reasonably large overlap. Maybe future architectures will be able to take advantage of this so not all parameters are needed in memory for almost all instances. If we are talking about all knowledge, then I agree that the compression ratio is very impressive already.

NaiveBayesian 3mo ago
Mixture of Experts is already used by pretty much all modern LLMs to address exactly this phenomenon. Hopefully, future models can be trained to be even more aware of external knowledge, accessible through web search / RAG / whatever it will be then, and might not need to internalize much knowledge at all.
- bbatha 3mo ago
  > future models can be trained to be even more aware of external knowledge Then you need longer contexts, which is proving to a much more stubborn problem than general knowledge compression.
- tacitusarc 3mo ago
  Note that MoE is a sparsification mechanism and doesn’t actually have to do with expertise or what would commonly be considered areas of expertise.
- xyzsparetimexyz 3mo ago
  I suppose the right approach would be to run queries like the ones you want your model to be able to do, sort weights/experts by usage frequency and then reareange the weight so that all but the most frequently used can stay on disk. Tricky though, it could be that a e.g. database question uses 90% of the network at some point or another Edit: I guess at the moment this is just having an LRU cache of experts

anon373839 3mo ago

So far, frontier capability keeps getting shrunk to fit consumer level hardware. The lag time being over a year, though, precludes this from being called SOTA by the time it arrives. There must be a limit, I agree, but there have been no signs of approaching it yet. The most recent cohort of small models have shown the biggest leap in capability so far.

lelanthran 3mo ago

> How would we do that? There's no historic precedent for that. Its fundamentally an information theory thing: what's the max amount of intelligence you can get out of 1 KB/MB/GB? There has to be a limit and I'm not convinced that it's far off. Sure, it might be very near using the current approach, but... it might also be might be very far off because we are using the wrong approach. I mean, look at the max amount of intelligence you can get out of a human brain powered by two bananas...

leoc 3mo ago
Though even if there are big further breakthroughs to be made here, they won’t necessarily be made soon, and if they aren’t made in the next ~10 years then they won’t really affect the outcome of chip investments made now or in the immediate future.

xscott 3mo ago

All the terms are squishy, but being sloppy about it, I think there's intelligence that needs to be in the model, and knowledge that could live in a database. Right now, models are memorizing a lot of stuff they don't need to. We know how to index lots of information on (comparatively slow) SSDs or across a network. See Karpathy's "Cognitive Core" idea. (I don't have a good link)

t-3 3mo ago
I was recently skimming through some 20-30 old books about Information Retrieval (https://en.wikipedia.org/wiki/Information_retrieval https://en.wikipedia.org/wiki/Information_retrieval), and it was immediately quite striking to me that LLMs are the culmination of the integration and development of many ideas in IR that were impractical or theoretical back then. Decoupling the database from the search interface would make the relationship between modern AI and IR even clearer.

petesergeant 3mo ago

> How would we do that? There's no historic precedent for that. Its fundamentally an information theory thing Maybe? Human science history is absolutely littered with examples of things that were “constrained” by a fundamental law … until they weren’t, because we’d misunderstood or misapplied the law.

odyssey7 3mo ago

Historical precedent: Quickselect, or any other discovery of a surprising algorithmic speed-up. LLMs haven’t been on the scene for very long. There’s still lots of room for efficiency discoveries. Plus, I keep hearing quantum is going to be a big deal in the next few years.

_glass 3mo ago
Super interesting, when Quantum computing would enable really large models, or much larger contexts. But QRAM is even more behind than pure fault-tolerant gates.

EarthMephit 3mo ago

I haven't tried it, but colibri is meant to allow running GLM 5.2 the 744B model in 32GB by using prefetches and streaming from fast SSDs. https://github.com/JustVugg/colibri https://github.com/JustVugg/colibri

kcb 3mo ago
Terrible performance on $20,000 worth of GPUs.

bebe8393jrir 3mo ago

[flagged]

fooker 3mo ago

I'm claiming that the max amount of intelligence you get out of a 100GB model is unlikely to be that much lower than what you can get out of a 1TB model.

xyzsparetimexyz 3mo ago
That depends on whether you can separate intelligence from memorization. I think we can agree that theres more than 1 TB of memorizable information in the world, even when highly compressed.
- fooker 3mo ago
  Sure, agreed. The question then is, how much memorization is really needed for intelligence if you can query structured information?

deleted 3mo ago

[deleted]

drdrek 3mo ago

But why do you assume that you need unlimited amount of intelligence to perform work? If you can cut out just the parts that are required for doing SDR work or Development, etc you can get performance beyond what we have with generalized intelligence. Even on the physical aspect think about it, if you need to do less networking between GPUs and can do more in each single chip, you can use more "tokens" for reasoning without increasing time/cost.