6 ms·

This logic seems mad. If people only need SLMs then hyperscalers can also centrally host higher-efficiency models, and still gain efficiencies of scale and conv

This logic seems mad. If people only need SLMs then hyperscalers can also centrally host higher-efficiency models, and still gain efficiencies of scale and convenience over hosting locally.

embedding-shape 2mo ago
You have to remember that articles like these are written for finance people who don't understand the underlying technology, by finance people who don't understand the underlying technology. In this case, the author is a "CFA Institute Enterprising Investor", previously a CIO and basically their entire life been "money, money & money", so hardly surprising they're pulling a lot of assumptions based on what they read. Read the paper the author talks about yourself instead (https://arxiv.org/abs/2511.07885 https://arxiv.org/abs/2511.07885), and also, contrary to what the author says in the article, do not do investments based on single papers made from academic studies, regardless of how much money this guy tells you you can make.
amelius 2mo ago
The logic seems mad to me because SLMs can simply not hold as much information as an LLM. Maybe if you combine an SLM with a database (as a tool) then it could work, but someone should first prove that.
- fph 2mo ago
  But do you really need a model that has the complete Duran Duran discography memorized and preloaded in RAM at all time?
  - amelius 2mo ago
    That's a different question. Probably not. But: 1. Training a large model with lots of information, then stripping the "useless" information from that model to obtain a small model => nobody has shown this. 2. Training a small model, letting it use a database tool so it scores the same as a large model without database => nobody has shown this.
    - js8 2mo ago
      I partly agree. Theory (of computation) shows it must be possible, however nobody has produced the small model (well, depends who you ask, what is small, article disputes that) and the database yet. To go very small (thousands of rules) so that the reasoner can be understood by humans and proven sound - might be computationally quite difficult.
    - slfnflctd 2mo ago
      This really elegantly sums up two of the most major areas of research in the field right now. I expect people are going to hammer at both problems for a while. It will be interesting to see how much useful stuff is built as a result.
  - embedding-shape 2mo ago
    I mean maybe yes? The hypothesis from the early GPT days was (and in a small way still remains): "If we just chuck more data into the training, does it get better at X, even if the data was seemingly unrelated to X?", and the workings of LLMs seem to kind be pointing in that direction, although with some ceiling. But seemingly models good at programming for example, would get worse at programming if you removed everything not-programming. Train a model solely on syntax, and it'll be worse than a general purpose LLM on syntax, in general at least.
    - js8 2mo ago
      "But seemingly models good at programming for example, would get worse at programming if you removed everything not-programming. Train a model solely on syntax, and it'll be worse than a general purpose LLM on syntax, in general at least." That might be because emergence of capabilities to reason about programs requires abstractions (such as fuzzy and modal logic) that are rarely present in software sources. That doesn't mean the reasoning model itself has to be large; neither does it have to emerge from the ML training on large language corpus, we might construct it by different means.
  - medwards666 2mo ago
    But what if I _really like_ Duran Duran???
    - sandytoast 2mo ago
      Then you’re hungry like the wolf
  - rotis 2mo ago
    Sure. I don't give a damn about them. But Phil Collins man. I love reviewing his body of work before I get worked up. Obvously we cannot cut him, because I need him. So how do you decide what to leave out?
- js8 2mo ago
  The cost you pay is in additional reasoning the SLMs have to do. As I write elsewhere, LLM "remembers" that "Socrates is mortal", or other commonly useful deduction. SLM might need to derive it first by reasoning from the DB, which slows it down. (Or worse, it might miss the correct reasoning because it's just too much side quests to follow.) But the advantage is flexibility.
regularfry 2mo ago
The valuations of the hyperscalars won't sustain just being more efficient than something you can run locally. There's a market there, but it's for margin on a commodity. They're priced for oligopoly on unique, premium products.
- phoghed 2mo ago
  I very much don’t want to run it locally. I want the same one running somewhere else that I can interact with from all my devices. Look at something like Grok Bot. Nobody is going to run this locally. You can already self host almost anything, yet most people and businesses don’t.
  - IsTom 2mo ago
    Yes, but that's not a 10T business.
    - bryanlarsen 2mo ago
      But it is quite likely a 1T market. That won't support the stock valuations of OpenAI, Anthropic or Nvidia, but it will still mean massive data center build out and little relief to RAM prices.
  - esseph 2mo ago
    > I very much don’t want to run it locally. I want the same one running somewhere else that I can interact with from all my devices. I run models on my desktop and access them from a phone app on the go. Wireguard tunnel. Responds fast and lets me kick off tasks or workflows via text or voice.
    - phoghed 2mo ago
      Exactly the type of thing I’d pay someone not to deal with. It’s like media streaming vs piracy. Even though I could get a nice local setup with plex and have higher quality streams with free pirated br rips, I just don’t want to deal with it. Maybe I could use codex or whatever to set that all up nowadays, but it was such a pain doing the initial setup and getting it all running and finding sources of media yadda yadda like a decade ago when I last tried lol
readthenotes1 2mo ago
That sounds a little bit like the guy who said that we don't need a computer in the home. There may be a great deal of advantage to be able to run a small language model on something I have in my hand, disconnected. Although mainframes have their use, the pendulum of centralized to distributed has gone back and forth and there are benefits to be gleaned from either model, sometimes at the same time.
genxy 2mo ago
Hyperscalers don't run computing at some multiple more efficient than on prem. The only way hyperscalers can compete is if they own the sand to cycles supply chain and ensure that raw compute is out priced in the market (ram,flash,compute). Ram and flash were an easy target because they are a commodity in name only.
- Majromax 2mo ago
  > Hyperscalers don't run computing at some multiple more efficient than on prem. I'd disagree here. I see two avenues for an efficiency multiple, albeit a single-digit multiple: * Client aggregation allows a hyperscaler to average out demand spikes from uncorrelated clients, reducing the peak:average demand ratio and allowing better budgeting of compute. * Dynamic batching allows typical requests to run in batches of more-than-1 and/or overlap, offering better internal compute utilization ratios (e.g. interleaving output and input streams). The small limit of on-device LLMs will run with batch sizes of one with strong memory bandwidth bottlenecks. For an example of these factors in action, see the API cost differential between batch, standard, and 'fast' processing. OpenAI prices these tiers at a 1:2:4 ratio.
  - genxy 2mo ago
    A hyperscaler can pack compute and that is it. It doesn't get you some multiple higher in efficiency. And it is a net negative when oversubscribed, which due to MBAs ruining all the clouds is how they operate. Customers pay more for less when using the cloud. Go get a job a hyperscaler, they want to smoke what you are smoking. I am not talking about diurnal cloud workloads, I am talking about the native efficiencies of hyperscalers vs on-prem. They have no magic and they all think they are going to make up their business overheads in exorbitant saas pricing.
    - philipallstar 2mo ago
      Surely they can: - buy in bulk, for lower prices and access to better hardware through big contracts - build in bulk (i.e. spread out software improvements over a lot of data centres and customers) - offer additional services such as edge caching and multi-region data redundancy That doesn't mean that hyperscalers don't also do things wrong, but it seems very odd to pretend there's nothing to them.
ForHackernews 2mo ago
You mean like how Western Digital is only a 168 billion dollar business, but Dropbox is a $7.4B behemoth... er, wait?
jjav 2mo ago
> efficiencies of scale and convenience over hosting locally The privacy cost of sending everything to a third party is huge. Running locally fully resolves that, so it is the obvious choice if only cost can be managed.
mpweiher 2mo ago
The convenience of having the thing in your pocket or on your local machine without having to worry about a network connection, sign up, payment, privacy issues etc. cannot be beat. The efficiency doesn't matter as long as it's good enough, because you already have the computer.

This logic seems mad. If people only need SLMs then hyperscalers can also centrally host higher-efficiency models, and still gain efficiencies of scale and convenience over hosting locally.

embedding-shape 2mo ago

You have to remember that articles like these are written for finance people who don't understand the underlying technology, by finance people who don't understand the underlying technology. In this case, the author is a "CFA Institute Enterprising Investor", previously a CIO and basically their entire life been "money, money & money", so hardly surprising they're pulling a lot of assumptions based on what they read. Read the paper the author talks about yourself instead (https://arxiv.org/abs/2511.07885 https://arxiv.org/abs/2511.07885), and also, contrary to what the author says in the article, do not do investments based on single papers made from academic studies, regardless of how much money this guy tells you you can make.

amelius 2mo ago

The logic seems mad to me because SLMs can simply not hold as much information as an LLM. Maybe if you combine an SLM with a database (as a tool) then it could work, but someone should first prove that.

fph 2mo ago
But do you really need a model that has the complete Duran Duran discography memorized and preloaded in RAM at all time?
- amelius 2mo ago
  That's a different question. Probably not. But: 1. Training a large model with lots of information, then stripping the "useless" information from that model to obtain a small model => nobody has shown this. 2. Training a small model, letting it use a database tool so it scores the same as a large model without database => nobody has shown this.
  - js8 2mo ago
    I partly agree. Theory (of computation) shows it must be possible, however nobody has produced the small model (well, depends who you ask, what is small, article disputes that) and the database yet. To go very small (thousands of rules) so that the reasoner can be understood by humans and proven sound - might be computationally quite difficult.
  - slfnflctd 2mo ago
    This really elegantly sums up two of the most major areas of research in the field right now. I expect people are going to hammer at both problems for a while. It will be interesting to see how much useful stuff is built as a result.
- embedding-shape 2mo ago
  I mean maybe yes? The hypothesis from the early GPT days was (and in a small way still remains): "If we just chuck more data into the training, does it get better at X, even if the data was seemingly unrelated to X?", and the workings of LLMs seem to kind be pointing in that direction, although with some ceiling. But seemingly models good at programming for example, would get worse at programming if you removed everything not-programming. Train a model solely on syntax, and it'll be worse than a general purpose LLM on syntax, in general at least.
  - js8 2mo ago
    "But seemingly models good at programming for example, would get worse at programming if you removed everything not-programming. Train a model solely on syntax, and it'll be worse than a general purpose LLM on syntax, in general at least." That might be because emergence of capabilities to reason about programs requires abstractions (such as fuzzy and modal logic) that are rarely present in software sources. That doesn't mean the reasoning model itself has to be large; neither does it have to emerge from the ML training on large language corpus, we might construct it by different means.
- medwards666 2mo ago
  But what if I _really like_ Duran Duran???
  - sandytoast 2mo ago
    Then you’re hungry like the wolf
- rotis 2mo ago
  Sure. I don't give a damn about them. But Phil Collins man. I love reviewing his body of work before I get worked up. Obvously we cannot cut him, because I need him. So how do you decide what to leave out?
js8 2mo ago
The cost you pay is in additional reasoning the SLMs have to do. As I write elsewhere, LLM "remembers" that "Socrates is mortal", or other commonly useful deduction. SLM might need to derive it first by reasoning from the DB, which slows it down. (Or worse, it might miss the correct reasoning because it's just too much side quests to follow.) But the advantage is flexibility.

regularfry 2mo ago

The valuations of the hyperscalars won't sustain just being more efficient than something you can run locally. There's a market there, but it's for margin on a commodity. They're priced for oligopoly on unique, premium products.

phoghed 2mo ago
I very much don’t want to run it locally. I want the same one running somewhere else that I can interact with from all my devices. Look at something like Grok Bot. Nobody is going to run this locally. You can already self host almost anything, yet most people and businesses don’t.
- IsTom 2mo ago
  Yes, but that's not a 10T business.
  - bryanlarsen 2mo ago
    But it is quite likely a 1T market. That won't support the stock valuations of OpenAI, Anthropic or Nvidia, but it will still mean massive data center build out and little relief to RAM prices.
- esseph 2mo ago
  > I very much don’t want to run it locally. I want the same one running somewhere else that I can interact with from all my devices. I run models on my desktop and access them from a phone app on the go. Wireguard tunnel. Responds fast and lets me kick off tasks or workflows via text or voice.
  - phoghed 2mo ago
    Exactly the type of thing I’d pay someone not to deal with. It’s like media streaming vs piracy. Even though I could get a nice local setup with plex and have higher quality streams with free pirated br rips, I just don’t want to deal with it. Maybe I could use codex or whatever to set that all up nowadays, but it was such a pain doing the initial setup and getting it all running and finding sources of media yadda yadda like a decade ago when I last tried lol

readthenotes1 2mo ago

That sounds a little bit like the guy who said that we don't need a computer in the home. There may be a great deal of advantage to be able to run a small language model on something I have in my hand, disconnected. Although mainframes have their use, the pendulum of centralized to distributed has gone back and forth and there are benefits to be gleaned from either model, sometimes at the same time.

genxy 2mo ago

Hyperscalers don't run computing at some multiple more efficient than on prem. The only way hyperscalers can compete is if they own the sand to cycles supply chain and ensure that raw compute is out priced in the market (ram,flash,compute). Ram and flash were an easy target because they are a commodity in name only.

Majromax 2mo ago
> Hyperscalers don't run computing at some multiple more efficient than on prem. I'd disagree here. I see two avenues for an efficiency multiple, albeit a single-digit multiple: * Client aggregation allows a hyperscaler to average out demand spikes from uncorrelated clients, reducing the peak:average demand ratio and allowing better budgeting of compute. * Dynamic batching allows typical requests to run in batches of more-than-1 and/or overlap, offering better internal compute utilization ratios (e.g. interleaving output and input streams). The small limit of on-device LLMs will run with batch sizes of one with strong memory bandwidth bottlenecks. For an example of these factors in action, see the API cost differential between batch, standard, and 'fast' processing. OpenAI prices these tiers at a 1:2:4 ratio.
- genxy 2mo ago
  A hyperscaler can pack compute and that is it. It doesn't get you some multiple higher in efficiency. And it is a net negative when oversubscribed, which due to MBAs ruining all the clouds is how they operate. Customers pay more for less when using the cloud. Go get a job a hyperscaler, they want to smoke what you are smoking. I am not talking about diurnal cloud workloads, I am talking about the native efficiencies of hyperscalers vs on-prem. They have no magic and they all think they are going to make up their business overheads in exorbitant saas pricing.
  - philipallstar 2mo ago
    Surely they can: - buy in bulk, for lower prices and access to better hardware through big contracts - build in bulk (i.e. spread out software improvements over a lot of data centres and customers) - offer additional services such as edge caching and multi-region data redundancy That doesn't mean that hyperscalers don't also do things wrong, but it seems very odd to pretend there's nothing to them.

ForHackernews 2mo ago

You mean like how Western Digital is only a 168 billion dollar business, but Dropbox is a $7.4B behemoth... er, wait?

jjav 2mo ago

> efficiencies of scale and convenience over hosting locally The privacy cost of sending everything to a third party is huge. Running locally fully resolves that, so it is the obvious choice if only cost can be managed.

mpweiher 2mo ago

The convenience of having the thing in your pocket or on your local machine without having to worry about a network connection, sign up, payment, privacy issues etc. cannot be beat. The efficiency doesn't matter as long as it's good enough, because you already have the computer.