4 ms·

> China is pursuing these because they cannot compete on the frontier. ? Claude, ChatGPT, etc are heinously expensive for tiny benefits lmao. Local + efficient

> China is pursuing these because they cannot compete on the frontier. ? Claude, ChatGPT, etc are heinously expensive for tiny benefits lmao. Local + efficient is clearly the future

ToucanLoucan 6mo ago
AI boosters cling to this notion because it's the only way the massive data center buildouts make any sense at all. I guess you could say the US is winning the frontier AI race. Okay. I'm never going to grant a cloud service access to all the contents of my hard drive, that's just never going to happen, so if you expect me and a lot of people like me who feel similarly to get on this train, you better have a local, lightweight model too or we're not even having a discussion, the answer is just no.
- Our_Benefactors 6mo ago
  The thing is, frontier model providers don’t take your feelings into account even a little bit. It’s totally irrelevant to the discussion about the service they can provide, because that service is predicated on access to high power GPU slices that local models can’t touch. Those providers won’t be in an existential crisis because some people choose the privacy route, it’s a cost of doing business.
  - ToucanLoucan 6mo ago
    Right but that service being sold is predicated on products being sold to users, yes? Or are we still pretending that the hyperscalers can just pass the same $20 billion between themselves and that's going to be a growth industry forever?
    - ElevenLathe 6mo ago
      I suppose its possible that all the value to pay back the datacenter construction can be squeezed out of enterprise contracts where your employer can assent on the privacy questions, probably with some kind of complicated contract and insurance regime regulating things. Even if so, if China is coming behind 6 months later selling laptops with hyper-efficient local models that are 80% as good as "frontier" ones, I imagine they'll get the consumer business AND a fair share of the enterprise business as IT managers look at their options during the next refresh cycle. Given economies of scale, I think it's ultimately inevitable that the enterprise more-or-less follows the consumer on this, and the consumer is going to prefer local models. There's no ongoing cost after the initial purchase, and your data at least nominally stays within your control.
      - ToucanLoucan 6mo ago
        I'm inclined to agree. The business world itself, and frankly you could make this argument about the entire AI industry as it were, runs on "fine" and 80% capable can probably get you there. And it's arguably even better for the hyperscalers since then, ongoing costs of AI users are basically nil. You can still have your massive datacenters and just keep them for tasks complex enough that they're actually worth spooling up. Like I don't need an H100 or a dozen to summarize a PDF. And that's most of what I use AI for.
    - Our_Benefactors 6mo ago
      If we are betting on which is an easier sale, $20-100 a month w/tech support included vs $5k-10k and a requirement for moderate technical ability, I would invest in the former not the latter being the proposition that drives the conversation about AI use.
ericmay 6mo ago
> ? Claude, ChatGPT, etc are heinously expensive for tiny benefits lmao. Local + efficient is clearly the future Corporate America is where the money is, and corporate America will dictate what products are successful by virtue of spend. Individuals aren't going to be paying $100s or $1000s/month en masse for these models but businesses will be. Being local and efficient isn't that important at this stage but even so as American companies continue to scale and invest they'll be able to make those models more local and efficient if the market wants it. Sort of like how you had a big, giant desktop computer and now you've got a super computer in your phone which is in your pocket. Going straight to "local and efficient" means going straight to being behind because at some point, perhaps now even, the local and efficient model won't be able to keep up. For some reason people think that they somehow know something that Google or Nvidia or whoever, with hundreds of billions of dollars of real money at stake don't already know and it's both amusing and bizarre to see this play out again and again in off-hand comments like "lol tiny benefits". You buy an iPhone even though the cheap-o Wal-Mart Android phone for $100 "does the same thing". Except that in this case the Android phone just puts you out of business while those spending big money for "tiny benefits" beat you in the market.
- ForHackernews 6mo ago
  > You buy an iPhone even though the cheap-o Wal-Mart Android phone for $100 "does the same thing". People buy iPhones because of status signalling and network effects, neither of which appears to apply to AI model choice. LLMs are already rapidly on the way to being interchangeable commodities.
  - ericmay 6mo ago
    No they don't, it's not 2008. Anybody off the street can get an iPhone or a free iPhone with a mobile plan. They're commodity products. Even homeless people have them. To the extent LLMs are commodity products you're right (so far), but that is limited to the main model providers, such as ChatGPT, Claude, Gemini, &c. with interoperability on cloud platform providers and other technology providers like an Apple offering you a choice of LLM with Siri or something. If you want to suggest that some other model is in the same bucket as those primary 3, it goes back to the crappy, cheap phone analogy which is accurate. Yea you can make calls with it, but you make calls better with an iPhone.
visarga 6mo ago
> ? Claude, ChatGPT, etc are heinously expensive for tiny benefits lmao Unfortunately local inference is inefficient, 100s of times more inefficient than cloud. When you answer one request at a time you still have to fetch all active weights into compute units, once every token. When you run a batch of 300, you load it once and compute 300 at a time. Compared to cloud, local inference is less flexible. You can't scale up 5x or 20x, can't have spikes, and pay for it no matter if you use it or not. But usage factor is very low, like 5%. And to run a decent model your system costs $2000 or more.

> China is pursuing these because they cannot compete on the frontier. ? Claude, ChatGPT, etc are heinously expensive for tiny benefits lmao. Local + efficient is clearly the future

ToucanLoucan 6mo ago

AI boosters cling to this notion because it's the only way the massive data center buildouts make any sense at all. I guess you could say the US is winning the frontier AI race. Okay. I'm never going to grant a cloud service access to all the contents of my hard drive, that's just never going to happen, so if you expect me and a lot of people like me who feel similarly to get on this train, you better have a local, lightweight model too or we're not even having a discussion, the answer is just no.

Our_Benefactors 6mo ago
The thing is, frontier model providers don’t take your feelings into account even a little bit. It’s totally irrelevant to the discussion about the service they can provide, because that service is predicated on access to high power GPU slices that local models can’t touch. Those providers won’t be in an existential crisis because some people choose the privacy route, it’s a cost of doing business.
- ToucanLoucan 6mo ago
  Right but that service being sold is predicated on products being sold to users, yes? Or are we still pretending that the hyperscalers can just pass the same $20 billion between themselves and that's going to be a growth industry forever?
  - ElevenLathe 6mo ago
    I suppose its possible that all the value to pay back the datacenter construction can be squeezed out of enterprise contracts where your employer can assent on the privacy questions, probably with some kind of complicated contract and insurance regime regulating things. Even if so, if China is coming behind 6 months later selling laptops with hyper-efficient local models that are 80% as good as "frontier" ones, I imagine they'll get the consumer business AND a fair share of the enterprise business as IT managers look at their options during the next refresh cycle. Given economies of scale, I think it's ultimately inevitable that the enterprise more-or-less follows the consumer on this, and the consumer is going to prefer local models. There's no ongoing cost after the initial purchase, and your data at least nominally stays within your control.
    - ToucanLoucan 6mo ago
      I'm inclined to agree. The business world itself, and frankly you could make this argument about the entire AI industry as it were, runs on "fine" and 80% capable can probably get you there. And it's arguably even better for the hyperscalers since then, ongoing costs of AI users are basically nil. You can still have your massive datacenters and just keep them for tasks complex enough that they're actually worth spooling up. Like I don't need an H100 or a dozen to summarize a PDF. And that's most of what I use AI for.
  - Our_Benefactors 6mo ago
    If we are betting on which is an easier sale, $20-100 a month w/tech support included vs $5k-10k and a requirement for moderate technical ability, I would invest in the former not the latter being the proposition that drives the conversation about AI use.

ericmay 6mo ago

> ? Claude, ChatGPT, etc are heinously expensive for tiny benefits lmao. Local + efficient is clearly the future Corporate America is where the money is, and corporate America will dictate what products are successful by virtue of spend. Individuals aren't going to be paying $100s or $1000s/month en masse for these models but businesses will be. Being local and efficient isn't that important at this stage but even so as American companies continue to scale and invest they'll be able to make those models more local and efficient if the market wants it. Sort of like how you had a big, giant desktop computer and now you've got a super computer in your phone which is in your pocket. Going straight to "local and efficient" means going straight to being behind because at some point, perhaps now even, the local and efficient model won't be able to keep up. For some reason people think that they somehow know something that Google or Nvidia or whoever, with hundreds of billions of dollars of real money at stake don't already know and it's both amusing and bizarre to see this play out again and again in off-hand comments like "lol tiny benefits". You buy an iPhone even though the cheap-o Wal-Mart Android phone for $100 "does the same thing". Except that in this case the Android phone just puts you out of business while those spending big money for "tiny benefits" beat you in the market.

ForHackernews 6mo ago
> You buy an iPhone even though the cheap-o Wal-Mart Android phone for $100 "does the same thing". People buy iPhones because of status signalling and network effects, neither of which appears to apply to AI model choice. LLMs are already rapidly on the way to being interchangeable commodities.
- ericmay 6mo ago
  No they don't, it's not 2008. Anybody off the street can get an iPhone or a free iPhone with a mobile plan. They're commodity products. Even homeless people have them. To the extent LLMs are commodity products you're right (so far), but that is limited to the main model providers, such as ChatGPT, Claude, Gemini, &c. with interoperability on cloud platform providers and other technology providers like an Apple offering you a choice of LLM with Siri or something. If you want to suggest that some other model is in the same bucket as those primary 3, it goes back to the crappy, cheap phone analogy which is accurate. Yea you can make calls with it, but you make calls better with an iPhone.

visarga 6mo ago

> ? Claude, ChatGPT, etc are heinously expensive for tiny benefits lmao Unfortunately local inference is inefficient, 100s of times more inefficient than cloud. When you answer one request at a time you still have to fetch all active weights into compute units, once every token. When you run a batch of 300, you load it once and compute 300 at a time. Compared to cloud, local inference is less flexible. You can't scale up 5x or 20x, can't have spikes, and pay for it no matter if you use it or not. But usage factor is very low, like 5%. And to run a decent model your system costs $2000 or more.