4 ms·

> Hyperscalers don't run computing at some multiple more efficient than on prem. I'd disagree here. I see two avenues for an efficiency multiple, albeit a sin

> Hyperscalers don't run computing at some multiple more efficient than on prem. I'd disagree here. I see two avenues for an efficiency multiple, albeit a single-digit multiple: * Client aggregation allows a hyperscaler to average out demand spikes from uncorrelated clients, reducing the peak:average demand ratio and allowing better budgeting of compute. * Dynamic batching allows typical requests to run in batches of more-than-1 and/or overlap, offering better internal compute utilization ratios (e.g. interleaving output and input streams). The small limit of on-device LLMs will run with batch sizes of one with strong memory bandwidth bottlenecks. For an example of these factors in action, see the API cost differential between batch, standard, and 'fast' processing. OpenAI prices these tiers at a 1:2:4 ratio.

genxy 1mo ago
A hyperscaler can pack compute and that is it. It doesn't get you some multiple higher in efficiency. And it is a net negative when oversubscribed, which due to MBAs ruining all the clouds is how they operate. Customers pay more for less when using the cloud. Go get a job a hyperscaler, they want to smoke what you are smoking. I am not talking about diurnal cloud workloads, I am talking about the native efficiencies of hyperscalers vs on-prem. They have no magic and they all think they are going to make up their business overheads in exorbitant saas pricing.
- philipallstar 1mo ago
  Surely they can: - buy in bulk, for lower prices and access to better hardware through big contracts - build in bulk (i.e. spread out software improvements over a lot of data centres and customers) - offer additional services such as edge caching and multi-region data redundancy That doesn't mean that hyperscalers don't also do things wrong, but it seems very odd to pretend there's nothing to them.
  - genxy 1mo ago
    Yes, they can do all of these things. But then the margins are eaten up in the process, hyperscalers could be half the price of on-prem, but they are more. They have no magic, or the magic they do have they are wasting (which is good for the people). You can beat a hyperscaler on price at a single rack.

genxy 1mo ago

A hyperscaler can pack compute and that is it. It doesn't get you some multiple higher in efficiency. And it is a net negative when oversubscribed, which due to MBAs ruining all the clouds is how they operate. Customers pay more for less when using the cloud. Go get a job a hyperscaler, they want to smoke what you are smoking. I am not talking about diurnal cloud workloads, I am talking about the native efficiencies of hyperscalers vs on-prem. They have no magic and they all think they are going to make up their business overheads in exorbitant saas pricing.

philipallstar 1mo ago
Surely they can: - buy in bulk, for lower prices and access to better hardware through big contracts - build in bulk (i.e. spread out software improvements over a lot of data centres and customers) - offer additional services such as edge caching and multi-region data redundancy That doesn't mean that hyperscalers don't also do things wrong, but it seems very odd to pretend there's nothing to them.
- genxy 1mo ago
  Yes, they can do all of these things. But then the margins are eaten up in the process, hyperscalers could be half the price of on-prem, but they are more. They have no magic, or the magic they do have they are wasting (which is good for the people). You can beat a hyperscaler on price at a single rack.