5 ms·

In particular: "There is, however, one enormous difference that I didn’t think about: You can’t build a cloud vendor overnight. Azure doesn’t have to worry abo

In particular: "There is, however, one enormous difference that I didn’t think about: You can’t build a cloud vendor overnight. Azure doesn’t have to worry about a few executives leaving and building a worldwide network of data centers in 18 months." This isn't true at all. There are like 8 of these companies stood up in the last three or four years fueled by massive investment of sovereign funds - mostly the saudi, dubai, northern europe, etc. oil-derived funds - all spending billions of dollars doing exactly that and getting something done. The real problem is the ROI on AI spending is.. pretty much zero. The commonly asserted use cases are the following: Chatbots Developer tools RAG/search Not a one of these is going to generate $10 of additional revenue per sollar spent, nor likely even $2. Optimizing your customer services representatives from 8 conversations at once to an average of 12 or 16 is going to save you a whopping $2 per hour per CSR. It just isn't huge money. And RAG has many, many issues with document permissions that make the current approaches bad for enterprises - where the money is - who as a group haven't spent much of anything to even make basic search work.

sushid 2y ago
> And RAG has many, many issues with document permissions Why can't these providers access all documents and when answers are prompted, self-censor if the reply has references to documents that the end users do not have access permissions? In fact, I'm pretty sure that's how existing RAGaaS providers are handling document/file permissions.
- portaouflop 2y ago
  RAGaas, didn’t know I needed that in my life
- Terr_ 2y ago
  Would the texts of those documents be part of the LLM training data? If so, there's an reliable way to keep a determined user from fetching stuff back out.
- fwip 2y ago
  That's a pretty quick way to expose what information you're trying to keep secret.
bhouston 2y ago
> The real problem is the ROI on AI spending is.. pretty much zero. Companies in user acquisition/growth mode tend to have low internal ROI, but remember both Facebook and Google has the same issue -- then they introduced ads and all was well with their finances. Similar things will happen here.
- HillRat 2y ago
  LLMs are a utility, not a platform, and utility markets exert a downward pressure on pricing. Moreover, it's not obvious that -- once trained models hit the wild -- any one actor has or can develop significant competitive moats that would allow them to escape that price pressure. Beyond that, the digital marginal cost of services needs to be significantly reduced to keep these companies in business, but more efficient models leads to pushing inference out to end-user compute, which hollows out their business model (I assume that Apple dropping out of the OpenAI investment round was partially due to the wildly optimistic valuations involved, partially because they're betting on being able to optimize runtime costs down to iPhone levels). Basically, I'd argue that LLMs look less like a Web 2.0 social media opportunity and more like Hashicorp or Docker, except with operational expenses running many orders of magnitude higher with costs scaling linearly to revenue.
  - Ekaros 2y ago
    And on other side, how many more paying users will there be? And of users that known about AI or have tried it are happy to use whatever is free at the moment. Or just whatever is on Google or Bing with add next to it? Social media is free, subscription services are for stuff you can't get for free easily. But will there actually be similar need for AI? Be it any generation.
  - bhouston 2y ago
    > LLMs are a utility, not a platform, and utility markets exert a downward pressure on pricing. I think competition exerts a downward pressure on pricing, not being a utility personally. But I guess I agree with the utility analogy in that there are massively initial upfront costs and then the marginal costs are low. > more efficient models leads to pushing inference out to end-user compute, which hollows out their business model Faster CPUs have been coming forever but we keep coming up with ways of keeping them busy. I suspect the same pattern with AI. Thus server-based AI will always be better than local. In the future, I expect to be served by many dozens of persistent agents acting on my behalf (more agents as you go further into the future) and they won't be hosted on my smartphone.
pzs 2y ago
"The real problem is the ROI on AI spending is.. pretty much zero. The commonly asserted use cases are the following: Chatbots Developer tools RAG/search" I agree with you that ROI on _most_ AI spending is indeed poor, but AI is more than LLM's. Alas, what used to be called AI before the onset of the LLM era is not deemed sexy today, even though it can still make very good ROI when it is the appropriate tool for solving a problem.
- aketchum 2y ago
  AI is a term that changes year to year. I don't remember where I heard it but I like that definition that "as soon as computers can do it well it stops becoming AI and just becomes standard tech". Neural Networks were "AI" for a while - but if I use a NN for risk underwriting nobody will call that AI now. It is "just ML" and not exciting. Will AI = LLM forever now? If so what is the next round of advancements called?
  - Mistletoe 2y ago
    https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect
chatmasta 2y ago
While it might be possible for a deep-pocketed organization to spin up a cloud provider overnight, it doesn't mean that people will use it. In general, the switching cost of migrating compute infrastructure from one service to another is much higher than the switching cost of changing the LLM used for inference. Amazon doesn't need to worry about suddenly losing its entire customer base to Alibaba, Yandex, or Oracle.
- smolder 2y ago
  Amazon has spent a ton on developing features that lock in users beyond what is fundamentally 'cloud', i.e. the ability to lease computers as a commodity. I have always warned employers about the downside of adopting all that extraneous stuff, with little effect.
  - insane_dreamer 2y ago
    and all of that is part of the "moat" that the article author is talking about which is almost impossible to replicate in a short amount of time
    - deleted 2y ago
      [deleted]

sushid 2y ago

> And RAG has many, many issues with document permissions Why can't these providers access all documents and when answers are prompted, self-censor if the reply has references to documents that the end users do not have access permissions? In fact, I'm pretty sure that's how existing RAGaaS providers are handling document/file permissions.

portaouflop 2y ago
RAGaas, didn’t know I needed that in my life
Terr_ 2y ago
Would the texts of those documents be part of the LLM training data? If so, there's an reliable way to keep a determined user from fetching stuff back out.
fwip 2y ago
That's a pretty quick way to expose what information you're trying to keep secret.

bhouston 2y ago

> The real problem is the ROI on AI spending is.. pretty much zero. Companies in user acquisition/growth mode tend to have low internal ROI, but remember both Facebook and Google has the same issue -- then they introduced ads and all was well with their finances. Similar things will happen here.

HillRat 2y ago
LLMs are a utility, not a platform, and utility markets exert a downward pressure on pricing. Moreover, it's not obvious that -- once trained models hit the wild -- any one actor has or can develop significant competitive moats that would allow them to escape that price pressure. Beyond that, the digital marginal cost of services needs to be significantly reduced to keep these companies in business, but more efficient models leads to pushing inference out to end-user compute, which hollows out their business model (I assume that Apple dropping out of the OpenAI investment round was partially due to the wildly optimistic valuations involved, partially because they're betting on being able to optimize runtime costs down to iPhone levels). Basically, I'd argue that LLMs look less like a Web 2.0 social media opportunity and more like Hashicorp or Docker, except with operational expenses running many orders of magnitude higher with costs scaling linearly to revenue.
- Ekaros 2y ago
  And on other side, how many more paying users will there be? And of users that known about AI or have tried it are happy to use whatever is free at the moment. Or just whatever is on Google or Bing with add next to it? Social media is free, subscription services are for stuff you can't get for free easily. But will there actually be similar need for AI? Be it any generation.
- bhouston 2y ago
  > LLMs are a utility, not a platform, and utility markets exert a downward pressure on pricing. I think competition exerts a downward pressure on pricing, not being a utility personally. But I guess I agree with the utility analogy in that there are massively initial upfront costs and then the marginal costs are low. > more efficient models leads to pushing inference out to end-user compute, which hollows out their business model Faster CPUs have been coming forever but we keep coming up with ways of keeping them busy. I suspect the same pattern with AI. Thus server-based AI will always be better than local. In the future, I expect to be served by many dozens of persistent agents acting on my behalf (more agents as you go further into the future) and they won't be hosted on my smartphone.

pzs 2y ago

"The real problem is the ROI on AI spending is.. pretty much zero. The commonly asserted use cases are the following: Chatbots Developer tools RAG/search" I agree with you that ROI on _most_ AI spending is indeed poor, but AI is more than LLM's. Alas, what used to be called AI before the onset of the LLM era is not deemed sexy today, even though it can still make very good ROI when it is the appropriate tool for solving a problem.

aketchum 2y ago
AI is a term that changes year to year. I don't remember where I heard it but I like that definition that "as soon as computers can do it well it stops becoming AI and just becomes standard tech". Neural Networks were "AI" for a while - but if I use a NN for risk underwriting nobody will call that AI now. It is "just ML" and not exciting. Will AI = LLM forever now? If so what is the next round of advancements called?
- Mistletoe 2y ago
  https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect

chatmasta 2y ago

While it might be possible for a deep-pocketed organization to spin up a cloud provider overnight, it doesn't mean that people will use it. In general, the switching cost of migrating compute infrastructure from one service to another is much higher than the switching cost of changing the LLM used for inference. Amazon doesn't need to worry about suddenly losing its entire customer base to Alibaba, Yandex, or Oracle.

smolder 2y ago
Amazon has spent a ton on developing features that lock in users beyond what is fundamentally 'cloud', i.e. the ability to lease computers as a commodity. I have always warned employers about the downside of adopting all that extraneous stuff, with little effect.
- insane_dreamer 2y ago
  and all of that is part of the "moat" that the article author is talking about which is almost impossible to replicate in a short amount of time
  - deleted 2y ago
    [deleted]