6 ms·

Yes I generally agree with this. Ironically it seems as all the "AI wrappers" look like they have more of a moat than the models (as UX etc is actually quite ha

Yes I generally agree with this. Ironically it seems as all the "AI wrappers" look like they have more of a moat than the models (as UX etc is actually quite hard), which is not what I expected at all when the LLM boom started. I think a lot of LLM usage is actually pretty 'simple' at the moment - think tagging, extracting data, etc. This doesn't require the very top state of the art models (though Llama3.1 is very close to that). Hopefully OpenAI have some real jumps forward in the bag otherwise I am struggling to see how they can justify the valuations being floated around.

CuriouslyC 2y ago
The wrappers don't really have much of a moat either, OpenAI is just bad at front-end dev, so their velocity there is low.
- panarky 2y ago
  Since most of their code is GPT-generated, their velocity should increase with better models.
wslh 2y ago
I think the problem with valuations is the never ending financial history of hypes. The hype is correlated with the technology or industry but follows its own speculation hype. Even if the valuations and speculations are correct, they could be correct in years with ups and downs. If the dot com era is a good example it took half a decade or more to materialize. Other aspects that are not well studied yet is how online advertisement will be affected if most people around the world end up using a single interface such as ChatGPT. How SEO/SEM/Ads will work in that world? Does someone look at what sites benefit being listed in ChatGPT (e.g. Wikipedia).
sillysaurusx 2y ago
One killer feature is OpenAI’s vector database. I was surprised you can throw gigabytes of documents at it and chatgpt can see it all. It’s hard to simulate that via context window. That’s not necessarily a moat, but OpenAI is still shipping important features. I wonder how hard it is for Claude et al to replicate.
- brigadier132 2y ago
  They are outsourcing that to Qdrant, anyone can replicate it.
- Xenoamorphous 2y ago
  What do you mean, does it do anything other than the usual RAG?
  - sillysaurusx 2y ago
    Whatever it does, it’s indistinguishable from stuffing all the documents into the context window. Or at least I haven’t seen it fail yet.
    - KeplerBoy 2y ago
      Stuffing everything into the context window fails horribly every time i try it. The model just doesn't seem to be able to really process the entire input.
  - ramraj07 2y ago
    I’m also confused what they’re talking about.. Does OpenAI have some feature I’m not aware of?
    - dpkirchner 2y ago
      There's a way to upload documents that can be referred to in chat. I think they call it custom GPTs (which seems like a poor name): https://help.openai.com/en/articles/8554397-creating-a-gpt https://help.openai.com/en/articles/8554397-creating-a-gpt
      - singularity2001 2y ago
        Wait if this is what OP is referring to I am even more confused because the last time I built a custom GPT it was horribly slow and very inaccurate to look up uploaded information also you certainly couldn't upload gigabytes of your own PDFs. Did anything change??
crooked-v 2y ago
It's definitely what I expected. As soon as you get outside of the specific use case of chatbots, "AI stuff" is a feature, not a product.
devit 2y ago
Interactive use definitely wants the best model possible so that you have a higher chance of getting a correct and useful response. It might be hard however to decisively convince people that one model is significantly better than another though, so branding/first mover/etc. probably plays a big role.
deleted 2y ago
[deleted]
rmbyrro 2y ago
> how they can justify the valuations being floated around I'm sure they have at least a couple jumps in their bag. Let's not forget inertia, as well. Migrating models is not a trivial project. To gain OpenAI customers, competitors need a considerable jump to justify, which I don't expect any of them to achieve before OpenAI itself delivers their jumps.
- gk1 2y ago
  Products like Cursor literally have a dropdown that lets you switch between models seamlessly. From the surveys I’ve seen, most companies already use more than one model provider. The switching cost is fairly low.
  - rmbyrro 2y ago
    The main problem is actually in making your prompts perform similarly enough with a new model. For sure, switching models is trivial, but it'll disrupt the quality of your service without significant effort in migrating your prompts. I'm not saying this is an intrinsic moat, just that there's a barrier to change.
    - mejutoco 2y ago
      That same problem exists everytime your model served from a third-party is optimized/lobotomized. Only and offline model guarantees this wont happen.
    - deepsquirrelnet 2y ago
      This demonstrates a need for more robust engineering practices. Typically you would solve this by testing a validation set against a new model. Some tools like DSPy or Agenta are helping to encourage this. But creating good evaluators for generative responses isn’t easy, and in my experience people tend to punt.
      - rmbyrro 2y ago
        Prompting is different than programming. Auto-testing your prompts is good practice and helps during a migration. But it doesn't change the fact that the prompts won't work the same in a different model. It's like saying having Python tests eliminates bugs when rewriting a codebase into JavaScript.
rmbyrro 2y ago
> UX etc is actually quite hard It's analogous in the API space. OpenAI is demonstrating to be reasonably good at it for developers. They've been shipping significant features and improvements. Unless they lose pace, they have a moat, at least temporary.
- coffeebeqn 2y ago
  The models are so general that it is much easier to swap out GPT-4 for Claude or Gemini than most other APIs like payments for example
  - rmbyrro 2y ago
    Swaping models isn't the hard part. The prompts are.
    - jsemrau 2y ago
      Coming from a regulated industry that have well established rules about AI/ML in Production. In my opinion both of these model changes and prompt changes are governance issues that should be handled by your friendly CI/CD or Risk Management team(s).
llmfan 2y ago
Current models are very cheap. Like GPT-4 cost ~$40 million to train, right? Once we get 100 billion dollar models I suppose it'll be different.
m3kw9 2y ago
OpenAI has their own wrapper too and they have great UX.
SkyPuncher 2y ago
IMO, it’s basically a generalist vs a specialist. LLMs are the generalist. They’re amazing at a wide variety of things, but unable to go deep. The “wrappers” are the specialists. They’re doing the hard work of product discovery. This takes time, money, and a lot of trial and error to get it exactly right.
- repeekad 2y ago
  I think a big sticking point is chat isn't UX, unless of course 10 years from now we're all actually living in the movie Her... Rebuilding existing apps with AI as a first class citizen seems like the name of the game right now
cainxinth 2y ago
> Ironically it seems as all the "AI wrappers" look like they have more of a moat than the models (as UX etc is actually quite hard), which is not what I expected at all when the LLM boom started. In interviews lately, Sam Altman has been alluding to this more and more.
abss 2y ago
Yes, details matter. The whole idea of creating AGI that is simultaneously a generalist seems more and more like wishful thinking. The reality is that to solve real problems, a large number of correct reasoning steps are required, along with the ability to make choices about which type of inference is useful at each step to avoid the explosion of complexity inherent in any brute-force approach. This suggests that we will have AI experts in different domains, perhaps superior to humans, but we will have thousands or even millions of narrow areas of expertise. To create something akin to an all-knowing superintelligent deity, we would need to combine thousands of experts, which would also consume unsustainable amounts of energy. I wouldn't bet on AGI in the coming years; it's just hype and distracts the discussion until big money finds a way to establish monopolies. However, if both UX and reasoning expertise require deep customization and specialization, we have a real chance to use AI to solve deep social problems rather than transforming society into a dystopia where humans are morally and intellectually surpassed, and those remaining are controlled by corporations that could at any moment be taken over by sociopaths.

CuriouslyC 2y ago

The wrappers don't really have much of a moat either, OpenAI is just bad at front-end dev, so their velocity there is low.

panarky 2y ago
Since most of their code is GPT-generated, their velocity should increase with better models.

wslh 2y ago

I think the problem with valuations is the never ending financial history of hypes. The hype is correlated with the technology or industry but follows its own speculation hype. Even if the valuations and speculations are correct, they could be correct in years with ups and downs. If the dot com era is a good example it took half a decade or more to materialize. Other aspects that are not well studied yet is how online advertisement will be affected if most people around the world end up using a single interface such as ChatGPT. How SEO/SEM/Ads will work in that world? Does someone look at what sites benefit being listed in ChatGPT (e.g. Wikipedia).

sillysaurusx 2y ago

One killer feature is OpenAI’s vector database. I was surprised you can throw gigabytes of documents at it and chatgpt can see it all. It’s hard to simulate that via context window. That’s not necessarily a moat, but OpenAI is still shipping important features. I wonder how hard it is for Claude et al to replicate.

brigadier132 2y ago
They are outsourcing that to Qdrant, anyone can replicate it.
Xenoamorphous 2y ago
What do you mean, does it do anything other than the usual RAG?
- sillysaurusx 2y ago
  Whatever it does, it’s indistinguishable from stuffing all the documents into the context window. Or at least I haven’t seen it fail yet.
  - KeplerBoy 2y ago
    Stuffing everything into the context window fails horribly every time i try it. The model just doesn't seem to be able to really process the entire input.
- ramraj07 2y ago
  I’m also confused what they’re talking about.. Does OpenAI have some feature I’m not aware of?
  - dpkirchner 2y ago
    There's a way to upload documents that can be referred to in chat. I think they call it custom GPTs (which seems like a poor name): https://help.openai.com/en/articles/8554397-creating-a-gpt https://help.openai.com/en/articles/8554397-creating-a-gpt
    - singularity2001 2y ago
      Wait if this is what OP is referring to I am even more confused because the last time I built a custom GPT it was horribly slow and very inaccurate to look up uploaded information also you certainly couldn't upload gigabytes of your own PDFs. Did anything change??

crooked-v 2y ago

It's definitely what I expected. As soon as you get outside of the specific use case of chatbots, "AI stuff" is a feature, not a product.

devit 2y ago

Interactive use definitely wants the best model possible so that you have a higher chance of getting a correct and useful response. It might be hard however to decisively convince people that one model is significantly better than another though, so branding/first mover/etc. probably plays a big role.

deleted 2y ago

[deleted]

rmbyrro 2y ago

> how they can justify the valuations being floated around I'm sure they have at least a couple jumps in their bag. Let's not forget inertia, as well. Migrating models is not a trivial project. To gain OpenAI customers, competitors need a considerable jump to justify, which I don't expect any of them to achieve before OpenAI itself delivers their jumps.

gk1 2y ago
Products like Cursor literally have a dropdown that lets you switch between models seamlessly. From the surveys I’ve seen, most companies already use more than one model provider. The switching cost is fairly low.
- rmbyrro 2y ago
  The main problem is actually in making your prompts perform similarly enough with a new model. For sure, switching models is trivial, but it'll disrupt the quality of your service without significant effort in migrating your prompts. I'm not saying this is an intrinsic moat, just that there's a barrier to change.
  - mejutoco 2y ago
    That same problem exists everytime your model served from a third-party is optimized/lobotomized. Only and offline model guarantees this wont happen.
  - deepsquirrelnet 2y ago
    This demonstrates a need for more robust engineering practices. Typically you would solve this by testing a validation set against a new model. Some tools like DSPy or Agenta are helping to encourage this. But creating good evaluators for generative responses isn’t easy, and in my experience people tend to punt.
    - rmbyrro 2y ago
      Prompting is different than programming. Auto-testing your prompts is good practice and helps during a migration. But it doesn't change the fact that the prompts won't work the same in a different model. It's like saying having Python tests eliminates bugs when rewriting a codebase into JavaScript.

rmbyrro 2y ago

> UX etc is actually quite hard It's analogous in the API space. OpenAI is demonstrating to be reasonably good at it for developers. They've been shipping significant features and improvements. Unless they lose pace, they have a moat, at least temporary.

coffeebeqn 2y ago
The models are so general that it is much easier to swap out GPT-4 for Claude or Gemini than most other APIs like payments for example
- rmbyrro 2y ago
  Swaping models isn't the hard part. The prompts are.
  - jsemrau 2y ago
    Coming from a regulated industry that have well established rules about AI/ML in Production. In my opinion both of these model changes and prompt changes are governance issues that should be handled by your friendly CI/CD or Risk Management team(s).

llmfan 2y ago

Current models are very cheap. Like GPT-4 cost ~$40 million to train, right? Once we get 100 billion dollar models I suppose it'll be different.

m3kw9 2y ago

OpenAI has their own wrapper too and they have great UX.

SkyPuncher 2y ago

IMO, it’s basically a generalist vs a specialist. LLMs are the generalist. They’re amazing at a wide variety of things, but unable to go deep. The “wrappers” are the specialists. They’re doing the hard work of product discovery. This takes time, money, and a lot of trial and error to get it exactly right.

repeekad 2y ago
I think a big sticking point is chat isn't UX, unless of course 10 years from now we're all actually living in the movie Her... Rebuilding existing apps with AI as a first class citizen seems like the name of the game right now

cainxinth 2y ago

> Ironically it seems as all the "AI wrappers" look like they have more of a moat than the models (as UX etc is actually quite hard), which is not what I expected at all when the LLM boom started. In interviews lately, Sam Altman has been alluding to this more and more.

abss 2y ago

Yes, details matter. The whole idea of creating AGI that is simultaneously a generalist seems more and more like wishful thinking. The reality is that to solve real problems, a large number of correct reasoning steps are required, along with the ability to make choices about which type of inference is useful at each step to avoid the explosion of complexity inherent in any brute-force approach. This suggests that we will have AI experts in different domains, perhaps superior to humans, but we will have thousands or even millions of narrow areas of expertise. To create something akin to an all-knowing superintelligent deity, we would need to combine thousands of experts, which would also consume unsustainable amounts of energy. I wouldn't bet on AGI in the coming years; it's just hype and distracts the discussion until big money finds a way to establish monopolies. However, if both UX and reasoning expertise require deep customization and specialization, we have a real chance to use AI to solve deep social problems rather than transforming society into a dystopia where humans are morally and intellectually surpassed, and those remaining are controlled by corporations that could at any moment be taken over by sociopaths.