3 ms·

The data is the moat.

TeMPOraL 2y ago
Not for much longer, perhaps not even now. There's plenty of data avaliable to anyone, and people are finding ways to use that data more effectively. Mid-term, I believe the only real moat is going to be human labor - that is, RLHF and other funny acronyms that boil down to getting people to chat with the model and rate how they feel about its answers. Software improvements (architecture, training process, inference) are always one public paper or leak away from being available for free to anyone. Hardware improvements will spread too, because NVIDIA et al. would prefer to sell more chips than less chips. Meanwhile, human labor is notoriously expensive, only getting more expensive as economic conditions of people improve, and most importantly, whatever "spark" of human intelligence/consciousness there is, this is where it cannot be automated away - not until we get to human-level AGI. Human labor is the one thing that you can only scale by throwing more money at it - which is why modern businesses seek to remove it from the equation as much as possible. Hell, the whole pursuit of AGI is in big part motivated by hope of eliminating labor costs entirely. Except, in this one pursuit, until AGI is reached, labor is a critical resource that has no substitute. That's my mid-term prediction. Long-term, we'll hit AGI and moats won't matter anymore.
- franktankbank 2y ago
  > There's plenty of data avaliable to anyone I think you are just wrong here and so everything that follows is wishful.
  - ben_w 2y ago
    Wikipedia exists and can be downloaded, and the full archive includes talk pages, CC-BY-SA: https://en.wikipedia.org/wiki/Wikipedia:Database_download https://en.wikipedia.org/wiki/Wikipedia:Database_download "Plenty" may be vague, but it's not wrong.
    - xboxnolifes 2y ago
      Does wikipedia help bots make coherent decisions for my personal grocery shopping based on my own habits, price point, allergies, and preferences?
      - ben_w 2y ago
        Why wouldn't it? Are your allergies not listed, is your local economy not explained and thus median expectations of price sensitivity not inferrable, are the words you use to describe your preferences and habits so out of distribution?
reissbaker 2y ago
Considering that OpenAI is barely able to stay ahead of open-source, and in some cases has fallen behind (i.e. 4o vs DeepSeek V3), I think the data moat doesn't really exist. And the difference between 4o and o1 isn't training on more data: quite famously it's scaling test-time compute, not scaling training. "The data is the moat" was a pretty common belief a few years ago, but not anymore.
- rad_gruchalski 2y ago
  And you think they throw in $500B because they are altruistic?

Not for much longer, perhaps not even now. There's plenty of data avaliable to anyone, and people are finding ways to use that data more effectively. Mid-term, I believe the only real moat is going to be human labor - that is, RLHF and other funny acronyms that boil down to getting people to chat with the model and rate how they feel about its answers. Software improvements (architecture, training process, inference) are always one public paper or leak away from being available for free to anyone. Hardware improvements will spread too, because NVIDIA et al. would prefer to sell more chips than less chips. Meanwhile, human labor is notoriously expensive, only getting more expensive as economic conditions of people improve, and most importantly, whatever "spark" of human intelligence/consciousness there is, this is where it cannot be automated away - not until we get to human-level AGI. Human labor is the one thing that you can only scale by throwing more money at it - which is why modern businesses seek to remove it from the equation as much as possible. Hell, the whole pursuit of AGI is in big part motivated by hope of eliminating labor costs entirely. Except, in this one pursuit, until AGI is reached, labor is a critical resource that has no substitute. That's my mid-term prediction. Long-term, we'll hit AGI and moats won't matter anymore.

franktankbank 2y ago
> There's plenty of data avaliable to anyone I think you are just wrong here and so everything that follows is wishful.
- ben_w 2y ago
  Wikipedia exists and can be downloaded, and the full archive includes talk pages, CC-BY-SA: https://en.wikipedia.org/wiki/Wikipedia:Database_download https://en.wikipedia.org/wiki/Wikipedia:Database_download "Plenty" may be vague, but it's not wrong.
  - xboxnolifes 2y ago
    Does wikipedia help bots make coherent decisions for my personal grocery shopping based on my own habits, price point, allergies, and preferences?
    - ben_w 2y ago
      Why wouldn't it? Are your allergies not listed, is your local economy not explained and thus median expectations of price sensitivity not inferrable, are the words you use to describe your preferences and habits so out of distribution?

reissbaker 2y ago

Considering that OpenAI is barely able to stay ahead of open-source, and in some cases has fallen behind (i.e. 4o vs DeepSeek V3), I think the data moat doesn't really exist. And the difference between 4o and o1 isn't training on more data: quite famously it's scaling test-time compute, not scaling training. "The data is the moat" was a pretty common belief a few years ago, but not anymore.

rad_gruchalski 2y ago
And you think they throw in $500B because they are altruistic?