3 ms·

What this really says to me is the indefensibility of any current advances. There’s really cool stuff going on right now, but anyone can do it. Not to say anyon

What this really says to me is the indefensibility of any current advances. There’s really cool stuff going on right now, but anyone can do it. Not to say anyone can push the limits of research, but once the cat’s out of the bag, anyone with a few $B and dozen engineers can replicate a model that’s indistinguishably good from best in class to most users.

zurfer 3y ago
Barrier to entry with "few $B" is pretty high. Especially since the scaling laws indicate that it's only getting more expensive. And even if you manage to raise $Bs, you still need to be clever on how to deploy it (talent, compute, data) ...
- appplication 3y ago
  You’re totally right, a few $B is not something any of us are bootstrapping. But there is no secret sauce (at least none that stays secret for long), no meaningful patents, no network/platform effect, and virtually no ability to lock in customers. Compare to other traditional tech companies… think Uber/AirBnB/Databricks/etc. Their product isn’t an algorithm that a competitor can spin up in 6 months. These companies create real moats, for better or worse, which significantly reduce the ability for competitors to enter, even with tranches of cash. In contrast, essentially every product we’ve seen in the AI space is very replicable, and any differentiation is largely marginal, under the hood, and the details of which are obscured from customers.
  - zurfer 3y ago
    Every big tech in the beginning looked fragile/no moats. I think we'll see that data, knowledge and intelligence compound and at some point it will be as hard to penetrate as Meta's network effects.
    - HarHarVeryFunny 3y ago
      Maybe consolidate as well as compound. There's a tendency for any mature industry (which may initially have been bustling with competitors) to eventually consolidate into three players, and while we're currently at the point where it seems a well-funded new entrant can catch up with the leaders, that will likely become much harder in the future as tech advances. Never say never though - look at Tesla coming out of nowhere to push the big three automakers around! Eventually the established players become too complacent and set in their ways, creating an opening for a smaller more nimble competitor with a better idea. I don't think LLMs are the ultimate form of AI/AGI though. Eventually we'll figure out a better brain-inspired approach that learns continually from it's own experimentation and experience. Perhaps this change of approach will be when some much smaller competitor (someone like John Carmack, perhaps) rapidly come from nowhere and catch the big three flat footed as they tend to their ginormous LLM training sets, infrastructure and entrenched products.
      - lanstin 3y ago
        Also worth keeping in mind the lock in for the big tech firms is due to business decisions not the technology per se. If we had say micropaynents in http1 headers in 1998 we might have a much more decentralized system supported by distributed subscriptions rather than ads. To this day I cannot put up $50 to mastodon and have it split amongst the posts I like or boost or whatever. Instead we have all the top content authors trying to get me to subscribe to their email subscriptions which Isa vastly inferior interface and too expensive to get money to all the good writers out there.
- Sateeshm 3y ago
  There is no meaningful network effect or vendor lock-in - which is like the #1 thing that prevents companies from competing. That's the real problem for these AI companies.
HarHarVeryFunny 3y ago
Yes, it seems that AI in form of LLMs is just an idea whose time has come. We now have the compute, the data, and the architecture (transformer) to do it. As far as different groups leapfrogging each other for supremacy in various benchmarks, there might be a bit of a "4 minute mile" effect here too - once you know that something is possible then you can focus on replicating/exceeding it without having to worry are you hitting up against some hard limit. I think the transformer still doesn't get the credit due for enabling this LLM-as-AI revolution. We've had the compute and data for a while, but this breakthough - shared via a public paper - was what has enabled it and made it essentially a level playing field for anyone with the few $B etc the approach requires. I've never seen any claim by any of the transformer paper ("attention is all you need") authors that they understood/anticipated the true power of this model they created (esp. when applied at scale), which as the title suggests was basically regarded an incremental advance over other seq2seq approaches of the time. It seems like one of history's great accidental discoveries. I believe there is something very specific about the key-value matching "attention" mechanism of the transformer (perhaps roughly equivalent to some similar process used in our cortex?) that gives it it's power.
- visarga 3y ago
  > We now have the compute, the data, and the architecture (transformer) to do it. It's really not the model, it's the data and scaling. Otherwise the success of different architectures like Mamba would be hard to justify. Conversely, humans getting training on the same topics achieve very similar results, even though brains are very different at low level, not even the same number of neurons, not to mention different wiring. The merit for our current wave is 99% on the training data, its quality and size are the true AI heroes. And it took humanity our whole existence to build up to this training set, it cost "a lot" to explore and discover the concepts we put inside it. A single human, group or even a whole generation of humans would not be able to rediscover it from scratch in a lifetime. Our cultural data is smarter than us individually, it is as smart as humanity as a whole. One consequence of this insight is that we are probably on an AI plateau. We have used up most organic text. The next step is AI generating its own experiences in the world, but it's going to be a slow grind in many fields where environment feedback is not easy to obtain.

zurfer 3y ago

Barrier to entry with "few $B" is pretty high. Especially since the scaling laws indicate that it's only getting more expensive. And even if you manage to raise $Bs, you still need to be clever on how to deploy it (talent, compute, data) ...

appplication 3y ago
You’re totally right, a few $B is not something any of us are bootstrapping. But there is no secret sauce (at least none that stays secret for long), no meaningful patents, no network/platform effect, and virtually no ability to lock in customers. Compare to other traditional tech companies… think Uber/AirBnB/Databricks/etc. Their product isn’t an algorithm that a competitor can spin up in 6 months. These companies create real moats, for better or worse, which significantly reduce the ability for competitors to enter, even with tranches of cash. In contrast, essentially every product we’ve seen in the AI space is very replicable, and any differentiation is largely marginal, under the hood, and the details of which are obscured from customers.
- zurfer 3y ago
  Every big tech in the beginning looked fragile/no moats. I think we'll see that data, knowledge and intelligence compound and at some point it will be as hard to penetrate as Meta's network effects.
  - HarHarVeryFunny 3y ago
    Maybe consolidate as well as compound. There's a tendency for any mature industry (which may initially have been bustling with competitors) to eventually consolidate into three players, and while we're currently at the point where it seems a well-funded new entrant can catch up with the leaders, that will likely become much harder in the future as tech advances. Never say never though - look at Tesla coming out of nowhere to push the big three automakers around! Eventually the established players become too complacent and set in their ways, creating an opening for a smaller more nimble competitor with a better idea. I don't think LLMs are the ultimate form of AI/AGI though. Eventually we'll figure out a better brain-inspired approach that learns continually from it's own experimentation and experience. Perhaps this change of approach will be when some much smaller competitor (someone like John Carmack, perhaps) rapidly come from nowhere and catch the big three flat footed as they tend to their ginormous LLM training sets, infrastructure and entrenched products.
    - lanstin 3y ago
      Also worth keeping in mind the lock in for the big tech firms is due to business decisions not the technology per se. If we had say micropaynents in http1 headers in 1998 we might have a much more decentralized system supported by distributed subscriptions rather than ads. To this day I cannot put up $50 to mastodon and have it split amongst the posts I like or boost or whatever. Instead we have all the top content authors trying to get me to subscribe to their email subscriptions which Isa vastly inferior interface and too expensive to get money to all the good writers out there.
Sateeshm 3y ago
There is no meaningful network effect or vendor lock-in - which is like the #1 thing that prevents companies from competing. That's the real problem for these AI companies.

HarHarVeryFunny 3y ago

Yes, it seems that AI in form of LLMs is just an idea whose time has come. We now have the compute, the data, and the architecture (transformer) to do it. As far as different groups leapfrogging each other for supremacy in various benchmarks, there might be a bit of a "4 minute mile" effect here too - once you know that something is possible then you can focus on replicating/exceeding it without having to worry are you hitting up against some hard limit. I think the transformer still doesn't get the credit due for enabling this LLM-as-AI revolution. We've had the compute and data for a while, but this breakthough - shared via a public paper - was what has enabled it and made it essentially a level playing field for anyone with the few $B etc the approach requires. I've never seen any claim by any of the transformer paper ("attention is all you need") authors that they understood/anticipated the true power of this model they created (esp. when applied at scale), which as the title suggests was basically regarded an incremental advance over other seq2seq approaches of the time. It seems like one of history's great accidental discoveries. I believe there is something very specific about the key-value matching "attention" mechanism of the transformer (perhaps roughly equivalent to some similar process used in our cortex?) that gives it it's power.

visarga 3y ago
> We now have the compute, the data, and the architecture (transformer) to do it. It's really not the model, it's the data and scaling. Otherwise the success of different architectures like Mamba would be hard to justify. Conversely, humans getting training on the same topics achieve very similar results, even though brains are very different at low level, not even the same number of neurons, not to mention different wiring. The merit for our current wave is 99% on the training data, its quality and size are the true AI heroes. And it took humanity our whole existence to build up to this training set, it cost "a lot" to explore and discover the concepts we put inside it. A single human, group or even a whole generation of humans would not be able to rediscover it from scratch in a lifetime. Our cultural data is smarter than us individually, it is as smart as humanity as a whole. One consequence of this insight is that we are probably on an AI plateau. We have used up most organic text. The next step is AI generating its own experiences in the world, but it's going to be a slow grind in many fields where environment feedback is not easy to obtain.