8 ms·

I think we are in the middle of a steep S-curve of technology innovation. It is far from plateauing and there are still a bunch of major innovations that are l

I think we are in the middle of a steep S-curve of technology innovation. It is far from plateauing and there are still a bunch of major innovations that are likely to shift things even further. Interesting time and these companies are riding a wild wave. It is likely some will actually win big, but most will die - similar to previous technology revolutions. The ones that win will win not just on technology, but on talent retention, business relationships/partnerships, deep funding, marketing, etc. The whole package really. Losing is easy, miss out on one of these for a short period of time and you've easily lost. There is no major moat, except great execution across all dimensions.

ninetyninenine 2y ago
Likely we are because the actualized end result already exists: the human brain. The fact that human intelligence exists means that the idea of human level intelligence is not a pipe dream. The question is whether or not the basic underlying technology of the LLM can achieve that level of intelligence.
- thfuran 2y ago
  And brains do it on about 20 watts.
  - asterix_pano 2y ago
    Training takes many years though.
    - namaria 2y ago
      Training took hundreds of thousands of years. Everyone just gets a foundation model to fine tune for another couple of decades before it can start flipping burgers.
      - thfuran 2y ago
        It's been something like half a billion years since the first brain.
        namaria 2y ago
        That was a much smaller model, couldn't do much more than crawl around and run away.
        thfuran 2y ago
        It was still used as part of pretraining the current model.
        namaria 2y ago
        Nonsense, the current model is a new architectural approach. It was all explained in that recent paper, "Attention is all your meat"
foobiekr 2y ago
In particular: "There is, however, one enormous difference that I didn’t think about: You can’t build a cloud vendor overnight. Azure doesn’t have to worry about a few executives leaving and building a worldwide network of data centers in 18 months." This isn't true at all. There are like 8 of these companies stood up in the last three or four years fueled by massive investment of sovereign funds - mostly the saudi, dubai, northern europe, etc. oil-derived funds - all spending billions of dollars doing exactly that and getting something done. The real problem is the ROI on AI spending is.. pretty much zero. The commonly asserted use cases are the following: Chatbots Developer tools RAG/search Not a one of these is going to generate $10 of additional revenue per sollar spent, nor likely even $2. Optimizing your customer services representatives from 8 conversations at once to an average of 12 or 16 is going to save you a whopping $2 per hour per CSR. It just isn't huge money. And RAG has many, many issues with document permissions that make the current approaches bad for enterprises - where the money is - who as a group haven't spent much of anything to even make basic search work.
- sushid 2y ago
  > And RAG has many, many issues with document permissions Why can't these providers access all documents and when answers are prompted, self-censor if the reply has references to documents that the end users do not have access permissions? In fact, I'm pretty sure that's how existing RAGaaS providers are handling document/file permissions.
  - portaouflop 2y ago
    RAGaas, didn’t know I needed that in my life
  - Terr_ 2y ago
    Would the texts of those documents be part of the LLM training data? If so, there's an reliable way to keep a determined user from fetching stuff back out.
  - fwip 2y ago
    That's a pretty quick way to expose what information you're trying to keep secret.
alexashka 2y ago
> I think we are in the middle of a steep S-curve of technology innovation We are? What innovation? What do we need innovation for? What present societal problems can tech innovation possibly address? Surely none of the big ones, right? So then is it fit to call technological change - 'innovation'? I'd agree that LLMs improve upon having to read Wikipedia for topics I'm interested in but would investing billions in Wikipedia and organizing human knowledge have produced a better outcome than relying on a magic LLM? Almost certainly, in my mind. You see, people are pouring billions into LLMs and not Wikipedia not because it is a better product - but because they foresee a possibility of an abusive monopoly and that really excites them. That's not innovation - that's more of the same anti-social behaviour that makes any meaningful innovation extremely difficult.
- SoftTalker 2y ago
  One of Google's founding goals was "organizing human knowledge" IIRC. They ended up being an ad company.
  - achenet 2y ago
    In a way, that is in alignment with their founding goal of organizing human knowledge. An ad is just a piece of data (eg Bob is selling shovels for $10 + shipping), with the additional metadata that someone really wants you to see it (eg Bob paid Google $100 to tell every person who searched for "shovels" that he's selling shovels for $10 + shipping). Selling ads is organizing human knowledge. Yes, you could argue using a market system is not the best way to do this, but it is a way to do it.
- mellosouls 2y ago
  I'm not sure the Wikipedia example is a strong one as that site has it's own serious problems with "abusive monopolies" in its moderator cliques and biases (as with any social platform). At least with the current big AI players there is the potential for differentiation through competition. Unless there is some similar initiative with the Wikipedias, the problem of single supplier dominance is a difficult one to see as the way forward.
ActionHank 2y ago
The car wasn't a horse that was better, but a car has not changed drastically since they went mainstream. They've gotten better, more efficient, loaded with tech, but are still roughly 4 seats, 4 doors, 4 wheels, driven by petroleum. I know that this is a massive oversimplification, but I think we have seen the "shape" of LLMs\Gen AI\AI products already and it's all incremental improvements from here on out with more specialization. We are going to have SUVs, sports cars, and single seater cars, not flying cars. AI will be made more fit for purpose for more people to use, but isn't going to replace people outright in their jobs.
- pavlov 2y ago
  Feels like someone might have said this in 1981 about personal computers. "We've pretty much seen their shape. The IBM PC isn't fundamentally very different from the Apple II. Probably it's just all incremental improvements from here on out."
  - ActionHank 2y ago
    I would agree with your counter if it weren't for the realities of power usage, hardware constraints, evident diminishing returns on training larger models, and as always the fact that AI is still looking for the problem it solves, aside from mass employment. Computers solved a tangible problem in every area of life, AI is being forced everywhere and is arguably failing to make a big gain in areas that it should excel.
  - adamc 2y ago
    What do you think hasn't been? I think the big game changer in the PC space was graphics cards, but since their introduction, it has all been incremental improvement -- at first, pretty fast, then... slower. Much like CPU improvements, although those started earlier. I can't think of a point where the next generation of PCs was astoundingly different from the prior one... just better. It used to be that they were reliably faster or more capable almost every year, now the rate of improvements is almost negligible. (Yes, graphics are getting better, but not very fast if you aren't near the high end.)

ninetyninenine 2y ago

Likely we are because the actualized end result already exists: the human brain. The fact that human intelligence exists means that the idea of human level intelligence is not a pipe dream. The question is whether or not the basic underlying technology of the LLM can achieve that level of intelligence.

thfuran 2y ago
And brains do it on about 20 watts.
- asterix_pano 2y ago
  Training takes many years though.
  - namaria 2y ago
    Training took hundreds of thousands of years. Everyone just gets a foundation model to fine tune for another couple of decades before it can start flipping burgers.
    - thfuran 2y ago
      It's been something like half a billion years since the first brain.
      - namaria 2y ago
        That was a much smaller model, couldn't do much more than crawl around and run away.
        thfuran 2y ago
        It was still used as part of pretraining the current model.
        namaria 2y ago
        Nonsense, the current model is a new architectural approach. It was all explained in that recent paper, "Attention is all your meat"

foobiekr 2y ago

In particular: "There is, however, one enormous difference that I didn’t think about: You can’t build a cloud vendor overnight. Azure doesn’t have to worry about a few executives leaving and building a worldwide network of data centers in 18 months." This isn't true at all. There are like 8 of these companies stood up in the last three or four years fueled by massive investment of sovereign funds - mostly the saudi, dubai, northern europe, etc. oil-derived funds - all spending billions of dollars doing exactly that and getting something done. The real problem is the ROI on AI spending is.. pretty much zero. The commonly asserted use cases are the following: Chatbots Developer tools RAG/search Not a one of these is going to generate $10 of additional revenue per sollar spent, nor likely even $2. Optimizing your customer services representatives from 8 conversations at once to an average of 12 or 16 is going to save you a whopping $2 per hour per CSR. It just isn't huge money. And RAG has many, many issues with document permissions that make the current approaches bad for enterprises - where the money is - who as a group haven't spent much of anything to even make basic search work.

sushid 2y ago
> And RAG has many, many issues with document permissions Why can't these providers access all documents and when answers are prompted, self-censor if the reply has references to documents that the end users do not have access permissions? In fact, I'm pretty sure that's how existing RAGaaS providers are handling document/file permissions.
- portaouflop 2y ago
  RAGaas, didn’t know I needed that in my life
- Terr_ 2y ago
  Would the texts of those documents be part of the LLM training data? If so, there's an reliable way to keep a determined user from fetching stuff back out.
- fwip 2y ago
  That's a pretty quick way to expose what information you're trying to keep secret.

alexashka 2y ago

> I think we are in the middle of a steep S-curve of technology innovation We are? What innovation? What do we need innovation for? What present societal problems can tech innovation possibly address? Surely none of the big ones, right? So then is it fit to call technological change - 'innovation'? I'd agree that LLMs improve upon having to read Wikipedia for topics I'm interested in but would investing billions in Wikipedia and organizing human knowledge have produced a better outcome than relying on a magic LLM? Almost certainly, in my mind. You see, people are pouring billions into LLMs and not Wikipedia not because it is a better product - but because they foresee a possibility of an abusive monopoly and that really excites them. That's not innovation - that's more of the same anti-social behaviour that makes any meaningful innovation extremely difficult.

SoftTalker 2y ago
One of Google's founding goals was "organizing human knowledge" IIRC. They ended up being an ad company.
- achenet 2y ago
  In a way, that is in alignment with their founding goal of organizing human knowledge. An ad is just a piece of data (eg Bob is selling shovels for $10 + shipping), with the additional metadata that someone really wants you to see it (eg Bob paid Google $100 to tell every person who searched for "shovels" that he's selling shovels for $10 + shipping). Selling ads is organizing human knowledge. Yes, you could argue using a market system is not the best way to do this, but it is a way to do it.
mellosouls 2y ago
I'm not sure the Wikipedia example is a strong one as that site has it's own serious problems with "abusive monopolies" in its moderator cliques and biases (as with any social platform). At least with the current big AI players there is the potential for differentiation through competition. Unless there is some similar initiative with the Wikipedias, the problem of single supplier dominance is a difficult one to see as the way forward.

ActionHank 2y ago

The car wasn't a horse that was better, but a car has not changed drastically since they went mainstream. They've gotten better, more efficient, loaded with tech, but are still roughly 4 seats, 4 doors, 4 wheels, driven by petroleum. I know that this is a massive oversimplification, but I think we have seen the "shape" of LLMs\Gen AI\AI products already and it's all incremental improvements from here on out with more specialization. We are going to have SUVs, sports cars, and single seater cars, not flying cars. AI will be made more fit for purpose for more people to use, but isn't going to replace people outright in their jobs.

pavlov 2y ago
Feels like someone might have said this in 1981 about personal computers. "We've pretty much seen their shape. The IBM PC isn't fundamentally very different from the Apple II. Probably it's just all incremental improvements from here on out."
- ActionHank 2y ago
  I would agree with your counter if it weren't for the realities of power usage, hardware constraints, evident diminishing returns on training larger models, and as always the fact that AI is still looking for the problem it solves, aside from mass employment. Computers solved a tangible problem in every area of life, AI is being forced everywhere and is arguably failing to make a big gain in areas that it should excel.
- adamc 2y ago
  What do you think hasn't been? I think the big game changer in the PC space was graphics cards, but since their introduction, it has all been incremental improvement -- at first, pretty fast, then... slower. Much like CPU improvements, although those started earlier. I can't think of a point where the next generation of PCs was astoundingly different from the prior one... just better. It used to be that they were reliably faster or more capable almost every year, now the rate of improvements is almost negligible. (Yes, graphics are getting better, but not very fast if you aren't near the high end.)