3 ms·

I tend to agree. I'm curious whether this is Deepmind saying "I think we could do things better, let's do this ourselves" or the leadership of Alphabet saying "

I tend to agree. I'm curious whether this is Deepmind saying "I think we could do things better, let's do this ourselves" or the leadership of Alphabet saying "Get these ivy league intellectuals to prioritize productionizing products!" Would seem far more sensible to allow Deepmind to continue to release hit after hit in the ML research world, and simply embed "fly on the wall" PM's into their org that can independently productionize any golden nuggets they happen to create.

TeMPOraL 2y ago
I feel it's more of leadership of Alphabet saying, "Our people literally invented transformers, so how come we're at the bottom of AI race instead of at the top? A random non-profit took out research and run with it, and they're the hottest company in the world now. This is unacceptable![0]". Bad idea. People good or lucky enough to land in R&D like doing R&D. Force them to be product people, I expect most of them will leave. -- [0] - Like Muffin from Bluey, https://youtu.be/hZVlBQXVtZA?t=8 https://youtu.be/hZVlBQXVtZA?t=8.
- smallnamespace 2y ago
  Yes, but that doesn't explain how OpenAI able to walk and chew bubble gum at the same time, unless you think there is some extra spark that Google is lacking.
  - TeMPOraL 2y ago
    I think OpenAI was chilled out, and mostly lucky. Chilled in the sense that Google seems to be too full of itself, and their practice of publishing groundbreaking research without actually publishing anything other people can use is, frankly, annoying. OpenAI managed to leapfrog them by slapping a chat interface on a tuned GPT-3 and putting it on the Internet. The chat service bit is them being chill, but the real spark was the model. I say they were lucky, because AFAIK back then no one expected LLMs to show so many and so advanced general capabilities. This took everyone by surprise, and since people could already play with it, ChatGPT took off on its own - it had so much real, transformative value, that it spread out with zero marketing. That's a rare, bona fide case of "word of mouth", it was just that useful. But that wasn't a strategy, that was luck. To their credit though, OpenAI turned this early win into an opportunity and is excellent at exploiting it. Being small helps.
    - cma 2y ago
      GPT-3 was out for years already for Googlw to see, but with what OpenAI saw of it they began training an expensive GPT-4, before chatgpt success, maybe started before RLHF.
    - ethbr1 2y ago
      > it had so much real, transformative value, that it spread out with zero marketing IMHO, in retrospect, the failure gradient of early LLMs is underappreciated in driving adoption. Windows 95 failure: blue screen with inscrutable error code. Everyone noticed that. LLM failure: run-around non-answer (user shrugs and tries again) or confident and plausible incorrect answer (user doesn't recognize this without research). Essentially, the ways in which LLMs didn't work were the most hidden and hardest to discover failure mode. Which was perfectly tuned for the "I'm going to try this thing for 5 minutes and be amazed" first impression. Which allowed subsequent generations to backfill the capability gaps. Tl;dr - We shouldn't underappreciate quiet-failing as a product adoption driver.
      - FactKnower69 2y ago
        >We shouldn't underappreciate quiet-failing as a product adoption driver. I'm certain it drives a lot of early user retention in the short term, but I feel strongly that this is ultimately a very myopic view which will prove catastrophic in the long term in much the same way that swallowing exceptions at runtime builds compounding technical debt you'll have to reckon with sooner or later more broadly, there is just so much handwaving away all the black box parts of deep neural networks that are completely opaque and there seems to be very little interest in building the tooling to properly visualize, explore, and DEBUG latent space; until those priorities change this whole thing is a huge time bomb. imagine if instead of coming with full memory dumps and diagnostic codes, BSODs just said "sorry, your computer had an oopsie!", and not a single engineer at Microsoft had a complete understanding of why the BSOD happened in the first place; sometimes it just does that! whoops!
        ethbr1 2y ago
        > imagine if instead of coming with full memory dumps and diagnostic codes, BSODs just said "sorry, your computer had an oopsie!" So, MacOS? ;) In all seriousness, I wasn't opining on the usefulness of opaque/hidden errors, but rather the effectiveness of them. In an alternate reality where the first LLMs instead spit back an error reference instead of English, I don't think we would have seen nearly as rapid mass market adoption. And, not to put too fine a point on it, early conversational LLMs and image diffusion models were literally trained so their junk output is as plausible as possible.
- KoolKat23 2y ago
  I agree it is this, I also think people and society also give themselves way too much credit for their successes. There's plenty of smart hardworking people out there who continue to contribute but never stumble upon a unicorn. To a large extent its luck, a much larger contributor than people realize. All you can do is to play the game, consistently contribute and work hard on R&D and products and you improve your odds of stumbling upon success. But it's never guaranteed.

TeMPOraL 2y ago

I feel it's more of leadership of Alphabet saying, "Our people literally invented transformers, so how come we're at the bottom of AI race instead of at the top? A random non-profit took out research and run with it, and they're the hottest company in the world now. This is unacceptable![0]". Bad idea. People good or lucky enough to land in R&D like doing R&D. Force them to be product people, I expect most of them will leave. -- [0] - Like Muffin from Bluey, https://youtu.be/hZVlBQXVtZA?t=8 https://youtu.be/hZVlBQXVtZA?t=8.

smallnamespace 2y ago
Yes, but that doesn't explain how OpenAI able to walk and chew bubble gum at the same time, unless you think there is some extra spark that Google is lacking.
- TeMPOraL 2y ago
  I think OpenAI was chilled out, and mostly lucky. Chilled in the sense that Google seems to be too full of itself, and their practice of publishing groundbreaking research without actually publishing anything other people can use is, frankly, annoying. OpenAI managed to leapfrog them by slapping a chat interface on a tuned GPT-3 and putting it on the Internet. The chat service bit is them being chill, but the real spark was the model. I say they were lucky, because AFAIK back then no one expected LLMs to show so many and so advanced general capabilities. This took everyone by surprise, and since people could already play with it, ChatGPT took off on its own - it had so much real, transformative value, that it spread out with zero marketing. That's a rare, bona fide case of "word of mouth", it was just that useful. But that wasn't a strategy, that was luck. To their credit though, OpenAI turned this early win into an opportunity and is excellent at exploiting it. Being small helps.
  - cma 2y ago
    GPT-3 was out for years already for Googlw to see, but with what OpenAI saw of it they began training an expensive GPT-4, before chatgpt success, maybe started before RLHF.
  - ethbr1 2y ago
    > it had so much real, transformative value, that it spread out with zero marketing IMHO, in retrospect, the failure gradient of early LLMs is underappreciated in driving adoption. Windows 95 failure: blue screen with inscrutable error code. Everyone noticed that. LLM failure: run-around non-answer (user shrugs and tries again) or confident and plausible incorrect answer (user doesn't recognize this without research). Essentially, the ways in which LLMs didn't work were the most hidden and hardest to discover failure mode. Which was perfectly tuned for the "I'm going to try this thing for 5 minutes and be amazed" first impression. Which allowed subsequent generations to backfill the capability gaps. Tl;dr - We shouldn't underappreciate quiet-failing as a product adoption driver.
    - FactKnower69 2y ago
      >We shouldn't underappreciate quiet-failing as a product adoption driver. I'm certain it drives a lot of early user retention in the short term, but I feel strongly that this is ultimately a very myopic view which will prove catastrophic in the long term in much the same way that swallowing exceptions at runtime builds compounding technical debt you'll have to reckon with sooner or later more broadly, there is just so much handwaving away all the black box parts of deep neural networks that are completely opaque and there seems to be very little interest in building the tooling to properly visualize, explore, and DEBUG latent space; until those priorities change this whole thing is a huge time bomb. imagine if instead of coming with full memory dumps and diagnostic codes, BSODs just said "sorry, your computer had an oopsie!", and not a single engineer at Microsoft had a complete understanding of why the BSOD happened in the first place; sometimes it just does that! whoops!
      - ethbr1 2y ago
        > imagine if instead of coming with full memory dumps and diagnostic codes, BSODs just said "sorry, your computer had an oopsie!" So, MacOS? ;) In all seriousness, I wasn't opining on the usefulness of opaque/hidden errors, but rather the effectiveness of them. In an alternate reality where the first LLMs instead spit back an error reference instead of English, I don't think we would have seen nearly as rapid mass market adoption. And, not to put too fine a point on it, early conversational LLMs and image diffusion models were literally trained so their junk output is as plausible as possible.
KoolKat23 2y ago
I agree it is this, I also think people and society also give themselves way too much credit for their successes. There's plenty of smart hardworking people out there who continue to contribute but never stumble upon a unicorn. To a large extent its luck, a much larger contributor than people realize. All you can do is to play the game, consistently contribute and work hard on R&D and products and you improve your odds of stumbling upon success. But it's never guaranteed.