7 ms·

Parts of it were good, but it mostly felt like he was reading through a slideshow created by the Google marketing team.

by paxys 3y ago

throwaway4good 3y ago
I watched some of this after it was promoted to me on YouTube. It is very much: Look at this other cool blackbox that Google has made - I think it has a helpful personality. And sure: Applied AI / ML is a part of computer science but I had hoped it would be more of a walk-through of what has happened in terms of advances in theory, algorithms, architecture, training methodology, and perhaps some sort of explanation of why does it work giving this general purpose model a bunch of medical data? (Is it merely an elaborate fuzzy search, how does it extrapolate, is there actually reasoning going on and how does emerge from a neural net etc...)
knbknb 3y ago
Maybe this was a review talk, designed for an audience of undergraduate students from various majors.
babl-yc 3y ago
Agreed -- given the talk is titled "Exciting trends in Machine Learning" it feels pretty incomplete to gloss over ChatGPT blowing up. OpenAI bet big on (1) emergent behaviors as you scale language models and (2) RLHF / fine-tuning to follow instructions. Both those topics were lightly covered but very much from the "Google did this" perspective (word2vec, step-by-step reasoning).
- lern_too_spel 3y ago
  The original paper on emergent behaviors in LLMs is from Google: https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682 The difference between OpenAI and Google isn't due to different research directions between the two firms. It is a difference in ability to execute.
  - babl-yc 3y ago
    Google certainly had influential papers on language models having emergent behaviors, but this isn't the one that inspired scaling up GPT. It was published August 2022 and GPTs at OpenAI were getting scaled up long before this.
  - pama 3y ago
    This paper was interesting at the time but the main sentence in it was wrong: “Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models.” Such behaviors can totally be predicted if one looks at the evolution of the likelihoods for these behaviors by scaling and extrapolating from the small models.
    - benlivengood 3y ago
      I don't think many-shot prompting or chain-of-thought were predictable from model size. They just showed up at a particular model+data size.
    - canjobear 3y ago
      Maybe you can predict emergent abilities post hoc, but I recall no one predicting beforehand that, for example, a pure language model could do translation simply be giving a prompt that said “translate to French”
      - jcuenod 3y ago
        Then they're not "emergent"
        deanishe 3y ago
        I do not think that word means what you think it means.
        jcuenod 3y ago
        Can't tell if you're responding to me or parent. But that's what I was gonna say to @canjobear...
- elevatedastalt 3y ago
  The research breakthroughs all happened at Google, so it's not surprising.
  - janalsncm 3y ago
    RLHF was invented by OpenAI. DPO was invented at Stanford.
    - deleted 3y ago
      [deleted]
    - _t89y 3y ago
      Exactly. Google.
      - candiodari 3y ago
        And all the people that actually did the inventing promptly left Google. (yes I get there's 2 exceptions)
godelski 3y ago
What I hate about the ML community is that papers have become ad pieces now. I'm all for them releasing technical papers, but do we have to fuck up the review system for it. We don't need to railroad our research community and doing so is pretty risky. I don't buy the scale is all you need side. But if you do, doesn't matter if I'm wrong, we'll just get there when Sam gets his $7T. But if I'm right, we need new ideas if we're going to keep progressing without missing a step.
- _t89y 3y ago
  Domain adaptation across verticals is the only driver of innovation. Check out the NeSy computation engine. Its ``semantic parsing'' is domain parsing and its symbols are numeric. Scale if you want. It works for images. That's all that matters, right? Mapping language onto the latent space of images gets you crude semantic attributes sometimes. If you have to push for multimodal out of the gate maybe start with articulatory perception. These LVMs aren't going to cut it. ML research isn't meant to further your understanding of anything. You can't separate it from corporate interest and land grabbing. It's the same thing re-hashed every year by the same people. NLP is pretty much a subfield of IR at this point. I love how fast my comment was blacklisted to the bottom of this thread. lol
  - godelski 3y ago
    I think you need to understand your audience better. Remember that most people have a very shallow understanding of DL systems. You got blog posts that talk about attention from a mathematical perspective but don't even hint at softmax tempering and just mention that there's a dot product. Or where ML researchers don't know things like why doubling batch size doesn't cut training in half or using fp16 doesn't cut memory in half. So I think toning down the language, being clearer and adding links will help you be more successful in communicating.
    - _t89y 3y ago
      Thanks for your comment. Please let me know what's not clear. Still no takers on my ML is CV comment below. Research on on-device ASR and computer vision is primarily driven by the same organizations that stand to benefit the most from it. It's nearly impossible to talk about machine learning today without talking about computer vision. Just look at the daily papers from Hugging Face or any other outlet. Machine learning is basically synonymous with computer vision and natural language processing is basically synonymous with information retrieval. Research is corporate research. With very few exceptions. Two popular lines of current ML research, multimodal and this latest neuro-symbolic re-hash, are not about furthering our understanding of what we currently can and cannot do. Not about doing science. They are about maintaining the status quo. They are about short-form video content and Google Knowledge Panels. Multimodal ML research is a vision-first enterprise. This doesn't make sense for a number of reasons. Here are two: the latent space of images and the representations therein cannot adequately capture the nuances and expressivity of language; language is a more fundamental cognitive process than vision. And what do you get for language in current ML research? How about a ``neuro-symbolic semantic parser'' that is neither a semantic parsers or symbolic. lol https://arxiv.org/abs/2402.00854v1 https://arxiv.org/abs/2402.00854v1 What's it good for? Computation graphs, domain adaptation, Google Knowledge Panels. This is a directed attack on machine learning research taking concepts that could be pursued with scientific merit, such as multimodal perception and neuro-symbolic parsing, but are instead turned into marketing hype and leveraged by the powers that be for the things that keep them in power. My audience is anyone participating in this research. Y Combinator... that's the mob of onewheels and URB-E scooters dodging human waste in the Tenderloin on their commute from Nob Hill to the Mission, right? Maybe you are referring to another audience.

Parts of it were good, but it mostly felt like he was reading through a slideshow created by the Google marketing team.

by paxys 3y ago

Parts of it were good, but it mostly felt like he was reading through a slideshow created by the Google marketing team.

I watched some of this after it was promoted to me on YouTube. It is very much: Look at this other cool blackbox that Google has made - I think it has a helpful personality. And sure: Applied AI / ML is a part of computer science but I had hoped it would be more of a walk-through of what has happened in terms of advances in theory, algorithms, architecture, training methodology, and perhaps some sort of explanation of why does it work giving this general purpose model a bunch of medical data? (Is it merely an elaborate fuzzy search, how does it extrapolate, is there actually reasoning going on and how does emerge from a neural net etc...)

knbknb 3y ago

Maybe this was a review talk, designed for an audience of undergraduate students from various majors.

babl-yc 3y ago

Agreed -- given the talk is titled "Exciting trends in Machine Learning" it feels pretty incomplete to gloss over ChatGPT blowing up. OpenAI bet big on (1) emergent behaviors as you scale language models and (2) RLHF / fine-tuning to follow instructions. Both those topics were lightly covered but very much from the "Google did this" perspective (word2vec, step-by-step reasoning).

lern_too_spel 3y ago
The original paper on emergent behaviors in LLMs is from Google: https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682 The difference between OpenAI and Google isn't due to different research directions between the two firms. It is a difference in ability to execute.
- babl-yc 3y ago
  Google certainly had influential papers on language models having emergent behaviors, but this isn't the one that inspired scaling up GPT. It was published August 2022 and GPTs at OpenAI were getting scaled up long before this.
- pama 3y ago
  This paper was interesting at the time but the main sentence in it was wrong: “Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models.” Such behaviors can totally be predicted if one looks at the evolution of the likelihoods for these behaviors by scaling and extrapolating from the small models.
  - benlivengood 3y ago
    I don't think many-shot prompting or chain-of-thought were predictable from model size. They just showed up at a particular model+data size.
  - canjobear 3y ago
    Maybe you can predict emergent abilities post hoc, but I recall no one predicting beforehand that, for example, a pure language model could do translation simply be giving a prompt that said “translate to French”
    - jcuenod 3y ago
      Then they're not "emergent"
      - deanishe 3y ago
        I do not think that word means what you think it means.
        jcuenod 3y ago
        Can't tell if you're responding to me or parent. But that's what I was gonna say to @canjobear...
elevatedastalt 3y ago
The research breakthroughs all happened at Google, so it's not surprising.
- janalsncm 3y ago
  RLHF was invented by OpenAI. DPO was invented at Stanford.
  - deleted 3y ago
    [deleted]
  - _t89y 3y ago
    Exactly. Google.
    - candiodari 3y ago
      And all the people that actually did the inventing promptly left Google. (yes I get there's 2 exceptions)

godelski 3y ago

What I hate about the ML community is that papers have become ad pieces now. I'm all for them releasing technical papers, but do we have to fuck up the review system for it. We don't need to railroad our research community and doing so is pretty risky. I don't buy the scale is all you need side. But if you do, doesn't matter if I'm wrong, we'll just get there when Sam gets his $7T. But if I'm right, we need new ideas if we're going to keep progressing without missing a step.

_t89y 3y ago
Domain adaptation across verticals is the only driver of innovation. Check out the NeSy computation engine. Its ``semantic parsing'' is domain parsing and its symbols are numeric. Scale if you want. It works for images. That's all that matters, right? Mapping language onto the latent space of images gets you crude semantic attributes sometimes. If you have to push for multimodal out of the gate maybe start with articulatory perception. These LVMs aren't going to cut it. ML research isn't meant to further your understanding of anything. You can't separate it from corporate interest and land grabbing. It's the same thing re-hashed every year by the same people. NLP is pretty much a subfield of IR at this point. I love how fast my comment was blacklisted to the bottom of this thread. lol
- godelski 3y ago
  I think you need to understand your audience better. Remember that most people have a very shallow understanding of DL systems. You got blog posts that talk about attention from a mathematical perspective but don't even hint at softmax tempering and just mention that there's a dot product. Or where ML researchers don't know things like why doubling batch size doesn't cut training in half or using fp16 doesn't cut memory in half. So I think toning down the language, being clearer and adding links will help you be more successful in communicating.
  - _t89y 3y ago
    Thanks for your comment. Please let me know what's not clear. Still no takers on my ML is CV comment below. Research on on-device ASR and computer vision is primarily driven by the same organizations that stand to benefit the most from it. It's nearly impossible to talk about machine learning today without talking about computer vision. Just look at the daily papers from Hugging Face or any other outlet. Machine learning is basically synonymous with computer vision and natural language processing is basically synonymous with information retrieval. Research is corporate research. With very few exceptions. Two popular lines of current ML research, multimodal and this latest neuro-symbolic re-hash, are not about furthering our understanding of what we currently can and cannot do. Not about doing science. They are about maintaining the status quo. They are about short-form video content and Google Knowledge Panels. Multimodal ML research is a vision-first enterprise. This doesn't make sense for a number of reasons. Here are two: the latent space of images and the representations therein cannot adequately capture the nuances and expressivity of language; language is a more fundamental cognitive process than vision. And what do you get for language in current ML research? How about a ``neuro-symbolic semantic parser'' that is neither a semantic parsers or symbolic. lol https://arxiv.org/abs/2402.00854v1 https://arxiv.org/abs/2402.00854v1 What's it good for? Computation graphs, domain adaptation, Google Knowledge Panels. This is a directed attack on machine learning research taking concepts that could be pursued with scientific merit, such as multimodal perception and neuro-symbolic parsing, but are instead turned into marketing hype and leveraged by the powers that be for the things that keep them in power. My audience is anyone participating in this research. Y Combinator... that's the mob of onewheels and URB-E scooters dodging human waste in the Tenderloin on their commute from Nob Hill to the Mission, right? Maybe you are referring to another audience.