3 ms·

I had never heard of anyscale before but I'm curious why you think the entire industry will consolidate around LLMs. Maybe you mean transformers? Either way t

I had never heard of anyscale before but I'm curious why you think the entire industry will consolidate around LLMs. Maybe you mean transformers? Either way there are a lot of problems that LLMs just aren't suited for.

ldjkfkdsjnv 3y ago
Transformers sure, but theres evidence that LLMs will continue to outperform models trained for specific tasks. In the past you needed: 1. a model for sentiment analysis 2. a model for summarization 3. ...other NLP tasks other tasks: 4. a model for object detection in images 5. a model for face recognition in images Whereas now the LLM does all of the above better than the previous state of the art. This will continue and eat more fields of machine learning. It will happen for images and video. I argue that it will even extend to things like time series analysis
- losteric 3y ago
  LLMs come with a higher cost per transaction compared to dedicated models. imo the industry will consolidate around LLMs for early prototypes and as part of the workflow for building a corpus to train domain-specific models. Source: team went through this process. LLM cost ~1% unskilled human labelers, then we finetuned BERT to bring the cost down to < 0.001% (savings $120k/y compared to just using the LLM)
  - ldjkfkdsjnv 3y ago
    The cost will drop like a rock in the coming years. Its always the same pattern with technology. Theres probably 5-20 Billion in capital working on solving this problem right now.
    - losteric 3y ago
      Wouldn't cost savings also translate to simpler models? I anticipate the process of distilling knowledge to simpler models will also become much cheaper and non-technical
  - two_in_one 3y ago
    To be fair you need to add the cost of new development and estimate when it pays bask. Without that it's not obvious that custom solution is better. For small tasks it's likely not.
- adventured 3y ago
  It'll happen just as the smartphone ate numerous accessory gadgets. The tech consolidator works by: if it reasonably can, it will. PS5? Gaming machine, 4k disc player, streaming box, etc. Smartphone? Internet, phone, texting, camera/photos, storage, shopping, identity, gaming, music, video, directions. Soon we'll add highly useful AI bots to the list. Windows? Pretty much non-stop consolidated software from the industry into itself, across decades. Also known as eating the ecosystem. GPT & Co. will eat their best plugins.
- msp26 3y ago
  I disagree that LLMs are always better for things like classification. But the barrier to entry is so low with them that it's usually worth trying to use an LLM before training some other model.
  - ldjkfkdsjnv 3y ago
    The whole concept of AGI/super intelligence is that one model can do everything. Thats whats happening with LLMS. If you believe in AI, then you believe that specific models will become obsolete
    - deleted 3y ago
      [deleted]
    - deleted 3y ago
      [deleted]
    - spott 3y ago
      No, that isn’t the point. The point is that a model can do everything, not that it is the most cost effective or best way to do everything. There are tradeoffs with all tech, and there always will be.
- mistrial9 3y ago
  "Law of Diminishing Returns" .. your case is weakened by the repetitive confidence of the assertions, whether it is true or not remains to be seen via actual practice
  - ldjkfkdsjnv 3y ago
    Yeah definitely is speculative, but I worked in many NLP research teams in FAANG. Worked on large scale ML systems that support systems you have heard of. All of the work we did is now obsolete (except for cost), and the scientists skill sets were completely eroded.
- two_in_one 3y ago
  > It will happen for images and video. That's for sure. The main problem is how to put them together. There are several ways. First, tokenizing the media and inserting it in the stream. Second, a media controller which accepts verbal instructions generated by LLM. They can be combined. Interesting variation is controller with (immediate) feedback. It can be anything, for example database access. > I argue that it will even extend to things like time series analysis AFAIK transformers have been used for low level robotic control. And LLM for high level have been reported by MS and Google.

ldjkfkdsjnv 3y ago

Transformers sure, but theres evidence that LLMs will continue to outperform models trained for specific tasks. In the past you needed: 1. a model for sentiment analysis 2. a model for summarization 3. ...other NLP tasks other tasks: 4. a model for object detection in images 5. a model for face recognition in images Whereas now the LLM does all of the above better than the previous state of the art. This will continue and eat more fields of machine learning. It will happen for images and video. I argue that it will even extend to things like time series analysis

losteric 3y ago
LLMs come with a higher cost per transaction compared to dedicated models. imo the industry will consolidate around LLMs for early prototypes and as part of the workflow for building a corpus to train domain-specific models. Source: team went through this process. LLM cost ~1% unskilled human labelers, then we finetuned BERT to bring the cost down to < 0.001% (savings $120k/y compared to just using the LLM)
- ldjkfkdsjnv 3y ago
  The cost will drop like a rock in the coming years. Its always the same pattern with technology. Theres probably 5-20 Billion in capital working on solving this problem right now.
  - losteric 3y ago
    Wouldn't cost savings also translate to simpler models? I anticipate the process of distilling knowledge to simpler models will also become much cheaper and non-technical
- two_in_one 3y ago
  To be fair you need to add the cost of new development and estimate when it pays bask. Without that it's not obvious that custom solution is better. For small tasks it's likely not.
adventured 3y ago
It'll happen just as the smartphone ate numerous accessory gadgets. The tech consolidator works by: if it reasonably can, it will. PS5? Gaming machine, 4k disc player, streaming box, etc. Smartphone? Internet, phone, texting, camera/photos, storage, shopping, identity, gaming, music, video, directions. Soon we'll add highly useful AI bots to the list. Windows? Pretty much non-stop consolidated software from the industry into itself, across decades. Also known as eating the ecosystem. GPT & Co. will eat their best plugins.
msp26 3y ago
I disagree that LLMs are always better for things like classification. But the barrier to entry is so low with them that it's usually worth trying to use an LLM before training some other model.
- ldjkfkdsjnv 3y ago
  The whole concept of AGI/super intelligence is that one model can do everything. Thats whats happening with LLMS. If you believe in AI, then you believe that specific models will become obsolete
  - deleted 3y ago
    [deleted]
  - deleted 3y ago
    [deleted]
  - spott 3y ago
    No, that isn’t the point. The point is that a model can do everything, not that it is the most cost effective or best way to do everything. There are tradeoffs with all tech, and there always will be.
mistrial9 3y ago
"Law of Diminishing Returns" .. your case is weakened by the repetitive confidence of the assertions, whether it is true or not remains to be seen via actual practice
- ldjkfkdsjnv 3y ago
  Yeah definitely is speculative, but I worked in many NLP research teams in FAANG. Worked on large scale ML systems that support systems you have heard of. All of the work we did is now obsolete (except for cost), and the scientists skill sets were completely eroded.
two_in_one 3y ago
> It will happen for images and video. That's for sure. The main problem is how to put them together. There are several ways. First, tokenizing the media and inserting it in the stream. Second, a media controller which accepts verbal instructions generated by LLM. They can be combined. Interesting variation is controller with (immediate) feedback. It can be anything, for example database access. > I argue that it will even extend to things like time series analysis AFAIK transformers have been used for low level robotic control. And LLM for high level have been reported by MS and Google.