4 ms·

Anyscale's business model was completely disrupted by OpenAI. They are trying to shift to provide hosting/fine tuning for open source LLMs, but the model perfor

Anyscale's business model was completely disrupted by OpenAI. They are trying to shift to provide hosting/fine tuning for open source LLMs, but the model performance will get crushed by Gpt4/newer open AI models. In theory the alternative to openAI models is nice, in reality anyscale is now competing with Azure/AWS/etc to provide model hosting. Their original compute platform for running arbitrary ml workloads will become obsolete as the industry consolidates around LLMs.

deleted 3y ago
[deleted]
nightski 3y ago
I had never heard of anyscale before but I'm curious why you think the entire industry will consolidate around LLMs. Maybe you mean transformers? Either way there are a lot of problems that LLMs just aren't suited for.
- ldjkfkdsjnv 3y ago
  Transformers sure, but theres evidence that LLMs will continue to outperform models trained for specific tasks. In the past you needed: 1. a model for sentiment analysis 2. a model for summarization 3. ...other NLP tasks other tasks: 4. a model for object detection in images 5. a model for face recognition in images Whereas now the LLM does all of the above better than the previous state of the art. This will continue and eat more fields of machine learning. It will happen for images and video. I argue that it will even extend to things like time series analysis
  - losteric 3y ago
    LLMs come with a higher cost per transaction compared to dedicated models. imo the industry will consolidate around LLMs for early prototypes and as part of the workflow for building a corpus to train domain-specific models. Source: team went through this process. LLM cost ~1% unskilled human labelers, then we finetuned BERT to bring the cost down to < 0.001% (savings $120k/y compared to just using the LLM)
    - ldjkfkdsjnv 3y ago
      The cost will drop like a rock in the coming years. Its always the same pattern with technology. Theres probably 5-20 Billion in capital working on solving this problem right now.
      - losteric 3y ago
        Wouldn't cost savings also translate to simpler models? I anticipate the process of distilling knowledge to simpler models will also become much cheaper and non-technical
    - two_in_one 3y ago
      To be fair you need to add the cost of new development and estimate when it pays bask. Without that it's not obvious that custom solution is better. For small tasks it's likely not.
  - adventured 3y ago
    It'll happen just as the smartphone ate numerous accessory gadgets. The tech consolidator works by: if it reasonably can, it will. PS5? Gaming machine, 4k disc player, streaming box, etc. Smartphone? Internet, phone, texting, camera/photos, storage, shopping, identity, gaming, music, video, directions. Soon we'll add highly useful AI bots to the list. Windows? Pretty much non-stop consolidated software from the industry into itself, across decades. Also known as eating the ecosystem. GPT & Co. will eat their best plugins.
  - msp26 3y ago
    I disagree that LLMs are always better for things like classification. But the barrier to entry is so low with them that it's usually worth trying to use an LLM before training some other model.
    - ldjkfkdsjnv 3y ago
      The whole concept of AGI/super intelligence is that one model can do everything. Thats whats happening with LLMS. If you believe in AI, then you believe that specific models will become obsolete
      - deleted 3y ago
        [deleted]
      - deleted 3y ago
        [deleted]
      - spott 3y ago
        No, that isn’t the point. The point is that a model can do everything, not that it is the most cost effective or best way to do everything. There are tradeoffs with all tech, and there always will be.
  - mistrial9 3y ago
    "Law of Diminishing Returns" .. your case is weakened by the repetitive confidence of the assertions, whether it is true or not remains to be seen via actual practice
    - ldjkfkdsjnv 3y ago
      Yeah definitely is speculative, but I worked in many NLP research teams in FAANG. Worked on large scale ML systems that support systems you have heard of. All of the work we did is now obsolete (except for cost), and the scientists skill sets were completely eroded.
  - two_in_one 3y ago
    > It will happen for images and video. That's for sure. The main problem is how to put them together. There are several ways. First, tokenizing the media and inserting it in the stream. Second, a media controller which accepts verbal instructions generated by LLM. They can be combined. Interesting variation is controller with (immediate) feedback. It can be anything, for example database access. > I argue that it will even extend to things like time series analysis AFAIK transformers have been used for low level robotic control. And LLM for high level have been reported by MS and Google.
ianbutler 3y ago
OpenAI is listed as a customer on the homepage. One of the names that flits through their "trusted by" carousel.
- waleedk 3y ago
  [Author] OpenAI uses Ray, the open source version of Anyscale Platform, for training their models (video evidence: https://www.youtube.com/watch?v=CqiL5QQnN64 https://www.youtube.com/watch?v=CqiL5QQnN64). Ray (https://github.com/ray-project/ray https://github.com/ray-project/ray) is available for anyone to use. You can also use Aviary (https://github.com/ray-project/aviary https://github.com/ray-project/aviary) to serve any of those models yourself.
waleedk 3y ago
[Author] I see it differently. I see that Anyscale built some really great infrastructure for scaling AI that could be applied to many domains. What we've seen is the market shift to LLMs, and we've been able to adapt our tech to new domains. Our prices are competitive (starting at 25c per million tokens) because of the tech we've built that maximizes GPU utilization. That's what we do better than anyone else. I still think GPT-4 is best LLM out there. It's just very expensive and you don't need all that horsepower you can save bucketloads of money.
alfalfasprout 3y ago
Being in the industry it's actually quite the opposite. OpenAI had a first mover advantage. ChatGPT has a good interface and GPT4 still leads on many benchmarks. But it continues to get worse and worse and others are catching up with significantly lower parameter counts. Their moat is shrinking very quickly. Llama2 is an open source meta model, btw. Anyscale's ray makes it much easier to leverage the super fast pace of development in the OSS community (or released by large companies as OSS).
- ldjkfkdsjnv 3y ago
  I also work on FAANG NLP research teams. I'm just saying that anyscale is now competing with AWS. Supposedly their moat is shrinking, Google has yet to release a model that competes with GPT-4. Anyone using these models on a daily basis is not impressed with Llama 2, and cannot build useful systems with it, aside from a narrow vertical.
  - Hermitian909 3y ago
    Seeing this right now on the product side. I can't give away untuned GPT3.5, much less sell it. GPT4 is the absolute floor for what I can sell without setting up a pipeline.
  - alfalfasprout 3y ago
    > Anyone using these models on a daily basis is not impressed with Llama 2 Being at a FAANG adjacent company working with these models on a daily basis, Llama2 is more than sufficient for a large number of use cases. It ultimately depends what exactly you're looking for. Exposing the LLM directly as a chat interface to users? Then yeah, in many cases it won't feel sufficient. Using it for search, summarization, moderation, etc.? Then it's often more than capable. FWIW GPT-4 is not one single model as you're aware and it evolves over time. In our experience it has objectively degraded in quality over time. As for anyscale, ultimately it's more like databricks (and funny enough lots of ex databricks folks there). Anyone can just run hosted spark. Databrick's ecosystem has set them apart and it looks like that's what anyscale is building towards.

deleted 3y ago

[deleted]

nightski 3y ago

I had never heard of anyscale before but I'm curious why you think the entire industry will consolidate around LLMs. Maybe you mean transformers? Either way there are a lot of problems that LLMs just aren't suited for.

ldjkfkdsjnv 3y ago
Transformers sure, but theres evidence that LLMs will continue to outperform models trained for specific tasks. In the past you needed: 1. a model for sentiment analysis 2. a model for summarization 3. ...other NLP tasks other tasks: 4. a model for object detection in images 5. a model for face recognition in images Whereas now the LLM does all of the above better than the previous state of the art. This will continue and eat more fields of machine learning. It will happen for images and video. I argue that it will even extend to things like time series analysis
- losteric 3y ago
  LLMs come with a higher cost per transaction compared to dedicated models. imo the industry will consolidate around LLMs for early prototypes and as part of the workflow for building a corpus to train domain-specific models. Source: team went through this process. LLM cost ~1% unskilled human labelers, then we finetuned BERT to bring the cost down to < 0.001% (savings $120k/y compared to just using the LLM)
  - ldjkfkdsjnv 3y ago
    The cost will drop like a rock in the coming years. Its always the same pattern with technology. Theres probably 5-20 Billion in capital working on solving this problem right now.
    - losteric 3y ago
      Wouldn't cost savings also translate to simpler models? I anticipate the process of distilling knowledge to simpler models will also become much cheaper and non-technical
  - two_in_one 3y ago
    To be fair you need to add the cost of new development and estimate when it pays bask. Without that it's not obvious that custom solution is better. For small tasks it's likely not.
- adventured 3y ago
  It'll happen just as the smartphone ate numerous accessory gadgets. The tech consolidator works by: if it reasonably can, it will. PS5? Gaming machine, 4k disc player, streaming box, etc. Smartphone? Internet, phone, texting, camera/photos, storage, shopping, identity, gaming, music, video, directions. Soon we'll add highly useful AI bots to the list. Windows? Pretty much non-stop consolidated software from the industry into itself, across decades. Also known as eating the ecosystem. GPT & Co. will eat their best plugins.
- msp26 3y ago
  I disagree that LLMs are always better for things like classification. But the barrier to entry is so low with them that it's usually worth trying to use an LLM before training some other model.
  - ldjkfkdsjnv 3y ago
    The whole concept of AGI/super intelligence is that one model can do everything. Thats whats happening with LLMS. If you believe in AI, then you believe that specific models will become obsolete
    - deleted 3y ago
      [deleted]
    - deleted 3y ago
      [deleted]
    - spott 3y ago
      No, that isn’t the point. The point is that a model can do everything, not that it is the most cost effective or best way to do everything. There are tradeoffs with all tech, and there always will be.
- mistrial9 3y ago
  "Law of Diminishing Returns" .. your case is weakened by the repetitive confidence of the assertions, whether it is true or not remains to be seen via actual practice
  - ldjkfkdsjnv 3y ago
    Yeah definitely is speculative, but I worked in many NLP research teams in FAANG. Worked on large scale ML systems that support systems you have heard of. All of the work we did is now obsolete (except for cost), and the scientists skill sets were completely eroded.
- two_in_one 3y ago
  > It will happen for images and video. That's for sure. The main problem is how to put them together. There are several ways. First, tokenizing the media and inserting it in the stream. Second, a media controller which accepts verbal instructions generated by LLM. They can be combined. Interesting variation is controller with (immediate) feedback. It can be anything, for example database access. > I argue that it will even extend to things like time series analysis AFAIK transformers have been used for low level robotic control. And LLM for high level have been reported by MS and Google.

ianbutler 3y ago

OpenAI is listed as a customer on the homepage. One of the names that flits through their "trusted by" carousel.

waleedk 3y ago
[Author] OpenAI uses Ray, the open source version of Anyscale Platform, for training their models (video evidence: https://www.youtube.com/watch?v=CqiL5QQnN64 https://www.youtube.com/watch?v=CqiL5QQnN64). Ray (https://github.com/ray-project/ray https://github.com/ray-project/ray) is available for anyone to use. You can also use Aviary (https://github.com/ray-project/aviary https://github.com/ray-project/aviary) to serve any of those models yourself.

waleedk 3y ago

[Author] I see it differently. I see that Anyscale built some really great infrastructure for scaling AI that could be applied to many domains. What we've seen is the market shift to LLMs, and we've been able to adapt our tech to new domains. Our prices are competitive (starting at 25c per million tokens) because of the tech we've built that maximizes GPU utilization. That's what we do better than anyone else. I still think GPT-4 is best LLM out there. It's just very expensive and you don't need all that horsepower you can save bucketloads of money.

alfalfasprout 3y ago

Being in the industry it's actually quite the opposite. OpenAI had a first mover advantage. ChatGPT has a good interface and GPT4 still leads on many benchmarks. But it continues to get worse and worse and others are catching up with significantly lower parameter counts. Their moat is shrinking very quickly. Llama2 is an open source meta model, btw. Anyscale's ray makes it much easier to leverage the super fast pace of development in the OSS community (or released by large companies as OSS).

ldjkfkdsjnv 3y ago
I also work on FAANG NLP research teams. I'm just saying that anyscale is now competing with AWS. Supposedly their moat is shrinking, Google has yet to release a model that competes with GPT-4. Anyone using these models on a daily basis is not impressed with Llama 2, and cannot build useful systems with it, aside from a narrow vertical.
- Hermitian909 3y ago
  Seeing this right now on the product side. I can't give away untuned GPT3.5, much less sell it. GPT4 is the absolute floor for what I can sell without setting up a pipeline.
- alfalfasprout 3y ago
  > Anyone using these models on a daily basis is not impressed with Llama 2 Being at a FAANG adjacent company working with these models on a daily basis, Llama2 is more than sufficient for a large number of use cases. It ultimately depends what exactly you're looking for. Exposing the LLM directly as a chat interface to users? Then yeah, in many cases it won't feel sufficient. Using it for search, summarization, moderation, etc.? Then it's often more than capable. FWIW GPT-4 is not one single model as you're aware and it evolves over time. In our experience it has objectively degraded in quality over time. As for anyscale, ultimately it's more like databricks (and funny enough lots of ex databricks folks there). Anyone can just run hosted spark. Databrick's ecosystem has set them apart and it looks like that's what anyscale is building towards.