5 ms·

1. research talent. There's not actually that many people in the world that can adequately fine-tune a large cutting-edge model, and far fewer that can explore

1. research talent. There's not actually that many people in the world that can adequately fine-tune a large cutting-edge model, and far fewer that can explore less mainstream paths to produce value from models. Only way to get good researchers is to have name-brand leaders, like a top ML professor. 2. data. Can't do anything custom without good training data! How to get this varies widely across industry. Partnerships with established non-tech companies are a common path, which tend to rely on the network and background of founders. Even with both those things it's not easy to outcompete a large, motivated company in the same space, like a FAANG. They have the researchers, they have the data and partnerships, so the way to beat them is to move quickly and hope their A- and B-teams are working on something else.

alex1212 3y ago
Research / talent is how Mistral was able to justify its valuation at 3 weeks old. Pre product, pre anything.
- claytonjy 3y ago
  Yes, totally. Not something anyone reading this is going to replicate!
  - alex1212 3y ago
    Interestingly enough I think there is lack of talent on the investment side of things too. Very few investors have the right skillsets in their teams to be able to do deep technical due diligence required for true AI solutions.
  - Palmik 3y ago
    > Not something anyone reading this is going to replicate! They were L7/L6 ML researchers/eng at FAANG, I'd bet there are quite a few people like that lurking here.
    - claytonjy 3y ago
      I think there's rather more to it than that. Two of these guys are on the Llama paper; the hype and momentum from that is surely responsible for a huge chunk of their valuation. If you take the big LLM-relevant papers, most of the folks with this kind of profile are already off doing some kind of startup. The Mistral folks have impeccable timing, but are leaving FAANG somewhat late compared to their peers.
      - Palmik 3y ago
        Definitely. Just to be clear, my comment wasn't to diminish the Mistral folks, they are certainly a very impressive group, but rather to contest your implication about the audience here.
bugglebeetle 3y ago
> There's not actually that many people in the world that can adequately fine-tune a large cutting-edge model If you know how to run a Python script, you can fine-tune a LLama model: https://huggingface.co/blog/llama2#fine-tuning-with-peft https://huggingface.co/blog/llama2#fine-tuning-with-peft
- polygamous_bat 3y ago
  That's roughly akin to saying "if you have a wrench, you can fix a car", and posting a link to a YouTube tutorial with it.
  - bugglebeetle 3y ago
    No, not really. The script does the vast majority of the work. The only challenges here would be knowing how to use Google Colab and formatting/splitting your training and test data. That’s the computer science equivalent of adding wiper fluid to your car.
    - claytonjy 3y ago
      It's true we've made this easy, and that's awesome, but this is not what AI startups do; this is what other companies experimenting with AI do.
      - bugglebeetle 3y ago
        Ok, but that’s goalpost shifting. We’ve gone from there are not many people who can fine-tune a model (demonstrably untrue) to there are not many people who can do {???} that AI startups do. It’s unclear what is this special AI startup thing you’re referring to, but given that various fine-tuning strategies, like QLORA, emerged out of the open source community, this also seems unlikely to be true.
        claytonjy 3y ago
        Yeah, that's fair. I could have been more precise about what an advantage research talent can be. As an example, the startup-employed AI researchers I know had already PEFT'd llama2 within a day or two of the weights being out, determined that wasn't good enough for their needs, and began a deeper fine tuning effort. That's not something I can do, nor can most people, and it's a serious competitive advantage for those who can. It's a rather different interpretation of "can adequately fine-tune" than "can follow a tutorial". When I think "AI startup", I think of the places where these people work. I don't think there's many of those people, and I think their presence is a big competitive advantage for their employers.

alex1212 3y ago

Research / talent is how Mistral was able to justify its valuation at 3 weeks old. Pre product, pre anything.

claytonjy 3y ago
Yes, totally. Not something anyone reading this is going to replicate!
- alex1212 3y ago
  Interestingly enough I think there is lack of talent on the investment side of things too. Very few investors have the right skillsets in their teams to be able to do deep technical due diligence required for true AI solutions.
- Palmik 3y ago
  > Not something anyone reading this is going to replicate! They were L7/L6 ML researchers/eng at FAANG, I'd bet there are quite a few people like that lurking here.
  - claytonjy 3y ago
    I think there's rather more to it than that. Two of these guys are on the Llama paper; the hype and momentum from that is surely responsible for a huge chunk of their valuation. If you take the big LLM-relevant papers, most of the folks with this kind of profile are already off doing some kind of startup. The Mistral folks have impeccable timing, but are leaving FAANG somewhat late compared to their peers.
    - Palmik 3y ago
      Definitely. Just to be clear, my comment wasn't to diminish the Mistral folks, they are certainly a very impressive group, but rather to contest your implication about the audience here.

bugglebeetle 3y ago

> There's not actually that many people in the world that can adequately fine-tune a large cutting-edge model If you know how to run a Python script, you can fine-tune a LLama model: https://huggingface.co/blog/llama2#fine-tuning-with-peft https://huggingface.co/blog/llama2#fine-tuning-with-peft

polygamous_bat 3y ago
That's roughly akin to saying "if you have a wrench, you can fix a car", and posting a link to a YouTube tutorial with it.
- bugglebeetle 3y ago
  No, not really. The script does the vast majority of the work. The only challenges here would be knowing how to use Google Colab and formatting/splitting your training and test data. That’s the computer science equivalent of adding wiper fluid to your car.
  - claytonjy 3y ago
    It's true we've made this easy, and that's awesome, but this is not what AI startups do; this is what other companies experimenting with AI do.
    - bugglebeetle 3y ago
      Ok, but that’s goalpost shifting. We’ve gone from there are not many people who can fine-tune a model (demonstrably untrue) to there are not many people who can do {???} that AI startups do. It’s unclear what is this special AI startup thing you’re referring to, but given that various fine-tuning strategies, like QLORA, emerged out of the open source community, this also seems unlikely to be true.
      - claytonjy 3y ago
        Yeah, that's fair. I could have been more precise about what an advantage research talent can be. As an example, the startup-employed AI researchers I know had already PEFT'd llama2 within a day or two of the weights being out, determined that wasn't good enough for their needs, and began a deeper fine tuning effort. That's not something I can do, nor can most people, and it's a serious competitive advantage for those who can. It's a rather different interpretation of "can adequately fine-tune" than "can follow a tutorial". When I think "AI startup", I think of the places where these people work. I don't think there's many of those people, and I think their presence is a big competitive advantage for their employers.