6 ms·

I think this is more of an issue of smaller models with larger context aren't a magic bullet and I'm not sure anyone really paying attention thought they would

I think this is more of an issue of smaller models with larger context aren't a magic bullet and I'm not sure anyone really paying attention thought they would be. Falcon 40BN, after extensive use, is the only model that has crossed the "I'd build a product on this" threshold in open source. I expect my tune won't change there until we see larger Open Source models trained on high quality data. These are the models that will benefit from the extended context lengths. But longer context without a model capable of utilizing the context to actually do things, is yeah, unsurprisingly, not useful. There's also not really an incentive to currently make that happen in Open Source. If you can train a big model, with a unique high quality dataset you've assembled, using some novel techniques on some performant serving infrastructure, you basically have a blank check right now. Edit: On my last point, I'm going to say I was being very jaded. I think the community is doing a lot of awesome things and I know of a few efforts personally to train these larger models for open source consumption. I don't mean to be so overly negative here.

simonw 3y ago
It's pretty amazing to me that there are any open source models that have crossed that threshold - especially given that prior to the Llama release in March there were hardly any open source models that felt like they could catch up with GPT-3 at all. If you're looking for an incentive to make this happen in open source, how about the team behind MPT-30B just getting acquired for $1.3 billion. There are a LOT of well-funded groups trying to make an impact in open source models right now.
- ianbutler 3y ago
  I'm highly skeptical that people are just going to leave the secret sauce out in the open for anyone to directly compete with them, feels a bit too idealistic to me. If you look at Mosaic, their value, their secret sauce, is the infrastructure. The model, to say it cynically, was marketing. Now it's true it can be marketing and help push open source forward at the same time. More of a separate point, but I also think models are going to be fairly quickly commoditized on a couple years timescale, so the infrastructure to train a model or serve a model, or the thing built on the model is the value, but right now models are still the value because they're so new and we haven't finished (in fact we're really only starting) the race to the bottom.
  - simonw 3y ago
    Sure, OpenAI and Anthropic clearly have some secret sauce that they're holding back... but the rate at which the open source research community has been catching up is quite frankly astonishing. I would be very surprised if the closed source research teams can maintain their advantage over the next 12-24 months - especially given the number of enormous GPU clusters that are becoming available now.
    - ianbutler 3y ago
      I don't want to diminish the efforts of the community, I agree the work is enormous and the strides are pretty crazy. I honestly hope I'm wrong. I was talking with a friend and discussing that this is one of the coolest collaborative periods I've seen in tech so far. Hopefully it plays out your way and not mine :)
  - cced 3y ago
    What is secret about infrastructure?
  - bugglebeetle 3y ago
    I’ve been testing the MPT models and they’re better than all the other open source stuff I’ve tried. I’m using my task-based, one-shot prompts that are about 1-2K tokens. Seems pretty on par with GPT-3.5. Marketing or whatever, first time I’ve been at all impressed.
brucethemoose2 3y ago
Falcon was trained by a research think tank, and this is going to happen more and more as AI compute gets cheaper and the frameworks get better. We aren't going to be stuck with overpriced A100s running early PyTorch eager mode code for long. High quality "open" data is an tricky problem, but the community has already come up with some impressive datasets for finetunes, like the roleplaying collection used for Chronos and Pygmalion. I think this is going to get better too, epecially if cheap models can "distill" lower quality human datasets.
- ianbutler 3y ago
  > We aren't going to be stuck with overpriced A100s running early PyTorch eager mode code for long. I agree and am eagerly awaiting this on the hardware side and trying to participate in this where I can on the software side.
- KRAKRISMOTT 3y ago
  > Falcon was trained by a research think tank A research think tank subsidized by UAE’s oil wealth. Somebody get Masayoshi on the line, the Saudis must be hungry for deals for their vision fund.
  - senko 3y ago
    Unnecessary ad hominem. Every research tank is subsidized by someone.
    - KRAKRISMOTT 3y ago
      Very true, MIT should disclose their military affiliations too when publishing. Otherwise, they are just dishonest
      - meepmorp 3y ago
        They do. The grants themselves have requirements for acknowledging the funder in publications; most researchers also mention the funding sources for their work and projects on their website and in their CVs. People who get those grants are proud of the fact and like the money.
PeterStuer 3y ago
The phi-1 paper suggests small models trained on higher quality data hold promise as well. If confirmed that would be great news for the potential accesibility of training open source models. https://arxiv.org/abs/2306.11644 https://arxiv.org/abs/2306.11644
shoulderfake 3y ago
[dead]
cypress66 3y ago
Really? I found falcon 40b underwhelming compared to llama 65b
- 19h 3y ago
  Try falcon-40b-instruct, that’s optimized for better interaction with instructions and performs A LOT better
regularfry 3y ago
Doesn't this imply some kind of scaling law? An LLM of complexity Y will only benefit from a context length of at most Z? That feels truthy to me.
- brucethemoose2 3y ago
  It dependends on the context (hah). For, say, continuing a story, even a small LLM will benefit from a mega context even if it struggles to retrieve specific information... As long as perplexity isnt hit too much, which depends on the method. But this is more of an issue for summarization or information retrieval.

simonw 3y ago

It's pretty amazing to me that there are any open source models that have crossed that threshold - especially given that prior to the Llama release in March there were hardly any open source models that felt like they could catch up with GPT-3 at all. If you're looking for an incentive to make this happen in open source, how about the team behind MPT-30B just getting acquired for $1.3 billion. There are a LOT of well-funded groups trying to make an impact in open source models right now.

ianbutler 3y ago
I'm highly skeptical that people are just going to leave the secret sauce out in the open for anyone to directly compete with them, feels a bit too idealistic to me. If you look at Mosaic, their value, their secret sauce, is the infrastructure. The model, to say it cynically, was marketing. Now it's true it can be marketing and help push open source forward at the same time. More of a separate point, but I also think models are going to be fairly quickly commoditized on a couple years timescale, so the infrastructure to train a model or serve a model, or the thing built on the model is the value, but right now models are still the value because they're so new and we haven't finished (in fact we're really only starting) the race to the bottom.
- simonw 3y ago
  Sure, OpenAI and Anthropic clearly have some secret sauce that they're holding back... but the rate at which the open source research community has been catching up is quite frankly astonishing. I would be very surprised if the closed source research teams can maintain their advantage over the next 12-24 months - especially given the number of enormous GPU clusters that are becoming available now.
  - ianbutler 3y ago
    I don't want to diminish the efforts of the community, I agree the work is enormous and the strides are pretty crazy. I honestly hope I'm wrong. I was talking with a friend and discussing that this is one of the coolest collaborative periods I've seen in tech so far. Hopefully it plays out your way and not mine :)
- cced 3y ago
  What is secret about infrastructure?
- bugglebeetle 3y ago
  I’ve been testing the MPT models and they’re better than all the other open source stuff I’ve tried. I’m using my task-based, one-shot prompts that are about 1-2K tokens. Seems pretty on par with GPT-3.5. Marketing or whatever, first time I’ve been at all impressed.

brucethemoose2 3y ago

Falcon was trained by a research think tank, and this is going to happen more and more as AI compute gets cheaper and the frameworks get better. We aren't going to be stuck with overpriced A100s running early PyTorch eager mode code for long. High quality "open" data is an tricky problem, but the community has already come up with some impressive datasets for finetunes, like the roleplaying collection used for Chronos and Pygmalion. I think this is going to get better too, epecially if cheap models can "distill" lower quality human datasets.

ianbutler 3y ago
> We aren't going to be stuck with overpriced A100s running early PyTorch eager mode code for long. I agree and am eagerly awaiting this on the hardware side and trying to participate in this where I can on the software side.
KRAKRISMOTT 3y ago
> Falcon was trained by a research think tank A research think tank subsidized by UAE’s oil wealth. Somebody get Masayoshi on the line, the Saudis must be hungry for deals for their vision fund.
- senko 3y ago
  Unnecessary ad hominem. Every research tank is subsidized by someone.
  - KRAKRISMOTT 3y ago
    Very true, MIT should disclose their military affiliations too when publishing. Otherwise, they are just dishonest
    - meepmorp 3y ago
      They do. The grants themselves have requirements for acknowledging the funder in publications; most researchers also mention the funding sources for their work and projects on their website and in their CVs. People who get those grants are proud of the fact and like the money.

PeterStuer 3y ago

The phi-1 paper suggests small models trained on higher quality data hold promise as well. If confirmed that would be great news for the potential accesibility of training open source models. https://arxiv.org/abs/2306.11644 https://arxiv.org/abs/2306.11644

shoulderfake 3y ago

[dead]

cypress66 3y ago

Really? I found falcon 40b underwhelming compared to llama 65b

19h 3y ago
Try falcon-40b-instruct, that’s optimized for better interaction with instructions and performs A LOT better

regularfry 3y ago

Doesn't this imply some kind of scaling law? An LLM of complexity Y will only benefit from a context length of at most Z? That feels truthy to me.

brucethemoose2 3y ago
It dependends on the context (hah). For, say, continuing a story, even a small LLM will benefit from a mega context even if it struggles to retrieve specific information... As long as perplexity isnt hit too much, which depends on the method. But this is more of an issue for summarization or information retrieval.