5 ms·
Are the weights for these T5 models published anywhere?
by anentropic 3y ago
Are the weights for these T5 models published anywhere?
- verdverm 3y agounlikely, the paper was uploaded yesterday and they make no mention of such an intention. More generally, I think we've reached the point where the big players are investing a lot of resources and trade secrets into their giga-models. Having them effectively distilled and shared may expose these secrets or erode the competitive advantage. You can fine-tune an OpenAI model, but I don't think they are going to let you download the weights. I expect the same with Google and Microsoft.
- anentropic 3y agoI dunno, if it's all a valuable trade secret why bother publishing the paper then? It's not like these models are so big it's prohibitive for anyone else to reproduce the training, we're already seeing a profusion of open 1-11B param GPT-style models... the high-performing 770M T5 described here is smaller than that
- verdverm 3y agoI guess you didn't read the paper or other comments. This process requires the mega LLMs for training, so you need one of those first
- gbasin 3y agoyou don't need access to the weights though, unless I missed something? this is akin to the various groups using GPT-3.5/4 to generate data for fine-tuning smaller models
- verdverm 3y agoSure, but that will run you up a hefty bill and add significant latency during training.
- anentropic 3y agoI skim read the paper, the key bits I saw were: > First, given an LLM and an unlabeled dataset, we prompt the LLM to generate output labels along with rationales to justify the labels > Second, we leverage these rationales in addition to the task labels to train smaller downstream models. This doesn't sound like online use of LLMs to me, but rather using them to pre-generate augmented examples. At the end of the paper is this table: Dataset Train Validation Test e-SNLI 549,367 9,842 9,824 ANLI 16,946 1,000 1,000 CQA 8,766 975 1,221 SVAMP 720 80 200 Let's do a back-of-envelope calculation: OpenAI costs "gpt-3.5-turbo: $0.002 / 1K tokens" let's say each rationale consumes 250 tokens ...that's $275 to augment 550M examples in e-SNLI dataset Even if that estimate is quite far off in terms of token count, it seems like it'll only cost a few thousand dollars at most? GPT4 costs 10x as much. It's more than I personally would ever want to pay as a curious bystander/hobbyist, but it doesn't seem like a serious "moat" to competitors.
- tontomath 3y agoI think that is about $2 for 1M tokes, so 550M*250\1M * 2 = 550*250/2 = $ 68750, that is far from $275.
- deleted 3y ago[deleted]
- anentropic 3y agoBut it's not 550M examples, it's 0.55M examples Edit: just saw I wrote 550M in my original comment, even though I was thinking of it as the correct figure (550k)
- anentropic 3y agoEdit: obviously... also have to add the cost of training the T5-large on top of the LLM fees