4 ms·

I'm not saying data isn't important or even that it's not the most important thing. I'm making a statement of the current economics. Most of the cost of buildi

I'm not saying data isn't important or even that it's not the most important thing. I'm making a statement of the current economics. Most of the cost of building a LLM comes from compute. The changes we make to the data while meaningful are dwarfed in cost by the compute required to train.

CuriouslyC 3y ago
That's not true in the case of the phi-2 model though, and if the approach of phi-2 was scaled up to a class of hyper-efficient LLMs I think it would continue to not be true while also producing SOTA results.
- famouswaffles 3y ago
  >That's not true in the case of the phi-2 model though Isn't it ? Genuinely, what did phi-2 do exactly that is economically more expensive than compute ? From what I read, phi-2's training data was gotten from: 1. Synthetic Data Generation from more performant models. phi-2 is essentially not possible without spending the compute on that better model in the first place. And inference is much cheaper than training. 2. Selecting for Text data of a certain kind e.g Textbooks Meanwhile, it's still taking 14 days to train this tiny model on 96 A100 GPUs
  - CuriouslyC 3y ago
    I'm sure the man hour cost of dataset curation on Phi-2 were higher than its training compute cost. Regarding synthetic data, I think that could technically be classified as an asset that is not consumed in the training process, and thus you could amortize its compute cost over quite a bit (or even monetize it). Given that, even if the compute for synthetic data put the total compute cost over curation costs I don't think it's on an order that would contradict my point.
    - famouswaffles 3y ago
      > I'm sure the man hour cost of dataset curation on Phi-2 were higher than its training compute cost. There's no indication of human curation. From the first paper > We annotate the quality of a small subset of these files (about 100k samples) using GPT-4: given a code snippet, the model is prompted to “determine its educational value for a student whose goal is to learn basic coding concepts”. We then use this annotated dataset to train a random forest classifier that predicts the quality of a file/sample using its output embedding from a pretrained codegen model as features. We note that unlike GPT-3.5, which we use extensively to generate synthetic content (discussed below), we use GPT-4 minimally only for annotations on the quality of a small subset of The Stack and StackOverflow samples. >Given that, even if the compute for synthetic data put the total compute cost over curation costs I don't think it's on an order that would contradict my point. Don't worry that's not even necessary. 96 A100 GPU's for 2 weeks straight is a ~50k USD Endeavour. It'll be somewhat less for the likes of Microsoft but so will inference of GPT-4, GPT-3. Even if we forget about compute costs of those more powerful models, inference costs of the generated tokens won't be anywhere near that amount.

CuriouslyC 3y ago

That's not true in the case of the phi-2 model though, and if the approach of phi-2 was scaled up to a class of hyper-efficient LLMs I think it would continue to not be true while also producing SOTA results.

famouswaffles 3y ago
>That's not true in the case of the phi-2 model though Isn't it ? Genuinely, what did phi-2 do exactly that is economically more expensive than compute ? From what I read, phi-2's training data was gotten from: 1. Synthetic Data Generation from more performant models. phi-2 is essentially not possible without spending the compute on that better model in the first place. And inference is much cheaper than training. 2. Selecting for Text data of a certain kind e.g Textbooks Meanwhile, it's still taking 14 days to train this tiny model on 96 A100 GPUs
- CuriouslyC 3y ago
  I'm sure the man hour cost of dataset curation on Phi-2 were higher than its training compute cost. Regarding synthetic data, I think that could technically be classified as an asset that is not consumed in the training process, and thus you could amortize its compute cost over quite a bit (or even monetize it). Given that, even if the compute for synthetic data put the total compute cost over curation costs I don't think it's on an order that would contradict my point.
  - famouswaffles 3y ago
    > I'm sure the man hour cost of dataset curation on Phi-2 were higher than its training compute cost. There's no indication of human curation. From the first paper > We annotate the quality of a small subset of these files (about 100k samples) using GPT-4: given a code snippet, the model is prompted to “determine its educational value for a student whose goal is to learn basic coding concepts”. We then use this annotated dataset to train a random forest classifier that predicts the quality of a file/sample using its output embedding from a pretrained codegen model as features. We note that unlike GPT-3.5, which we use extensively to generate synthetic content (discussed below), we use GPT-4 minimally only for annotations on the quality of a small subset of The Stack and StackOverflow samples. >Given that, even if the compute for synthetic data put the total compute cost over curation costs I don't think it's on an order that would contradict my point. Don't worry that's not even necessary. 96 A100 GPU's for 2 weeks straight is a ~50k USD Endeavour. It'll be somewhat less for the likes of Microsoft but so will inference of GPT-4, GPT-3. Even if we forget about compute costs of those more powerful models, inference costs of the generated tokens won't be anywhere near that amount.