2 ms·

I'm sure the man hour cost of dataset curation on Phi-2 were higher than its training compute cost. Regarding synthetic data, I think that could technically be

I'm sure the man hour cost of dataset curation on Phi-2 were higher than its training compute cost. Regarding synthetic data, I think that could technically be classified as an asset that is not consumed in the training process, and thus you could amortize its compute cost over quite a bit (or even monetize it). Given that, even if the compute for synthetic data put the total compute cost over curation costs I don't think it's on an order that would contradict my point.

famouswaffles 3y ago
> I'm sure the man hour cost of dataset curation on Phi-2 were higher than its training compute cost. There's no indication of human curation. From the first paper > We annotate the quality of a small subset of these files (about 100k samples) using GPT-4: given a code snippet, the model is prompted to “determine its educational value for a student whose goal is to learn basic coding concepts”. We then use this annotated dataset to train a random forest classifier that predicts the quality of a file/sample using its output embedding from a pretrained codegen model as features. We note that unlike GPT-3.5, which we use extensively to generate synthetic content (discussed below), we use GPT-4 minimally only for annotations on the quality of a small subset of The Stack and StackOverflow samples. >Given that, even if the compute for synthetic data put the total compute cost over curation costs I don't think it's on an order that would contradict my point. Don't worry that's not even necessary. 96 A100 GPU's for 2 weeks straight is a ~50k USD Endeavour. It'll be somewhat less for the likes of Microsoft but so will inference of GPT-4, GPT-3. Even if we forget about compute costs of those more powerful models, inference costs of the generated tokens won't be anywhere near that amount.

2 ms·

I'm sure the man hour cost of dataset curation on Phi-2 were higher than its training compute cost. Regarding synthetic data, I think that could technically be

by CuriouslyC 3y ago

famouswaffles 3y ago
> I'm sure the man hour cost of dataset curation on Phi-2 were higher than its training compute cost. There's no indication of human curation. From the first paper > We annotate the quality of a small subset of these files (about 100k samples) using GPT-4: given a code snippet, the model is prompted to “determine its educational value for a student whose goal is to learn basic coding concepts”. We then use this annotated dataset to train a random forest classifier that predicts the quality of a file/sample using its output embedding from a pretrained codegen model as features. We note that unlike GPT-3.5, which we use extensively to generate synthetic content (discussed below), we use GPT-4 minimally only for annotations on the quality of a small subset of The Stack and StackOverflow samples. >Given that, even if the compute for synthetic data put the total compute cost over curation costs I don't think it's on an order that would contradict my point. Don't worry that's not even necessary. 96 A100 GPU's for 2 weeks straight is a ~50k USD Endeavour. It'll be somewhat less for the likes of Microsoft but so will inference of GPT-4, GPT-3. Even if we forget about compute costs of those more powerful models, inference costs of the generated tokens won't be anywhere near that amount.