3 ms·

> Note the model is trained on data generated by GPT-4. Is it? I couldn't find that in the page, and can't easily access the links. The previous paper used 1B

by IanCal 3y ago

> Note the model is trained on data generated by GPT-4. Is it? I couldn't find that in the page, and can't easily access the links. The previous paper used 1B tokens from GPT-3.5 > It's probably orders of magnitude more expensive to generate the data at current API prices. If you're generating a billion tokens, you might do better with dedicated instances, iirc they used to say if you were doing more than a few hundred million a month dedicated things were cheaper.

alecco 3y ago
It's in the Phi-1.5 technical paper. For phi-2 they bumped the number of tokens to 1.4 T and for sure most of it is generated, like previous models.
- IanCal 3y ago
  I might be missing it but I can't find where it says how the data was generated, it mostly refers back to the previous paper which started they used 3.5 I'd not be too surprised but I can't find anything in the technical report paper saying they're using 4 specifically.
  - alecco 3y ago
    Read the first paper "Textbooks Are All You Need". > We annotate the quality of a small subset of these files (about 100k samples) using GPT-4: given a code snippet, the model is prompted to “determine its educational value for a student whose goal is to learn basic coding concepts”.
    - IanCal 3y ago
      Yes, they didn't use GPT-4 to generate data. They use GPT-3.5 to generate 1B tokens of synthetic data. They used GPT-4 to annotate data to train a classifier to filter human written code. The quote directly after yours: > We then use this annotated dataset to train a random forest classifier that predicts the quality of a file/sample using its output embedding from a pretrained codegen model as features. We note that unlike GPT-3.5, which we use extensively to generate synthetic content (discussed below), we use GPT-4 minimally only for annotations on the quality of a small subset of The Stack and StackOverflow samples. We thus view our usage of GPT-4 as merely a way to avoid tedious human-annotation efforts

by IanCal 3y ago

alecco 3y ago

It's in the Phi-1.5 technical paper. For phi-2 they bumped the number of tokens to 1.4 T and for sure most of it is generated, like previous models.

IanCal 3y ago
I might be missing it but I can't find where it says how the data was generated, it mostly refers back to the previous paper which started they used 3.5 I'd not be too surprised but I can't find anything in the technical report paper saying they're using 4 specifically.
- alecco 3y ago
  Read the first paper "Textbooks Are All You Need". > We annotate the quality of a small subset of these files (about 100k samples) using GPT-4: given a code snippet, the model is prompted to “determine its educational value for a student whose goal is to learn basic coding concepts”.
  - IanCal 3y ago
    Yes, they didn't use GPT-4 to generate data. They use GPT-3.5 to generate 1B tokens of synthetic data. They used GPT-4 to annotate data to train a classifier to filter human written code. The quote directly after yours: > We then use this annotated dataset to train a random forest classifier that predicts the quality of a file/sample using its output embedding from a pretrained codegen model as features. We note that unlike GPT-3.5, which we use extensively to generate synthetic content (discussed below), we use GPT-4 minimally only for annotations on the quality of a small subset of The Stack and StackOverflow samples. We thus view our usage of GPT-4 as merely a way to avoid tedious human-annotation efforts