4 ms·

8 passes so 50B tokens seen. Still, any other >50% HumanEval model is much much bigger https://twitter.com/SebastienBubeck/status/1671326369626853376 https://tw

8 passes so 50B tokens seen. Still, any other >50% HumanEval model is much much bigger https://twitter.com/SebastienBubeck/status/1671326369626853376 https://twitter.com/SebastienBubeck/status/16713263696268533... Now this wouldn't be possible without the high quality synthetic dataset produced by GPT(1B tokens) but this is more evidence in line with Tiny Stories (https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759). That is, LLMs only need to be so big (both data and parameters) to learn the total sum of human knowledge (and deal with trash data).

skepticATX 3y ago
This result puts more emphasis on data quality, which might actually be problematic long term though? The vast majority of human knowledge is not represented in high quality textbooks. It’s not clear from these results but the paper seems to at least imply that the importance of the synthetic data is to “unlock” the pre-training data. My takeaway as a non-expert is that this is a good result for small and efficient models focused on well-defined domains, but a neutral or maybe even a bad result for a model that displays general intelligence.
- airgapstopgap 3y ago
  The vast majority of human data can be refined into something approximating high quality textbooks, which is what happened here too.
  - skepticATX 3y ago
    Only 1/7 of the data they are using is synthetic though. The other 6/7 was just filtered from a larger dataset, not refined in any way.
  - WJW 3y ago
    I wonder if that is true. Intuitively it seems to me that there are probably many areas of knowledge that cannot be reliably summarized into a textbook. Most subjective experiences, for example. Qualia have been notoriously difficult to describe too, even if the might be absolute. Even for fields that lend themselves well to being converted to textbook format there is often a tradeoff between accuracy and conciseness. The more you refine, the more nuance you throw away. It seems like a Very Hard Problem (tm) to know which data are superfluous and which are not, especially at scale.
    - cjbprime 3y ago
      Well, used to be a Very Hard Problem. Now you can just test it: train a model on the textbook and ask it to solve many problem challenges that it hasn't seen before. After you get a score for the model trained on the whole textbook, try removing each sentence in the book in turn. If removing that particular sentence decreased the test scores then keep it in, else throw it away.
      - WJW 3y ago
        I don't think that is sufficient: - Perhaps the model can be improved by adding sentences, but which ones? There is a potentially infinite amount of sentences to add and no good way to select them - Your proposed method handwaves the method of acquiring "many problem challenges that it hasn't seen before", which just moves the problem. Constructing a problem set containing the full range of potential problems to solve is again a Very Hard Problem. The second problem is not just theoretical either: see https://sitn.hms.harvard.edu/flash/2020/racial-discrimination-in-face-recognition-technology/ https://sitn.hms.harvard.edu/flash/2020/racial-discriminatio... for example, where a facial recognition algorithm performed much worse on non-white women because the training set didn't contain enough pictures of them. AI software is notorious for finding this kind of loophole and for overfitting itself for the training set rather than for the real world.
    - mejutoco 3y ago
      Very interesting point. I wonder: are subjective experiences knowledge? Not scientific knowledge. One one side they are part of who we are. On the other side, same as an airplane does not copy a bird 100%, it makes sense that to make a machine "think" we would feed it rational content. That is, content that follows the scientific method.
      - WJW 3y ago
        I suppose that depends on what type of AI you are trying to build. One that is used to help design airplanes can mostly get by with pure scientific knowledge, although it still needs to know about softer phenomena like claustrophobia and personal space to understand that packing in humans as close as physically possible is not desired. On the other end of the AI spectrum, an AI being trained to be a childrens toy, old people companion, nursing robot or even psychologist would be incredibly deficient if it didn't understand emotions. I don't think using purely content that follows the scientific method would be sufficient for such an AI.
IanCal 3y ago
The speed of training is really interesting to me too. Llambda labs rent out 8 A100s for ~$9-12/hour. Total training cost there would be ~$850-1200. Another ~$2000 for the GPT training data, some more given there would be input tokens too. Very approachable prices. > Now this wouldn't be possible without the high quality synthetic dataset produced by GPT(1B tokens) Just a note that this is GPT3.5 (I assume turbo?).

skepticATX 3y ago

This result puts more emphasis on data quality, which might actually be problematic long term though? The vast majority of human knowledge is not represented in high quality textbooks. It’s not clear from these results but the paper seems to at least imply that the importance of the synthetic data is to “unlock” the pre-training data. My takeaway as a non-expert is that this is a good result for small and efficient models focused on well-defined domains, but a neutral or maybe even a bad result for a model that displays general intelligence.

airgapstopgap 3y ago
The vast majority of human data can be refined into something approximating high quality textbooks, which is what happened here too.
- skepticATX 3y ago
  Only 1/7 of the data they are using is synthetic though. The other 6/7 was just filtered from a larger dataset, not refined in any way.
- WJW 3y ago
  I wonder if that is true. Intuitively it seems to me that there are probably many areas of knowledge that cannot be reliably summarized into a textbook. Most subjective experiences, for example. Qualia have been notoriously difficult to describe too, even if the might be absolute. Even for fields that lend themselves well to being converted to textbook format there is often a tradeoff between accuracy and conciseness. The more you refine, the more nuance you throw away. It seems like a Very Hard Problem (tm) to know which data are superfluous and which are not, especially at scale.
  - cjbprime 3y ago
    Well, used to be a Very Hard Problem. Now you can just test it: train a model on the textbook and ask it to solve many problem challenges that it hasn't seen before. After you get a score for the model trained on the whole textbook, try removing each sentence in the book in turn. If removing that particular sentence decreased the test scores then keep it in, else throw it away.
    - WJW 3y ago
      I don't think that is sufficient: - Perhaps the model can be improved by adding sentences, but which ones? There is a potentially infinite amount of sentences to add and no good way to select them - Your proposed method handwaves the method of acquiring "many problem challenges that it hasn't seen before", which just moves the problem. Constructing a problem set containing the full range of potential problems to solve is again a Very Hard Problem. The second problem is not just theoretical either: see https://sitn.hms.harvard.edu/flash/2020/racial-discrimination-in-face-recognition-technology/ https://sitn.hms.harvard.edu/flash/2020/racial-discriminatio... for example, where a facial recognition algorithm performed much worse on non-white women because the training set didn't contain enough pictures of them. AI software is notorious for finding this kind of loophole and for overfitting itself for the training set rather than for the real world.
  - mejutoco 3y ago
    Very interesting point. I wonder: are subjective experiences knowledge? Not scientific knowledge. One one side they are part of who we are. On the other side, same as an airplane does not copy a bird 100%, it makes sense that to make a machine "think" we would feed it rational content. That is, content that follows the scientific method.
    - WJW 3y ago
      I suppose that depends on what type of AI you are trying to build. One that is used to help design airplanes can mostly get by with pure scientific knowledge, although it still needs to know about softer phenomena like claustrophobia and personal space to understand that packing in humans as close as physically possible is not desired. On the other end of the AI spectrum, an AI being trained to be a childrens toy, old people companion, nursing robot or even psychologist would be incredibly deficient if it didn't understand emotions. I don't think using purely content that follows the scientific method would be sufficient for such an AI.

IanCal 3y ago

The speed of training is really interesting to me too. Llambda labs rent out 8 A100s for ~$9-12/hour. Total training cost there would be ~$850-1200. Another ~$2000 for the GPT training data, some more given there would be input tokens too. Very approachable prices. > Now this wouldn't be possible without the high quality synthetic dataset produced by GPT(1B tokens) Just a note that this is GPT3.5 (I assume turbo?).