4 ms·

There is not enough written text to scale a model like ChatGPT at even 10x its current parameters, so improvements will have to come from optimizations and/or l

by epups 4y ago

There is not enough written text to scale a model like ChatGPT at even 10x its current parameters, so improvements will have to come from optimizations and/or longer training (at diminishing returns).

motoxpro 4y ago
Enough data wont really be a problem. https://us.sganalytics.com/blog/2-5-quintillion-bytes-of-data-generated-everyday-top-data-science-trends-2020/ https://us.sganalytics.com/blog/2-5-quintillion-bytes-of-dat... I wouldn't be surprised if we create more data in one year than is in the entire internet from the start
- KeplerBoy 4y ago
  quintillion bytes of social media probably have very limited value.
  - motoxpro 4y ago
    Really? All of humanity imparting everything they think and know publicly seems incredibly valuable. 100% serious.
    - BooneJS 4y ago
      If you're looking to make a tweet generator, sure. If you're looking to harness the power of facts, no.
      - rvnx 4y ago
        The main issue might be that the training set may get polluted by a lot of AI generated text (content that is not supposed to be there, but is there because AI-generated content is becoming widespread online).
        zamnos 4y ago
        Storage is cheap. With ChatGPT being the preminent system, and with OpenAI recording all of its output, it should be possible to exclude modified answers from the training data.
      - motoxpro 4y ago
        So interesting that people think facts are the only thing that's valuable. Your comment isn't a fact, does that mean it's worthless?
        ranguna 4y ago
        When looked at individually it's not worthless, but when you are talking about a corpus that significantly lacks facts, then the corpus itself is worthless for any mainstream usage other than a social network text generator.
ThrowawayTestr 4y ago
How do you know that? There are thousands upon thousands of books that haven't been digitized.
ninethirty 4y ago
GPT-4 Will Have 100 Trillion Parameters — 500x the Size of GPT-3
- ETH_start 4y ago
  That's likely a massive over-estimation. I would bet it's around 1 trillion.
- imtringued 4y ago
  That is just an internet rumor and officially denied by OpenAI.

by epups 4y ago

motoxpro 4y ago

Enough data wont really be a problem. https://us.sganalytics.com/blog/2-5-quintillion-bytes-of-data-generated-everyday-top-data-science-trends-2020/ https://us.sganalytics.com/blog/2-5-quintillion-bytes-of-dat... I wouldn't be surprised if we create more data in one year than is in the entire internet from the start

KeplerBoy 4y ago
quintillion bytes of social media probably have very limited value.
- motoxpro 4y ago
  Really? All of humanity imparting everything they think and know publicly seems incredibly valuable. 100% serious.
  - BooneJS 4y ago
    If you're looking to make a tweet generator, sure. If you're looking to harness the power of facts, no.
    - rvnx 4y ago
      The main issue might be that the training set may get polluted by a lot of AI generated text (content that is not supposed to be there, but is there because AI-generated content is becoming widespread online).
      - zamnos 4y ago
        Storage is cheap. With ChatGPT being the preminent system, and with OpenAI recording all of its output, it should be possible to exclude modified answers from the training data.
    - motoxpro 4y ago
      So interesting that people think facts are the only thing that's valuable. Your comment isn't a fact, does that mean it's worthless?
      - ranguna 4y ago
        When looked at individually it's not worthless, but when you are talking about a corpus that significantly lacks facts, then the corpus itself is worthless for any mainstream usage other than a social network text generator.

ThrowawayTestr 4y ago

How do you know that? There are thousands upon thousands of books that haven't been digitized.

ninethirty 4y ago

GPT-4 Will Have 100 Trillion Parameters — 500x the Size of GPT-3

ETH_start 4y ago
That's likely a massive over-estimation. I would bet it's around 1 trillion.
imtringued 4y ago
That is just an internet rumor and officially denied by OpenAI.