7 ms·

If this is true, then: 1. Training took 21 yottaflops. When was the last time you saw the yotta- prefix for anything? 2. The training cost of GPT-4 is now onl

by xeckr 3y ago

If this is true, then: 1. Training took 21 yottaflops. When was the last time you saw the yotta- prefix for anything? 2. The training cost of GPT-4 is now only 1/3 of what it was about a year ago. It is absolutely staggering how quickly the price of training an LLM is dropping, which is great news for open source. The google memo was right about the lack of a moat.

theLiminator 3y ago
The real moat is an abundance of high quality data.
- classified 3y ago
  ... stolen without regard for copyright and licensing.
  - pas 3y ago
    Fair use!? /s
- quickthrower2 3y ago
  IMO the real moat right now is expertise / smart teams and cash.
  - hospitalJail 3y ago
    The infrastructure/training libraries already exists. I'm sure you can get people who worked at scale that can figure out how to glue things together. Reddit, twitter, etc.. raising prices is going to make it more expensive.
    - quickthrower2 3y ago
      If you are right then it just becomes who wants to throw the most cash in like a giant game of poker but where you don’t know the pot odds.
- JimmyRuska 3y ago
  Well open AI raised eye brows by crawling the internet and using everyone's data to make a commercial product One day some new startup will train on all of libgen and torrent networks, but it will be very hard to prove. You'll keep getting these gaps up in questionable morality and legality, and even openai will complain about playing fair
  - why_only_15 3y ago
    Many people train on libgen/torrent in the form of books3 (e.g. LLaMa does this).
  - fragmede 3y ago
    Google Classroom, teenager's essays, written by humans, for learning what it means to be human, and graded by humans, is a richer dataset than anything else I can think of that anyone else couldn't get their hands on.
    - londons_explore 3y ago
      An awful lot of teachers can grade a 10 page essay in about 90 seconds... Skim read it, mark out some grammar errors, assign it a grade based on the quality of the opening and closing paragraphs.
      - fragmede 3y ago
        Yup, and they're doing it the whole country over, and putting that data in to Google Classrooms for Bard to know "this is C-grade work" and "this is A-grade work". Knowing what's deemed good and bad writing is where I'm thinking this dataset shines for training LLMs.
  - pas 3y ago
    ThePile already contains some content from a torrent, and there's as lawsuit alleging that Meta has committed copyright infringement by using it. https://www.theverge.com/2023/7/9/23788741/sarah-silverman-openai-meta-chatgpt-llama-copyright-infringement-chatbots-artificial-intelligence-ai https://www.theverge.com/2023/7/9/23788741/sarah-silverman-o...
- baq 3y ago
  Yeah they have the internet from before LLMs were used for anything, so the data is not poisoned. Not unlike carbon dating becoming useless for estimating age of anything made after nuclear atmospheric tests, or low-background steel.
  - joiqj 3y ago
    You talk as if humans weren't perfectly capable of coming up with nonsense. Blogs upon blogs full of worthless pap that is there for SEO reasons have existed for like a decade already.
    - pixl97 3y ago
      And those blogs took a decade+ to make, and now in another year we'll make that much information again. Then it will be that much information in a month. Then that much pap in a day. And in the past it was still a million people making that much crap. Now it's a single "entity" making that much crap with it's own style and mistakes.
YeGoblynQueenne 3y ago
>> The training cost of GPT-4 is now only 1/3 of what it was about a year ago. It is absolutely staggering how quickly the price of training an LLM is dropping, which is great news for open source. The google memo was right about the lack of a moat. That really doesn't change anything at all. The more training large models gets cheaper, the more large corporations are able to train larger models than everyone else. Suppose the gross price of rice was $0.001 a kg. That's dirt cheap! Yet, if I had a million dollars and you had a thousand dollars, I could still buy a thousand times more rice than you.
- thifhi 3y ago
  Surely there are diminishing returns for the AI computing though? I mean, is a model with 10x the parameter count 10x better? I think it is still possible that the training costs will be irrelevant for all players at some point with this non-linear scale. Access to data is another story
  - PoignardAzur 3y ago
    It's not clear. Scaling laws still seem to hold AFAICT. Right now the bottleneck is "how big a model can you fit on an H100 TPU". It's possible that in a few years, when bigger cards come out and/or we get better at compressing models, we'll get even better models just by increasing the scale.
  - swalsh 3y ago
    10x the parameters? Maybe not in a single model, but maybe 10x the expert models has 10x the value. I'm sure there are diminishing returns eventually, but we're probably not close to that.
  - mvkel 3y ago
    It's still SO early. We are in the "640K [of memory] ought to be enough for anybody" phase of LLMs. So much more to go.
- weird-eye-issue 3y ago
  > Yet, if I had a million dollars and you had a thousand dollars, I could still buy a thousand times more rice than you. And?
londons_explore 3y ago
> The google memo was right about the lack of a moat. 5 months on, and nobody has yet beaten their result quality. I think there is a moat. Also, I think for many usecases, smarter is better. If a few cents can buy a more accurate answer, then it is always worth paying those few cents. So, while more hardware and more data can train a bigger better model, then that is the moat.
- makestuff 3y ago
  The moat is there until someone releases (or leaks) comparable training data.
  - droopyEyelids 3y ago
    And that gets more difficult every day, as previously accesible sources of data turn off their api. though google may have something up its sleeve with the corpus of google books! I have been wondering if openAI secretly pulled in scihub or zlibrary to neutralize that potential advantage.
    - londons_explore 3y ago
      Are there any stats on how many words are in google books, vs how many words are on the open web? My feeling is that the web has a lot more on it than the total of all libraries - simply because anyone can start a blog, but publishing a book requires quite some commitment.
      - droopyEyelids 3y ago
        I think you're right but I also think the text in published books would be at least an order of magnitude more valuable than the same length of text from the web
2OEH8eoCRo0 3y ago
> great news for open source. Yes, and great news for shills, bad actors, agitators, trolls, foreign intel, and propagandists. I'm impressed by the tech but terrified because for once I cannot conceive of what this means for the future. My guess is that this kills the open web and laws get passed which bury it.
- flangola7 3y ago
  Everybody is self-soothing with the idea that OpenAI's (frankly, half hearted) push for regulation is just mundane regulatory capture and profit seeking, and not the fact that it will, at best, absolutely destroy everything about the internet and technology that we've come to love and know. Should a 4chan torrent show up like LLaMA, with weights and code for a base GPT4-level model, modern society is done. Golden age over.
  - netsec_burn 3y ago
    From your perspective, how would modern society be "done" if GPT-4 was generally available? How would it be substantially different from LLaMA?
    - flangola7 3y ago
      GPT-4 is far more capable than LLaMA. Just as one area of impact - captchas would become permanently ineffective. If you're experienced in developing captchas and everything they do for us, you know the implications of that alone lead to a very dystopian internet and world. I like to answer a question with a question: if you sit and think about it, what both unintentional misuses and intentional abuses can you think of? It helps to write down a list of known abilities, then thinking up several "what if..." negative utilities or implications of each, then iterating further to see second, third, fourth order effects.
      - dragonwriter 3y ago
        > If you're experienced in developing captchas and everything they do for us What they have done, fairly overtly for a long time, is train AI to defeat captchas. That this was self-limiting was somewhat obvious.

by xeckr 3y ago

theLiminator 3y ago

The real moat is an abundance of high quality data.

classified 3y ago
... stolen without regard for copyright and licensing.
- pas 3y ago
  Fair use!? /s
quickthrower2 3y ago
IMO the real moat right now is expertise / smart teams and cash.
- hospitalJail 3y ago
  The infrastructure/training libraries already exists. I'm sure you can get people who worked at scale that can figure out how to glue things together. Reddit, twitter, etc.. raising prices is going to make it more expensive.
  - quickthrower2 3y ago
    If you are right then it just becomes who wants to throw the most cash in like a giant game of poker but where you don’t know the pot odds.
JimmyRuska 3y ago
Well open AI raised eye brows by crawling the internet and using everyone's data to make a commercial product One day some new startup will train on all of libgen and torrent networks, but it will be very hard to prove. You'll keep getting these gaps up in questionable morality and legality, and even openai will complain about playing fair
- why_only_15 3y ago
  Many people train on libgen/torrent in the form of books3 (e.g. LLaMa does this).
- fragmede 3y ago
  Google Classroom, teenager's essays, written by humans, for learning what it means to be human, and graded by humans, is a richer dataset than anything else I can think of that anyone else couldn't get their hands on.
  - londons_explore 3y ago
    An awful lot of teachers can grade a 10 page essay in about 90 seconds... Skim read it, mark out some grammar errors, assign it a grade based on the quality of the opening and closing paragraphs.
    - fragmede 3y ago
      Yup, and they're doing it the whole country over, and putting that data in to Google Classrooms for Bard to know "this is C-grade work" and "this is A-grade work". Knowing what's deemed good and bad writing is where I'm thinking this dataset shines for training LLMs.
- pas 3y ago
  ThePile already contains some content from a torrent, and there's as lawsuit alleging that Meta has committed copyright infringement by using it. https://www.theverge.com/2023/7/9/23788741/sarah-silverman-openai-meta-chatgpt-llama-copyright-infringement-chatbots-artificial-intelligence-ai https://www.theverge.com/2023/7/9/23788741/sarah-silverman-o...
baq 3y ago
Yeah they have the internet from before LLMs were used for anything, so the data is not poisoned. Not unlike carbon dating becoming useless for estimating age of anything made after nuclear atmospheric tests, or low-background steel.
- joiqj 3y ago
  You talk as if humans weren't perfectly capable of coming up with nonsense. Blogs upon blogs full of worthless pap that is there for SEO reasons have existed for like a decade already.
  - pixl97 3y ago
    And those blogs took a decade+ to make, and now in another year we'll make that much information again. Then it will be that much information in a month. Then that much pap in a day. And in the past it was still a million people making that much crap. Now it's a single "entity" making that much crap with it's own style and mistakes.

YeGoblynQueenne 3y ago

>> The training cost of GPT-4 is now only 1/3 of what it was about a year ago. It is absolutely staggering how quickly the price of training an LLM is dropping, which is great news for open source. The google memo was right about the lack of a moat. That really doesn't change anything at all. The more training large models gets cheaper, the more large corporations are able to train larger models than everyone else. Suppose the gross price of rice was $0.001 a kg. That's dirt cheap! Yet, if I had a million dollars and you had a thousand dollars, I could still buy a thousand times more rice than you.

thifhi 3y ago
Surely there are diminishing returns for the AI computing though? I mean, is a model with 10x the parameter count 10x better? I think it is still possible that the training costs will be irrelevant for all players at some point with this non-linear scale. Access to data is another story
- PoignardAzur 3y ago
  It's not clear. Scaling laws still seem to hold AFAICT. Right now the bottleneck is "how big a model can you fit on an H100 TPU". It's possible that in a few years, when bigger cards come out and/or we get better at compressing models, we'll get even better models just by increasing the scale.
- swalsh 3y ago
  10x the parameters? Maybe not in a single model, but maybe 10x the expert models has 10x the value. I'm sure there are diminishing returns eventually, but we're probably not close to that.
- mvkel 3y ago
  It's still SO early. We are in the "640K [of memory] ought to be enough for anybody" phase of LLMs. So much more to go.
weird-eye-issue 3y ago
> Yet, if I had a million dollars and you had a thousand dollars, I could still buy a thousand times more rice than you. And?

londons_explore 3y ago

> The google memo was right about the lack of a moat. 5 months on, and nobody has yet beaten their result quality. I think there is a moat. Also, I think for many usecases, smarter is better. If a few cents can buy a more accurate answer, then it is always worth paying those few cents. So, while more hardware and more data can train a bigger better model, then that is the moat.

makestuff 3y ago
The moat is there until someone releases (or leaks) comparable training data.
- droopyEyelids 3y ago
  And that gets more difficult every day, as previously accesible sources of data turn off their api. though google may have something up its sleeve with the corpus of google books! I have been wondering if openAI secretly pulled in scihub or zlibrary to neutralize that potential advantage.
  - londons_explore 3y ago
    Are there any stats on how many words are in google books, vs how many words are on the open web? My feeling is that the web has a lot more on it than the total of all libraries - simply because anyone can start a blog, but publishing a book requires quite some commitment.
    - droopyEyelids 3y ago
      I think you're right but I also think the text in published books would be at least an order of magnitude more valuable than the same length of text from the web

2OEH8eoCRo0 3y ago

> great news for open source. Yes, and great news for shills, bad actors, agitators, trolls, foreign intel, and propagandists. I'm impressed by the tech but terrified because for once I cannot conceive of what this means for the future. My guess is that this kills the open web and laws get passed which bury it.

flangola7 3y ago
Everybody is self-soothing with the idea that OpenAI's (frankly, half hearted) push for regulation is just mundane regulatory capture and profit seeking, and not the fact that it will, at best, absolutely destroy everything about the internet and technology that we've come to love and know. Should a 4chan torrent show up like LLaMA, with weights and code for a base GPT4-level model, modern society is done. Golden age over.
- netsec_burn 3y ago
  From your perspective, how would modern society be "done" if GPT-4 was generally available? How would it be substantially different from LLaMA?
  - flangola7 3y ago
    GPT-4 is far more capable than LLaMA. Just as one area of impact - captchas would become permanently ineffective. If you're experienced in developing captchas and everything they do for us, you know the implications of that alone lead to a very dystopian internet and world. I like to answer a question with a question: if you sit and think about it, what both unintentional misuses and intentional abuses can you think of? It helps to write down a list of known abilities, then thinking up several "what if..." negative utilities or implications of each, then iterating further to see second, third, fourth order effects.
    - dragonwriter 3y ago
      > If you're experienced in developing captchas and everything they do for us What they have done, fairly overtly for a long time, is train AI to defeat captchas. That this was self-limiting was somewhat obvious.