4 ms·

Is a truly open source 2 trillion token model even possible? Even if Meta released this under Apache 2.0, there's the sticky question of the training data lice

Is a truly open source 2 trillion token model even possible? Even if Meta released this under Apache 2.0, there's the sticky question of the training data licenses.

echelon 3y ago
> Even if Meta released this under Apache 2.0, there's the sticky question of the training data licenses. You need to pay Disney royalties out of every paycheck because you watched Toy Story and Star Wars. These movies updated your own neural weights.
- moonchrome 3y ago
  If you recreated those works from memory yeah you would be subject to copyright. There's a whole set of rules around fair use and derivative work.
  - echelon 3y ago
    Where is AI guilty of reproducing Star Wars verbatim, then? If the AI has seen Star Wars and that's enough to find it liable, then you should be too. If the AI has seen Star Wars to understand science fiction and modern culture, then it's no different from us or any other artist.
    - brucethemoose2 3y ago
      If a human recites large chunks of Star Wars verbatim, and then sells that copy as a service, thats certainly enough to find the person liable. YouTube zaps videos that contain too much copyrighted stuff for this very reason.
      - echelon 3y ago
        My point is that nobody is using AI to wholesale reproduce works and that training on copyrighted works isn't violating anyone's copyright.
- rossy 3y ago
  This is always such a reductive position. There's a little bit of evidence that ANNs learn and create in a similar way to biological neural networks, and a lot of evidence that they're quite alien to us, including being way better than the average human at reproducing parts of their training set verbatim (eg. Copilot reproducing `Q_rsqrt` verbatim, including the magic numbers, formatting and comments.) So it's not obvious that training an ANN should be treated the same as a human watching a movie under copyright law, which is designed to protect the human creative process only.
  - echelon 3y ago
    > a lot of evidence that they're quite alien to us It's an exact moral equivalent to learning. I wouldn't expect PyTorch to be running the citric acid cycle under the hood, and that shouldn't matter. > Copilot reproducing `Q_rsqrt` verbatim, including the magic numbers, formatting and comments. Humans can recite lots of things verbatim. Song lyrics, Shakespeare, digits of pi, Bible verses, you name it. We're so early, and the models are only one little engineering trick away from not doing this unless requested. We'll probably be able to tackle this at the level of the weights themselves. > copyright law, which is designed to protect the human creative process only. This is the point. Does AI contribute positively to the economy and the human condition? Does it outperform the old status quo? We're going to find out, and I'd expect for the law to fit to the outcome it views as optimal. I'm betting that the law will view this similar to industrialization. That more people will be enabled through AI than inconvenienced, that more works will be created, and that more money will move and flow than before.

Is a truly open source 2 trillion token model even possible? Even if Meta released this under Apache 2.0, there's the sticky question of the training data licenses.

echelon 3y ago

> Even if Meta released this under Apache 2.0, there's the sticky question of the training data licenses. You need to pay Disney royalties out of every paycheck because you watched Toy Story and Star Wars. These movies updated your own neural weights.

moonchrome 3y ago
If you recreated those works from memory yeah you would be subject to copyright. There's a whole set of rules around fair use and derivative work.
- echelon 3y ago
  Where is AI guilty of reproducing Star Wars verbatim, then? If the AI has seen Star Wars and that's enough to find it liable, then you should be too. If the AI has seen Star Wars to understand science fiction and modern culture, then it's no different from us or any other artist.
  - brucethemoose2 3y ago
    If a human recites large chunks of Star Wars verbatim, and then sells that copy as a service, thats certainly enough to find the person liable. YouTube zaps videos that contain too much copyrighted stuff for this very reason.
    - echelon 3y ago
      My point is that nobody is using AI to wholesale reproduce works and that training on copyrighted works isn't violating anyone's copyright.
rossy 3y ago
This is always such a reductive position. There's a little bit of evidence that ANNs learn and create in a similar way to biological neural networks, and a lot of evidence that they're quite alien to us, including being way better than the average human at reproducing parts of their training set verbatim (eg. Copilot reproducing `Q_rsqrt` verbatim, including the magic numbers, formatting and comments.) So it's not obvious that training an ANN should be treated the same as a human watching a movie under copyright law, which is designed to protect the human creative process only.
- echelon 3y ago
  > a lot of evidence that they're quite alien to us It's an exact moral equivalent to learning. I wouldn't expect PyTorch to be running the citric acid cycle under the hood, and that shouldn't matter. > Copilot reproducing `Q_rsqrt` verbatim, including the magic numbers, formatting and comments. Humans can recite lots of things verbatim. Song lyrics, Shakespeare, digits of pi, Bible verses, you name it. We're so early, and the models are only one little engineering trick away from not doing this unless requested. We'll probably be able to tackle this at the level of the weights themselves. > copyright law, which is designed to protect the human creative process only. This is the point. Does AI contribute positively to the economy and the human condition? Does it outperform the old status quo? We're going to find out, and I'd expect for the law to fit to the outcome it views as optimal. I'm betting that the law will view this similar to industrialization. That more people will be enabled through AI than inconvenienced, that more works will be created, and that more money will move and flow than before.