6 ms·

Great to see such free to use and self-hostable models, but it's said that open now means only that. One cannot replicate this model without access to the train

Great to see such free to use and self-hostable models, but it's said that open now means only that. One cannot replicate this model without access to the training data.

ru552 3y ago
There's a large amount of liability in disclosing your training data.
- imjonse 3y ago
  Calling the model 'truly open' without is not technically correct though.
  - Lacerda69 3y ago
    It's open enough for all practical purposes IMO.
    - nicklecompte 3y ago
      It's not "open enough" to do an honest evaluation of these systems by constructing adversarial benchmarks.
    - imjonse 3y ago
      As open as an executable binary that you are allowed to download and use for free.
      - bevekspldnw 2y ago
        In the age of SaaS I’ll take it. It’s not like I have a few million dollars to pay for training even if I had all the code and data.
- londons_explore 2y ago
  I expect we'll see some AI companies in the future throwing away the training dataset. Maybe some have already. During a court case, the other side can demand discovery over your training dataset, for example to see if it contains a particular copyrighted work. But if you've already deleted the dataset, you're far more likely to win any case against you that hinges on what was in the dataset if the plaintiff can't even prove their work was included. And you can argue that the dataset was very expensive to store (which is true), and therefore deleted shortly after training was complete. You have no obligation to keep something for the benefit of potential future plaintiffs you aren't even aware of yet.
generalizations 3y ago
...And a massive pile of cash/compute hardware.
- kiney 3y ago
  not that massive, we're talking six figures. There was a blogpost about this a while back on the startpage of HN.
  - htrp 3y ago
    for finetuning or parameter training from scratch?
    - kiney 3y ago
      from scratch: https://research.myshell.ai/jetmoe https://research.myshell.ai/jetmoe
      - kaibee 3y ago
        That's for an 8B model.
        cptcobalt 3y ago
        This is over trivializing it, but there isn't much more inherent complexity in training an 8B or larger model other than more money, more compute, more data, more time. Overall, the principles are similar.
        lostmsu 3y ago
        Assuming linear growth to number of parameters that's 7.5 figures instead of 6 for 8x22B model.
  - moffkalast 3y ago
    6 figures are a massive pile of cash.

Great to see such free to use and self-hostable models, but it's said that open now means only that. One cannot replicate this model without access to the training data.

ru552 3y ago

There's a large amount of liability in disclosing your training data.

imjonse 3y ago
Calling the model 'truly open' without is not technically correct though.
- Lacerda69 3y ago
  It's open enough for all practical purposes IMO.
  - nicklecompte 3y ago
    It's not "open enough" to do an honest evaluation of these systems by constructing adversarial benchmarks.
  - imjonse 3y ago
    As open as an executable binary that you are allowed to download and use for free.
    - bevekspldnw 2y ago
      In the age of SaaS I’ll take it. It’s not like I have a few million dollars to pay for training even if I had all the code and data.
londons_explore 2y ago
I expect we'll see some AI companies in the future throwing away the training dataset. Maybe some have already. During a court case, the other side can demand discovery over your training dataset, for example to see if it contains a particular copyrighted work. But if you've already deleted the dataset, you're far more likely to win any case against you that hinges on what was in the dataset if the plaintiff can't even prove their work was included. And you can argue that the dataset was very expensive to store (which is true), and therefore deleted shortly after training was complete. You have no obligation to keep something for the benefit of potential future plaintiffs you aren't even aware of yet.

generalizations 3y ago

...And a massive pile of cash/compute hardware.

kiney 3y ago
not that massive, we're talking six figures. There was a blogpost about this a while back on the startpage of HN.
- htrp 3y ago
  for finetuning or parameter training from scratch?
  - kiney 3y ago
    from scratch: https://research.myshell.ai/jetmoe https://research.myshell.ai/jetmoe
    - kaibee 3y ago
      That's for an 8B model.
      - cptcobalt 3y ago
        This is over trivializing it, but there isn't much more inherent complexity in training an 8B or larger model other than more money, more compute, more data, more time. Overall, the principles are similar.
        lostmsu 3y ago
        Assuming linear growth to number of parameters that's 7.5 figures instead of 6 for 8x22B model.
- moffkalast 3y ago
  6 figures are a massive pile of cash.