5 ms·

I might be having a moment - but I can't find any links to a git repo, huggingface, or anything about the models/weights/checkpoints directly from the article.

I might be having a moment - but I can't find any links to a git repo, huggingface, or anything about the models/weights/checkpoints directly from the article. I just see a zip download that AFAIK also doesn't contain the weights/checkpoints. I find this a bit odd, the contents of the zip (from the gdrive preview) look like they should be in a git repo, and I assume they download the model from somewhere? GDrive usually has rate limits which I'm concerned about. If anyone from databricks reads this - are there plans to publish this on a git repo somewhere, as well as the weights/checkpoints? EDIT: Oh I just noticed > Contact us at hello-dolly@databricks.com if you would like to get access to the trained weights. This... seems odd for a article titled "Democratizing the magic of ChatGPT with open models"?

MagicMoonlight 4y ago
So it’s another classic private only model that they’ll pull as soon as the suckers have trained it up for them
thequadehunter 4y ago
Lol. This is classic ML crap. Files with no documentation, no links, multiple files with the same-ish name but no explanation for which one is what.
nofinator 4y ago
Yes, the ZIP on Google Drive owned by one of their engineers is weird considering they have a pretty active GitHub presence of open source projects, though it does use an Apache license like their others. Perhaps Databricks suspected another big announcement coming soon and wanted to get this announcement out?
dopidopHN 4y ago
Thanks I missed that email while skimming
amrb 4y ago
Are they pulling a Facebook, on model access?
- dragonwriter 4y ago
  I think they are dodging unclear legal issues surrounding certain steps of the model-building process while being as open as possible with the components given that constraint, allowing downstream users to make their own legal risk vs. effort choices.
  - amrb 4y ago
    Given the hardware/energy need to train it be nice, to have a legal document that said something like this model has no warranty, it may be a break through machine or a hand grenade. Use at you own risk!
  - pwendell 4y ago
    Yes, this.
- thewataccount 4y ago
  From what I can tell they're fine-tuning EleutherAI's GPT-J. Alpaca was made to fine-tune LLaMa, however they also released their dataset they used to do this, and it looks like Dolly is this dataset applied to GPT-J, and does not use LLaMa itself.
slimsag 4y ago
The README also says this: > This fine-tunes the [GPT-J 6B](https://huggingface.co/EleutherAI/gpt-j-6B https://huggingface.co/EleutherAI/gpt-j-6B) model on the [Alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca https://huggingface.co/datasets/tatsu-lab/alpaca) dataset using a Databricks notebook. > Please note that while GPT-J 6B is Apache 2.0 licensed, the Alpaca dataset is licensed under Creative Commons NonCommercial (CC BY-NC 4.0). ...so, this cannot be used for commercial purposes
- dragonwriter 4y ago
  > ...so, this cannot be used for commercial purposes. The legal relation between models and training data sets seems murky; of course, with the build tooling, you can also substitute in another instruction-following training set if you want to avoid licensing issues with the Alpaca set, whereas if you aren't concerbed with them, you can just blaze ahead.
- thewataccount 4y ago
  > ...so, this cannot be used for commercial purposes Can't they also release the fine-tuned weights as non-commercial as well?
- dwallin 4y ago
  Essentially every model worth anything has been trained on a unfathomably large amount of data under copyright, with every possible licensing scheme you could imagine, under the assumption that it is fair use. While you can argue that it's all built on a house of cards (and a court may well agree with you) it's kind of arbitrary to draw a line here.
  - Taek 4y ago
    Fair use via "this is completely impossible to regulate so you might as well embrace it"
  - judge2020 4y ago
    > under the assumption that it is fair use. No, because you as a human looking at "art" over your lifetime and learning from it is not "fair use" of the copyright, it's no-use at all. This is the crux of every argument for both for language models and AI Art models, that their tools are learning how to draw, learning what styles and characteristics of input art correspond the most with words, and creating art with that knowledge just like any other human, not simply collaging together different pieces of art.
pwendell 4y ago
Full source code is up here now: https://github.com/databrickslabs/dolly https://github.com/databrickslabs/dolly Sorry it took us a day to get the external repo setup.
- thewataccount 4y ago
  Awesome thank you! Was the Alpaca dataset being licensed as non-commercial only the reason you aren't releasing the weights? Is it possible to just release them under the same license?
  - pwendell 4y ago
    Yes the issue is that some of the training data is arguably tainted with some noncommercial license (it's nuanced, discussed below in my comment). We are releasing weights to people who request but we just wanted to have an email request flow so that we can make sure people know it's just for noncommercial purposes. Working on a model without this issue. Certainly our goal is totally open models anyone can use for anything.
    - thewataccount 4y ago
      Understandable, thank you for the response! I've been a bit jaded by the "open/democratizing ai" stuff and then having companies stiff us at actually making it open - but not wanting to be the first to litigate these new types of issues ml brings is very understandable. Question - Would you consider benchmarking a single 4090 for your training? While training in a few hours with 8x A100's is impressive, myself and I think others are curious how that translates to consumer hardware. IMO running/fine-tuning on consumer hardware is the ultimate endgame for all ai models.
      - robwwilliams 4y ago
        Look forward to a response. We are heading toward a 6X Bizon 4090 system as a test bed. https://bizon-tech.com/bizon-zx5500.html https://bizon-tech.com/bizon-zx5500.html
m3affan 4y ago
Databricks is on a roll

MagicMoonlight 4y ago

So it’s another classic private only model that they’ll pull as soon as the suckers have trained it up for them

thequadehunter 4y ago

Lol. This is classic ML crap. Files with no documentation, no links, multiple files with the same-ish name but no explanation for which one is what.

nofinator 4y ago

Yes, the ZIP on Google Drive owned by one of their engineers is weird considering they have a pretty active GitHub presence of open source projects, though it does use an Apache license like their others. Perhaps Databricks suspected another big announcement coming soon and wanted to get this announcement out?

dopidopHN 4y ago

Thanks I missed that email while skimming

amrb 4y ago

Are they pulling a Facebook, on model access?

dragonwriter 4y ago
I think they are dodging unclear legal issues surrounding certain steps of the model-building process while being as open as possible with the components given that constraint, allowing downstream users to make their own legal risk vs. effort choices.
- amrb 4y ago
  Given the hardware/energy need to train it be nice, to have a legal document that said something like this model has no warranty, it may be a break through machine or a hand grenade. Use at you own risk!
- pwendell 4y ago
  Yes, this.
thewataccount 4y ago
From what I can tell they're fine-tuning EleutherAI's GPT-J. Alpaca was made to fine-tune LLaMa, however they also released their dataset they used to do this, and it looks like Dolly is this dataset applied to GPT-J, and does not use LLaMa itself.

slimsag 4y ago

The README also says this: > This fine-tunes the [GPT-J 6B](https://huggingface.co/EleutherAI/gpt-j-6B https://huggingface.co/EleutherAI/gpt-j-6B) model on the [Alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca https://huggingface.co/datasets/tatsu-lab/alpaca) dataset using a Databricks notebook. > Please note that while GPT-J 6B is Apache 2.0 licensed, the Alpaca dataset is licensed under Creative Commons NonCommercial (CC BY-NC 4.0). ...so, this cannot be used for commercial purposes

dragonwriter 4y ago
> ...so, this cannot be used for commercial purposes. The legal relation between models and training data sets seems murky; of course, with the build tooling, you can also substitute in another instruction-following training set if you want to avoid licensing issues with the Alpaca set, whereas if you aren't concerbed with them, you can just blaze ahead.
thewataccount 4y ago
> ...so, this cannot be used for commercial purposes Can't they also release the fine-tuned weights as non-commercial as well?
dwallin 4y ago
Essentially every model worth anything has been trained on a unfathomably large amount of data under copyright, with every possible licensing scheme you could imagine, under the assumption that it is fair use. While you can argue that it's all built on a house of cards (and a court may well agree with you) it's kind of arbitrary to draw a line here.
- Taek 4y ago
  Fair use via "this is completely impossible to regulate so you might as well embrace it"
- judge2020 4y ago
  > under the assumption that it is fair use. No, because you as a human looking at "art" over your lifetime and learning from it is not "fair use" of the copyright, it's no-use at all. This is the crux of every argument for both for language models and AI Art models, that their tools are learning how to draw, learning what styles and characteristics of input art correspond the most with words, and creating art with that knowledge just like any other human, not simply collaging together different pieces of art.

pwendell 4y ago

Full source code is up here now: https://github.com/databrickslabs/dolly https://github.com/databrickslabs/dolly Sorry it took us a day to get the external repo setup.

thewataccount 4y ago
Awesome thank you! Was the Alpaca dataset being licensed as non-commercial only the reason you aren't releasing the weights? Is it possible to just release them under the same license?
- pwendell 4y ago
  Yes the issue is that some of the training data is arguably tainted with some noncommercial license (it's nuanced, discussed below in my comment). We are releasing weights to people who request but we just wanted to have an email request flow so that we can make sure people know it's just for noncommercial purposes. Working on a model without this issue. Certainly our goal is totally open models anyone can use for anything.
  - thewataccount 4y ago
    Understandable, thank you for the response! I've been a bit jaded by the "open/democratizing ai" stuff and then having companies stiff us at actually making it open - but not wanting to be the first to litigate these new types of issues ml brings is very understandable. Question - Would you consider benchmarking a single 4090 for your training? While training in a few hours with 8x A100's is impressive, myself and I think others are curious how that translates to consumer hardware. IMO running/fine-tuning on consumer hardware is the ultimate endgame for all ai models.
    - robwwilliams 4y ago
      Look forward to a response. We are heading toward a 6X Bizon 4090 system as a test bed. https://bizon-tech.com/bizon-zx5500.html https://bizon-tech.com/bizon-zx5500.html

m3affan 4y ago

Databricks is on a roll