27 ms·
Hello Dolly: Democratizing the magic of ChatGPT with open models
- kapilsreed 4y ago[flagged]
- deleted 4y ago[deleted]
- Mizza 4y agoIt's immediately become difficult to untangle the licensing here. Is this safe for production use - I have no idea if I can expect a DMCA from Mark if I step out of bounds with this or other post-Alpaca models, unless I'm missing something important. Meta really botched the Llama release.
- rnosov 4y agoThis has nothing to do with facebook. The foundational model here is GPT-J which is opensource and safe to use. Sadly, it is inferior to state-of-the-art models such as LLaMA.
- Mizza 4y agoBut they're "using data from Alpaca". I don't know what that means, isn't Alpaca using data generated by ChatGPT, which isn't "clean" to use? Or data from Facebook, which isn't "clean" to use? I'm drowning.
- rnosov 4y agoThey are instruction tuning it using the dataset released by stanford-alpaca team. The dataset itself is synthetic (created using GPT-3) and somewhat noisy and in my view can be easily recreated if OpenAI ever tries to go after it (which is very unlikely). Anyway, facebook has nothing to do with anything used by this project.
- Mizza 4y agoSo, this is a "dirty" model, in that is was created by data which violated OpenAI ToS. Obviously, this kind of violation is basically fine if you're a massive corporation who the rules don't apply to, but it's a huge risk if you're a small fish.
- sebzim4500 4y agoThat's between OpenAI and the people that recorded the data. No one else needs to care.
- rnosov 4y agoToS are not the law. It would be similar to your power company claiming copyright over the code written using "their" electricity. Not going to happen. I wouldn't be too concerned.
- Spivak 4y agoEspecially when OpenAI explicitly doesn't have a claim to copyright on the model output.
- sp332 4y agoNo, but you could be banned from using OpenAI products in the future, which seems like quite a liability for a researcher or company.
- rnosov 4y agoThat would be anticompetitive practice that is actually against the law in many countries[1]. In the unlikely event of OpenAI ever engaging in such things they will be sued into oblivion. [1] https://en.wikipedia.org/wiki/Refusal_to_deal https://en.wikipedia.org/wiki/Refusal_to_deal
- iudqnolq 4y agoNo it wouldn't. Wikipedia has a crap definition that inexplicably focuses on cartels where multiple companies coordinate the refusal, which this definitely isn't. The FTC has a better definition for US law [1]. Companies routinely ban users for ToS violations. Just look at any thread about Google on here to see people complaining about it. [1]: https://www.ftc.gov/advice-guidance/competition-guidance/guide-antitrust-laws/single-firm-conduct/refusal-deal https://www.ftc.gov/advice-guidance/competition-guidance/gui...
- bilekas 4y agoI don't know the full details but Alpaca is from Stanford and only based on the LLamA (not a derivative work afaik). That said : Also Meta's licensing here https://github.com/facebookresearch/llama/blob/main/LICENSE https://github.com/facebookresearch/llama/blob/main/LICENSE Can't be sure what that license actually reffers to, the language model or just the tooling in the Git Repo. I agree its a minefield, but with Meta I would eer on the side of caution.
- bilekas 4y ago> Meta really botched the Llama release. It's no surprise really though, from what I see they recognised some way to monitize and rolled back their commitment. But this Dolly doesn't depend on Llama (unless I'M missing something), so you don't have to use it.
- leobg 4y agoWhy? Dolly had nothing to do with Llama or its weights. Besides: How would anyone ever know which model generated the output you are serving? AFAIK there is no fingerprint in any model’s output. And even if there was, it would probably be destroyed by fine tuning “over it”.
- stametseater 4y ago> AFAIK there is no fingerprint in any model’s output. It seems like there easily could be. What if some of the data they trained it on didn't exist anywhere else except in the training set, and was put there specifically for this purpose? For instance they could have taught it a few poems that don't exist anywhere else. If you can coax the LLM of unknown origin into reciting those poems back to you, you know where it came from.
- kurthr 4y agoEven easier have a small set of 8-10 character gibberish tokens it's trained on in a particular contexts (eg a non-existent poem). Then feed it one or several poems and see if a gibberish token pops out.
- eigenvalue 4y agoI think they call these canary GUIDs. If you manage to generate one from an LLM then you can conclude with certainty that the model saw that document during training.
- neilv 4y ago> Besides: How would anyone ever know which model generated the output you are serving? There's precedent for "whatever you can get away with" in tech companies, but establishing a culture of that at the start of this new big change could end up undesirable for most people. For example, it could relieve demand for more legal and sustainable ways, until it's too late. (Look at the history of digital entertainment media piracy and DRM and legislation, for example. Or look at the history of software piracy, where some big companies seem to actually want their product to be pirated, partly because it builds a bigger moat against competitors, and they can legally strongarm some of those pirates later.)
- pwendell 4y agoYes it's nuanced, but will be simplified going forward. This uses a fully open source (liberally licensed) model and we also open sourced (liberally licensed) our own training code. However, the uptraining dataset of ~50,000 samples was generated with OpenAI's text-davinci-003 model, and depending on how one interprets their terms, commercial use of the resulting model may violate the OpenAI terms of use. For that reason we are advising only noncommercial use of this model for now. The next step here is to create a set of uptraining samples that is 100% open. Stay tuned.
- Taek 4y agoAre you in touch with the OpenAssistant team? I believe they already have a more or less complete set of samples (100,000!) that were produced in an open environment and aren't encumbered by any licensing.
- pwendell 4y agoNo I haven't heard of that, we'll engage with that team. This is exactly what we need will look into it.
- babyyoda 4y agoGiven that Alpaca strictly specified that they released purely for academic use and any commercial use was prohibited given doing so would violate terms of service, I don’t see this as viable for use. Looks like marketing gimmick
- bilekas 4y agoThis is really great news and something I felt was missing from the market so far. It seems everyone wants to create `moats` or walled-gardens with some aspect of their models etc. Nice job DataBricks, nice numbers too. Looking forward to more improvements.
- detrites 4y agoThought the same until I read this: > Contact us at hello-dolly@databricks.com if you would like to get access to the trained weights.
- bilekas 4y agoThis is not an issue though, they would just be the weights used by DataBricks, there is no reason you can't add your own right ? Like giving away a website template without the demo content, it's perfectly normal.
- jppope 4y agodata transfer might actually be the problem there not something like trying to hide the model
- yieldcrv 4y agobittorrent, come on
- bilekas 4y agocontext `come on`, what is the point in sharing the weights ? Someone lay out the reason they should package the weights with this, when they're allowing you to apply your own ? This repo isn't what you think it is.
- yieldcrv 4y ago‘come on’ meaning that there is an obvious 23 year old solution to data transfer constraints called bittorrent
- sillysaurusx 4y agoInteresting. DALL-E, Dalai (https://cocktailpeanut.github.io/dalai/ https://cocktailpeanut.github.io/dalai/), and now Dolly are all pronounced the same way. It feels like there should be an xkcd for this.
- outside1234 4y agoWhat could go wrong
- mejutoco 4y agoAFAIK DALL-E is pronounced as Dalí, as in Salvador Dalí. https://en.wikipedia.org/wiki/Salvador_Dal%C3%AD https://en.wikipedia.org/wiki/Salvador_Dal%C3%AD
- chatmasta 4y agoI figured it was a reference to the Dalai Lama (which doesn't invalidate your comment, since that's also pronounced like Dalí). LLM -> Llama -> Dalai Lama
- 4ndrewl 4y agoI thought "Dalai" pronounced "Dall Eye" rhymes with "Shall I" "Dali" pronounced "Dahl eee" rhymes with "Carly"
- sillysaurusx 4y agoHmm. Is Salvador Dalí pronounced differently than Dolly or Dalai? The wikipedia page has "dah-lee" as the phonetic, and https://www.google.com/search?q=pronounce+salvador+dali https://www.google.com/search?q=pronounce+salvador+dali sounds the same as https://www.google.com/search?q=pronounce+dalai+lama https://www.google.com/search?q=pronounce+dalai+lama. So it seems like all three are identical.
- woeirua 4y agoThis is the real risk to OpenAI's business model. If it turns out that you can get most of the same outcome with drastically smaller and cheaper models, then OpenAI is going to have a hell of a time keeping customers around as it will just be a race to the bottom on price and bigger, more expensive models will lose just from a hardware cost standpoint.
- rashkov 4y agoWhat about the high quality training data that OpenAI has encoded into ChatGPT? Do these other models come close to that?
- woeirua 4y agoWhy couldn't you just use OpenAI's API to feed prompts and then take the outputs and use them to train your own model to exfiltrate the best features of GPT?
- wsgeorge 4y agoBecause it would be against their TOS, and things could look ugly, legally.
- AJ007 4y agoIt's still an open question if any of these models, trained on copyright work, will themselves be eligible for copyright protection.
- ImprobableTruth 4y agoIs this a bit? If it's illegal to train on copyrighted material, then OAI has broken the law ten times over by training GPT3. There's absolutely zero reason for them to sue, they'll just ban the responsible people.
- nickthegreek 4y agoI think their TOS forbids using the API for this. I don't think it covers the use of the web interface.
- thewataccount 4y agoI might be having a moment - but I can't find any links to a git repo, huggingface, or anything about the models/weights/checkpoints directly from the article. I just see a zip download that AFAIK also doesn't contain the weights/checkpoints. I find this a bit odd, the contents of the zip (from the gdrive preview) look like they should be in a git repo, and I assume they download the model from somewhere? GDrive usually has rate limits which I'm concerned about. If anyone from databricks reads this - are there plans to publish this on a git repo somewhere, as well as the weights/checkpoints? EDIT: Oh I just noticed > Contact us at hello-dolly@databricks.com if you would like to get access to the trained weights. This... seems odd for a article titled "Democratizing the magic of ChatGPT with open models"?
- MagicMoonlight 4y agoSo it’s another classic private only model that they’ll pull as soon as the suckers have trained it up for them
- thequadehunter 4y agoLol. This is classic ML crap. Files with no documentation, no links, multiple files with the same-ish name but no explanation for which one is what.
- nofinator 4y agoYes, the ZIP on Google Drive owned by one of their engineers is weird considering they have a pretty active GitHub presence of open source projects, though it does use an Apache license like their others. Perhaps Databricks suspected another big announcement coming soon and wanted to get this announcement out?
- dopidopHN 4y agoThanks I missed that email while skimming
- amrb 4y agoAre they pulling a Facebook, on model access?
- _9hey 4y agoI’d like some clarification of terms - when they say it takes 3 hours to train, they’re not saying from scratch are they? There’s already a huge amount of training to get to that point, isn’t that correct? If so, then it’s pretty audacious to claim they’ve democratized an LLM because the original training likely cost an epic amount of money. Then who knows how much guidance their training has incorporated, and it could have a strong undesirable viewpoint bias based on the original training.
- joshhart 4y agoThe 3 hours is the instruction fine-tuning. The base foundational model is GPT-J which was already provided by Eleuther-AI and has been around for a couple of years. Note: I work at Databricks and am familiar with this project but didn't work on it.
- Taek 4y agoDo you know why GPT-J is being used instead of NeoX or any of the other larger open source models?
- joshhart 4y ago7B is a sweet spot where you can do something with limited resources both for training and inference. Going beyond that you spill out of an A100 without tricks. We will continue iterating on this with other models.
- dragonwriter 4y agoIf fine tuning a small model, which can be run on consumer hardware once trained, provides quality results, why use a larger base model?
- htrp 4y agoTLDR: Download GPT-J-6B from Eleuther Download Alpaca Fine Tuning Code + Alpaca Examples Train for 6 hours or so. Get vaguely good RLHF model
- typon 4y agoKey point is vaguely good. Scale is still important and that manifests in the difference between gpt3.5 and gpt4 based chatgpts. It's qualitatively and quantitatively so much better in pretty much every benchmark. There is no way around the bitter lesson.
- feanaro 4y agoIsn't it the case that we literally have no clue how GPT4 and GPT3.5 are different in terms of training, given OpenAI doesn't want to disclose anything at all?
- computerex 4y agoWe don' have the details, it is true. But empirically and based on their report gpt-4 is notably better than chatgpt.
- feanaro 4y agoBetter, yes, and for that we have evidence. But is the improvement stemming simply from even more data? That's what I'm questioning.
- stevenhuang 4y agoIt's speculated it has same number of parameters, but more compute and is multi modal.
- computerex 4y agoThis paper is pretty approachable and goes over the "scaling laws" in detail: https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682 In short, yes. More data, higher quality data, more epochs on the data. That is the name of the game.
- imwithstoopid 4y agohere come the "Me Too!!" announcements from everyone trying to catch some of the energy of this new market how long until IBM, Tesla and Oracle announce Me-Too LLMs?
- bob1029 4y ago> Surprisingly, instruction-following does not seem to require the latest or largest models: our model is only 6 billion parameters, compared to 175 billion for GPT-3. We started seeing this in our testing. OpenAI's Curie model is responding very well to our fine-tuning experiments for chatbot-style interface. I am trying to keep us focused on quality of training data rather than obsessing over raw network size. Davinci (and derivatives) might turn out to be overkill for our use cases.
- deleted 4y ago[deleted]
- nothrowaways 4y ago"ChatGPT, a proprietary instruction-following model" pun intended.
- mydpy 4y agoWhat a great time to be in this field. It’s advancing so quickly!
- kvmakes 4y ago[flagged]
- jawadch93 4y ago[dead]
- gavi 4y agoIts trained on alpaca dataset which in turn was generated from open ai davinci, wondering if it is actually transferring the weights by generating content from the source model?
- Zaheer 4y agoHow hard would it be to embed this into a NPM module so anyone can use it in their servers / apps locally?
- aabajian 4y agoAnyone care to comment on why the output of these models changes so dramatically given so little Q&A training? It's a 6 billion parameter model with only 50 thousand Q&A samples. It's clear the model already "knows" the format of a Tweet (short length, attention-grabbing, contains hashtags). The model also knows stuff about language models (word2vec, tokenization), and can include entities from the question in its response (Dolly, Databricks). Yet, it just doesn't put these pieces together in the right way without the Q&A training. Edit: For kicks, I asked GPT-4 this question: https://imgur.com/a/sM4uyBn https://imgur.com/a/sM4uyBn
- Davexon 4y agoJust like in Alpaca, the answer lies in the LLama dataset where the focus is on instruction following. GPT-J knows all the answers but just now was taught to understand the questions thoroughly before writing the answer.
- pwendell 4y agoYes this was a very surprising result... that the relatively small uptraining was able to unlock so much latent knowledge in the model.
- jppope 4y agoDoes anyone else find it ironic that all these ChatGPT "clones" are popping up when OpenAi is supposed to be the ones open sourcing and sharing their work? I guess: "You Either Die A Hero, Or You Live Long Enough To See Yourself Become The Villain"?
- Taek 4y agoSam Altman has turned into a megalomaniac.
- deleted 4y ago[deleted]
- brandall10 4y agoPossibly, but it is a bit unusual that he has zero equity in the company. So it might not be for monetary reasons.
- airstrike 4y agoI still haven't fully ruled out that his consciousness has been replaced by an AGI
- JohnFen 4y ago> when OpenAi is supposed to be the ones open sourcing and sharing their work? OpenAI renounced being open source. Don't let the name fool you.
- throwaway4837 4y agoI think all of the "AI alignment" talk is mostly fearmongering. It's a cunningly smart way to get ignorant people scared enough of AI so they have no choice but to trust the OpenAI overlords when they say they need AI to be closed. Then OpenAI gets a free pass to be the gatekeeper of the model, and people stop questioning the fact that they went from Open to Closed. AI being tuned to be "safe" by an exceedingly small set of humans is the thing we should be afraid of. It's the effective altruism effect: if you bombard people enough with "safety" and "alignment" speak, they will look past the fact that you're mainly interested in being a monopoly. My bigger conspiracy theory is that Bill Gates getting behind "AI alignment" is a calculated move to get people to look past Microsoft's unilateral involvement.
- epups 4y agoI think this is cool, but it's on the range of complexity that I would expect from a personal project. When you put a whole organization behind it, I feel you could have provided something extra - better datasets? Improved weights from a ton of training?
- cuuupid 4y agoI don’t love the lack of quantitative comparison to Alpaca but a commercial model (which sounds like it’s in the works) would finally move the needle on democratizing access to LLMs. Will also commend the authors for not falling into the “LLMs can’t perform without 200B params!” fallacy. For anyone reading, 6B params is enough to train on a 3090. A PC rig for training or running inference with this would put you back maybe 4k$. The end game here is likely getting the model to perform well in millions of parameters on specific tasks. Most business uses of ChatGPT are pretty closed domain tasks, it wouldn’t be a huge step to distill this model on a specific task and get it down to 150-350M params (which is roughly BART size and can run on AWS Lambda even).
- highwaylights 4y agoI see that in its five book suggestions that it has suggested you should read Hitchhikers Guide twice. Not many humans would even get this answer correct. I am impressed.
- crosen99 4y agoFine-tuning these models reminds me of the good ol' days with tube TVs where the slightest twist of the vertical hold dial meant the difference between a clear picture and useless, dizzying, visual nonsense.
- bogwog 4y agoOpen Assistant is doing the same thing, but actually creating a dataset that isn't on questionable legal grounds by creating a gamified web app where people can contribute: https://open-assistant.io/dashboard https://open-assistant.io/dashboard I wonder how small can these models get? From 175B to 6B with comparable performance is huge, but can it go lower?
- deleted 4y ago[deleted]
- harryara 4y ago[flagged]
- pdeyhim2 4y ago[flagged]
- WuzzaTechie 4y ago[flagged]
- dataIQ____ 4y ago[flagged]
- dataIQ____ 4y ago[flagged]
- ehsjybh 4y ago[flagged]
- pbandit 4y agoThat's awesome, one step forward into democratizing data and LLMs, now, how do we get access to the weights?
- rscottsm 4y agoIt's a great example of how LLMs can be fine-tuned for specific behaviors. The gain in output quality relative to the expense of tweaking an older, relatively small model is quite impressive. Dolly may not be the Bugatti of LLM chatbots. But it illustrates that high performance is increasingly within reach for the rest of us.
- arc986 4y agoHow did they do this with $30 only?
- gackinop 4y ago[flagged]
- pavrao01 4y ago[flagged]
- pavrao01 4y ago[flagged]
- hariganeshan 4y ago[flagged]
- stevieoz 4y agoInteresting. Allows each organization to develop their own chatgpt model for a fractional cost.
- stevieoz 4y agoInteresting. This allows organization to build their own chatgpt models at a fractional cost.
- Kiou 4y agoWith 1 server? Exciting times for broader llm adoption
- derms 4y agoLooking forward to trying it out :)
- dontknowyet 4y agoI always asked myself if ChatGPT will be the only (paid) solution. Thank you Databricks for being an open source competitor here
- tjerkkostelijk 4y agothat is incredible to me, i can't wait to see the rest of the progress of this project.
- clukasik 4y agoVery cool that this powerful new tech is within reach to the masses in a platform like Databricks!
- platniumHornet 4y agoInteresting to see how Dolly has improved/leveraged an off-the-shelf older model (https://huggingface.co/EleutherAI/gpt-j-6B https://huggingface.co/EleutherAI/gpt-j-6B) with dramatic results.
- thedatageek 4y agoPretty impressive stuff !!