25 ms·
The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
- amrb 4y agoI love the human factor, llama was a thing only research would of had access too, of course it leaked and everyone swarms to get it a try and we get some first class support for apple hardware!
- drcode 4y agoMeta before leak: we take safety very seriously and will only give access to responsible researchers Meta after leak: lol lmfao
- seydor 4y ago... and thanks for the open source fast implementations that we can now embed wherever we want or sell
- imjonse 4y agoThis makes is sound as if the Stanford and Berkeley teams also benefited from the leak, whereas I doubt they didn't have official access. So Alpaca/Vicuna/Koala projects would have probably happened anyway. The leak helped with popularity and demand and also somewhat positive PR for Meta, which makes me think they do not mind the leak that much.
- kmeisthax 4y agoMeta is actively trying to take down publicly available copies of LLaMA: https://github.com/github/dmca/blob/master/2023/03/2023-03-21-meta.md https://github.com/github/dmca/blob/master/2023/03/2023-03-2...
- techdragon 4y agoHaha good luck with that now… it’s in the digital ether available to all on IPFS… at worst you might have to ask around for someone to help you, but its “distributed” widely enough now I don’t think even a billionaire can put this back into the bottle.
- smoldesu 4y agoAnd in 6 months it will be outdated. So long LLaMA, and thanks for all the fish. You will be remembered as the slightly-sexier version of GPT-J that was most well-renowned for... checks clipboard ...Macbook acceleration.
- Taek 4y agoLLaMA is part of LLM history in a way that Bard will probably never be
- smoldesu 4y agoThat's kinda my point. LLaMA's entire history will be enlightening the great erudites who missed-out on real open models like GPT-Neo. Since Meta won't cooperate with the community to improve on it, it's a dead-end. It's anything but the genesis of local AI.
- jhbadger 4y agoGiven that free alternatives like Vicuna (from the University of California and CMU) are better than LLaMA, are freely and legally available for download, and are compatible with code like llama.cpp, even if every copy of LLaMA is taken down it will have no effect on the development of chatbots. It might even improve things as people who would otherwise go for the better known LLaMA will move towards these newer, better, models.
- Tepix 4y agoThey are all built on top of Llama…
- jhbadger 4y agoYes, but that doesn't matter mow. The University of California has released Vicuna as open source. It doesn't need the Llama model to be installed at this point. Nor do you need any of Meta's code to run it either as you can use llama.cpp (not created by Meta). That's the whole point of the article. It's open source now. There's nothing Meta can do.
- sp332 4y agoThis is incorrect. According to the official https://github.com/lm-sys/FastChat#vicuna-weights https://github.com/lm-sys/FastChat#vicuna-weights you need the original Llama weights before applying the Vicuna diff.
- Tepix 4y agoRight. Most of the fine-tuned models we've seen so far have been by university teams. Meta is not being very selective here. I applied for the download myself and got the links after two days (using a university email address).
- 0xDEF 4y agoThe "leak" is being portrayed as something highly subversive done by the darn 4chan hackers. Before the "leak" Meta was sending the model to pretty much anyone who claimed to be a PhD student or researcher and had a credible college email. Meta has probably been planning to release the model sooner than later. Let's hope they release it under a true open source license.
- DANmode 4y agoSeems unlikely at this point if they're machine-gun firing DMCA Notices.
- seydor 4y agoIt's risky , they dont benefit. They will prefer to keep plausible deniability
- pingwing 4y agoDoesn't surprise me that Meta wants everyone to install their code onto their machines, lol.
- whimsicalism 4y agoFeel like if they cared they would have at least watermarked the weights in some way but maybe I'm an idiot.
- drowsspa 4y agoIt sounds like that king that wanted people to overcome their aversion for potatoes. So he put armed guards around the potato fields but instructed them to be very lax and allowed the people to rob it
- irrational 4y ago> OpenAI published a detailed blog post outlining some of the principles used to ensure safety in their models. The post emphasize in areas such as privacy, factual accuracy Am I the only one amused by the phrase “factual accuracy”? How many stories have we read like the one where it tries to ghost light the guy that this year is actually last year. “Oh, your phone must be wrong too, because there is no way I could be wrong.” Though, maybe that is what factually accurate means. It is convinced that it is always factually accurate, even though it is not.
- smoldesu 4y ago> It is convinced that it is always factually accurate, even though it is not. I don't think that's true. ChatGPT (or any LLM) isn't convinced much of anything. It might present something confidently (which is what most people want) but that's a side-effect of it's programming, not an indication of how good it feels on the answer. If you reply to anything ChatGPT says with "No, you're wrong." it will try to write a new, confident and satisfying answer that responds to your assertion. LLMs will always be "wrong" because they have no distinction between fiction and fact. Everything it reads is mapped into language, not concept space or an attitude or a worldview.
- moonchrome 4y ago>Everything it reads is mapped into language, not concept space Umm I'm pretty sure it's discovered concepts through compressing text - it seems perfectly capable of generalizing concepts
- groffee 4y ago[dead]
- musha68k 4y agoWould be interesting to read some unbiased science on that. My hunch would be that any concept it might detect still depends on 2nd order patterns acquired through training? The ultimate “book smart” at best, basically. I’m not a ML scientist though, again would be interesting to read an unbiased evaluation of those aspects. Edit: downvoted, not sure why though. Isn’t this a valid question a lot of non-ML folks would ask themselves?
- adeon 4y agoI'm a bit worried the LLaMA leak will make the labs much more cautious about who they distribute models to for future projects, closing down things even more. I've had tons of fun implementing LLaMA, learning and playing around with variations like Vicuna. I learned a lot and probably wouldn't have got so interested in this space if the leak didn't happen.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- lagniappe 4y agoAn alternative interpretation was the LLaMa leak was an effort to shake or curtail the progress of ChatGPT's viral dominance at the time.
- seydor 4y ago"And as long as they’re going to steal it, we want them to steal ours. They’ll get sort of addicted, and then we’ll somehow figure out how to collect sometime in the next decade". That was ironically Bill Gates https://www.latimes.com/archives/la-xpm-2006-apr-09-fi-micropiracy9-story.html https://www.latimes.com/archives/la-xpm-2006-apr-09-fi-micro...
- rileyphone 4y agoIt took him a while to come around https://en.wikipedia.org/wiki/An_Open_Letter_to_Hobbyists https://en.wikipedia.org/wiki/An_Open_Letter_to_Hobbyists
- elcomet 4y agoThey clearly expected the leak, they distributed it very widely to researchers. The important thing is the licence, not the access: you are not allowed to use it for commercial purpose.
- whimsicalism 4y agoIt appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force?
- matrix_overload 4y agoChatGPT being an ultra-hot topic, so every article tangentially related to it gets twice the views?
- anonzzzies 4y agoIt is vastly better than anything else so far though. The rest will catch up but openai is not sleeping and they are well funded.
- jhbadger 4y agoI thought that was the case before trying Vicuna. I agree that LLaMA and Alpaca are inferior to ChatGPT but I'm really not sure Vicuna is. It even (unfortunately) copies some of ChatGPT's quirks, like getting prudish when asking it to write a love scene ("It would not be appropriate for me to write...")
- nabakin 4y agoI've tried Vicuna but it still seems inferior to ChatGPT imo. Maybe if it was applied to a version of LLaMA with a number of parameters matching GPT-4 but I'm not sure of that either
- whimsicalism 4y agoI admittedly have not interacted with Vicuna yet.
- deleted 4y ago[deleted]
- seydor 4y agoSomeone needs to legally challenge openAI on using the output of their models to train other commercial models. If web scraping is legal, then this must be legal too , even if openAI tries to curtail it. After all it was all trained on data they don't have rights to.
- EGreg 4y agoI am shocked that it speaks the way it does when it was trained on random stuff it doesn’t have rights to. They say they trained it on databases they had bought access to etc. And it seems that way. Because how does ChatGPT: 1. Do what you ask instead of continuing your instructions? 2. Use such nice and helpful language as opposed to just random average of what people say? 3. And most of all — how does it have a structure where it helpfully restates things, summarizes things, warns you against doing dangerous stuff… no way is it just continuing the most probable random Internet text!!
- seydor 4y agoIt is steered by RLHF to give helpful, nice, structured continuations. it was totally trained on random text they never paid a dime for.
- EGreg 4y agoCan you please elaborate and answer 1,2,3 directly — I’d love to find out more. Maybw links to the techniques
- seydor 4y agoI am not an expert but as others have said, the InstructGPT and Alpaca models do that https://arxiv.org/abs/2203.02155# https://arxiv.org/abs/2203.02155# https://crfm.stanford.edu/2023/03/13/alpaca.html https://crfm.stanford.edu/2023/03/13/alpaca.html
- jacquesm 4y agoThere is a lot of massaging of inputs and outputs but at the same time: that's done by tweaking the model reinforcing those parts that are desirable and suppressing those parts that are not, not by rewriting the output, though there may be filters that check for 'forbidden fruits'. And it isn't the 'random average' of what people say, that would give you junk, the whole idea is that it tries to get to something better than a random average of what people say. And by curating your sources you are of course going to help the model to achieve something a bit more sensible as well. Finally: you are probably not looking at just one model, but at a set of models.
- doubtfuluser 4y ago[Edited] Isn’t the copyright question a red-Hering? We are talking about models on the verge of generating output not distinguishable from human output. How is then a copyright breach - if it’s only caused by AI, but not by human - enforced long term? I’m not in favor of the 6 month moratorium- but seriously, we are going to face tough questions very soon - and they will shake a lot of assumptions we have. We should now really act as society to get standards in place, standards that are enforceable. Otherwise the LeCun’s et al. Will have some pretty bad impact before we start doing something. We need to work on this globally and fast to not screw it up. I’m nowadays more worried than ever about elections in the near future. Maybe we will have something like real IDs attached to content (First useful use case for crypto) or maybe we will all stop getting information from people we don’t know (yay filter bubble). I hope people smarter than me will find something.
- throwawayapples 4y ago"The training and serving code, along with an online demo, are publicly available for non-commercial use." (from Vicuna's home page.) In what universe is that "open source"?!
- justinjlynn 4y agoIt isn't.
- nailer 4y agoNothing in the article is open source. A proprietary model got leaked and there are other proprietary apps that are stupidly building on the leaked model.
- satvikpendem 4y agoOpenAssistant is an actually open source model, I'm using it now in preview and on April 15 they will release the full model. It's the closest to ChatGPT so far in my experience.
- sp332 4y agoThey also said they have no plans to release the dataset, so it's not reproduceable either.
- jdietrich 4y agohttps://www.gnu.org/philosophy/open-source-misses-the-point.html https://www.gnu.org/philosophy/open-source-misses-the-point....
- isoprophlex 4y agoIs this a tactical leak, stemming from a "commoditize your complement" strategy? Open source as a strategic weapon, without having to explain board members/shareholders/whatever that you threw around money on training an open sourced model?
- bugglebeetle 4y agoI would assume so. Meta’s ML/AI team is very strong, but they probably don’t have a comparable product offering to ChatGPT ready for public use. So instead, they bought themselves some time by letting the open source community run wild with a lesser model and eat into OpenAI’s moat.
- BulgarianIdiot 4y agoThey didn't leak it. Someone else did.
- blameitonme 4y agoI dont think theyre saying Meta AI leaked it, but they anticipated someone else will and still went ahead with it as they wanted the consequences.
- jjoonathan 4y agoIt's extremely common for a "leak" to actually be fully intentional, but the organization in question just wants plausible deniability to mitigate legal/political/popular blowback. In order to preserve plausible deniability, the leak will look genuine in all aspects that are easy to simulate. "Someone else did it" is easy to simulate. A better gauge would be to see if anyone is caught and punished. If so, it was probably a real leak.
- ericpauley 4y agoI think the key here is that, given the way that Meta distributed the model, a leak was inevitable. So while they may not have directly orchestrated a leak it must have been an intended result.
- throw778899 4y agoquite funny that the weights leaked & the senior author then left to build an llm startup
- superkuh 4y agoI've spent an embarassing amount of time since the llamas leaked playing with them, the tools to run them, and writing wrappers for them. They are technically alternatives in the sense that they're incomparably better chat bots than anything in the past. But at least for the 30B and under versions (65B is too big for me to run), no matter what fine tuning is done (alpaca, gpt4all, vicuna, etc), the llamas themselves are incomparably worse at doing useful tasks than openai's gpt3.5 models like text-davinci-003, or even the gimped gpt3.5-turbo. I wish it wasn't so, but the llamas are toys. Amazing toys, but toys. What openai is getting out of gpt3.5 (and presumbably 4, though I have no access) are actually useful responses for getting work done.
- cubefox 4y agoYour conclusion seems not to be warranted since you haven't tried out the 65B model.
- superkuh 4y agoI agree, but I think my experience is representative. So far most human people don't have the resources to be able to use 65B. And most small companies / university groups don't have the resources to fine-tune a 65B. I've talked to a couple dozen people in real time who've played with up to 30B but no one I know has the resources to run the 65B at all or fast enough to actually use and get an opinion of. None of the open source llama projects out there are using 65B in practice (despite support for it) so I think my 30B and under conclusions are applicable to the topic the article covers. I'd love to be wrong and I'm excited for this to change in the future.
- lolinder 4y agollama.cpp has great support for 65B, and I've been using it on a Linux box (I upgraded my RAM for that purpose). 64GB of RAM for a desktop is like $160 today, so it's not out of reach for most people here if they cared to. Admittedly, it's quite slow and therefore not useful for chatting or real-time applications, and it's unreliable enough in its quality that I'd like to be able to iterate faster. Definitely more of a toy at this point, at least when run on CPU.
- nailer 4y ago> Llama was not initially open-sourced, but a week after its release, the model was leaked on 4chan, sparking thousands of downloads. The author very clearly does not know what Open source is. Proprietary code that’s been leaked isn’t open source, and code that is derived from proprietary code is still proprietary. Windows had it source code leaked, that doesn’t make it open source. So did the game Portal. Not open source either. Something being leaked does not change the license.
- satvikpendem 4y agoAgreed, I don't really understand all these models coming out that are LLaMA derivatives, they are still bound by the non commercial license, making it useless for non personal use. The only one I know that's fully open source is OpenAssistant by LAION.
- alfor 4y agoWhat kind of GPU is needed to run the 65B models, can a consumer grade card do the job? Is it worth it? Or do you use cloud instances?
- AeiumNE 4y agoIt's possible to run with a cpu and then use system memory.
- alfor 4y agoDo you know what kind of speed you can expect with inference on CPU?
- MacsHeadroom 4y agoYes, around 2 tokens per second on a newer CPU with fast RAM.
- superkuh 4y agoYou can't fit a 65B even at 4bits (~50GB) on a consumer grade GPU. With an average geek's home desktop computer running 65B inference on a CPU with 64GB of system RAM you could expect about 1 token every 2 seconds.
- alchemist1e9 4y agoIs it possible to build systems with multiple GPUs to run the 65B or larger when they appear? I’m not really sure and looking for clarification from anyone who knows. My understanding is it is possible to split the layers between the GPUs so a system with 4 high end consumer GPUs might work well.
- int_19h 4y agoI tried that with RTX 4090 as the primary card and 3090 as eGPU over Thunderbolt. It works, but the inference is very slow, presumably because it has to pump all that data back and forth between the two (and Thunderbolt isn't fast enough to keep up even with 3090 by itself in games). In fact, even running 30B across two GPUs in 8-bit mode like that was slower than running it on one GPU in 4-bit. My takeaway is that if you actually want to use multiple GPUs, you need hardware that's designed to accommodate that, and most consumer-grade stuff, even high-end, is not built with two GPUs that physically large in mind.
- hrpnk 4y agoWhat's amazing to see is the effort to attempt to run the models on consumer-grade hardware, going as far as running 4-bit quantized models on phones or raspberry pi. All the debacle about mmap optimizations to llama.cpp [1] and the style these were committed to the product is a great testimony of open source. Both in the positive aspect (progress) and the negative ones (visibility affecting human judgement and collaboration). The sheer amount of experimentation is also providing a standard interface for different models that can easily be integrated and tried out. [1] https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
- deleted 4y ago[deleted]
- fpgaminer 4y agoSlightly tangential, but I had intended to start playing around with LLaMA and building some agents. I got the 4-bit versions up and running on my 3090 before I was quickly nerd snipped by a performance problem... The popular repo for quantizing and running LLaMA is the GPTQ-for-llama repo on github, which mostly copies from the GPTQ authors. The CUDA kernels are needed to support the specific kind of quantization that GPTQ does. Problem is, while those CUDA kernels are great at short prompt lengths, they fall apart at long prompt lengths. You could see people complaining about this, seeing their inference speeds slowly tanking as their chats/prompts/etc got longer. So off I went, spending the last week or so re-writing the kernels in Triton. I've now got my kernels running faster than the CUDA kernels at all sizes [0]. And I'm busily optimizing and fusing other areas. The latest MLP fusion kernels gave another couple percentage boost in performance. Yet I still haven't actually played with LLaMA and made those agents I wanted... sigh And now I'm debating diving into the Triton source code, because they removed integer unpacking instructions during one of their recent rewrites. So I had to use a hack in my kernels which causes them to use more bandwidth than they otherwise should. Think of the performance they could have with those! ... (someone please stop me...) [0] https://github.com/fpgaminer/GPTQ-triton/ https://github.com/fpgaminer/GPTQ-triton/
- Aeolun 4y agoAh well, at least you are spending your time productively.
- Nimitz14 4y agoAny recommendations for material to know to do exactly this sort of optimization work (involving triton)? I guess it's a mix of knowing computer architecture and compilers?
- fpgaminer 4y agoTriton itself is fairly "easy", at least as far as "low level optimization languages" go. It's just (restricted) python. If you know PyTorch, you can muddle your way through Triton. They have a few tutorials. Reading up on nvidia architectures, PTX, and CUDA are likely to improve your skill at Triton.
- daralthus 4y agoIs there a multimodal version yet?
- IronWolve 4y agoI dont think people know many cool things you can do out of the box with text-generation-webui interface for chat models. With extensions for voice in/out, stable diffusion images in/out, long term memory and custom npc backstories its pretty much virtual reality in a box. Just some examples of things you can do. How about create a D&D (or any RPG) game, the NPC can be the dungeon master, creating monsters/loot and rendering pics in real time. Add additional NPC characters to join your party and actually take turns, you could play solo adventures. Even play via microphone, if the voice extension gets modified, you could have each character have its own voice via tts. The extensions are opensource to make anything you want, connect it to web or any service. Have the npc chat avatars trigger on actions. You can even train models, want to create a NPC based on a book? Feed it book series, tweak the personality, and you can chat with them, or make up new stories. The training model interface is included. Or for adults you could even create a virtual partner, or any type of NPC/avatar you want. Have them text you stable diffusion pics, chat with you on sms. etc. AND, the thing is, its out NOW on github with text-generation-webui. I was able to create a D&D dungeon master with stable diffusion in about 10 minutes. I did already have stable diffusion running thou, just enabled the api. I can't wait to see how this new amazing software can take off to form new ideas, games, technology.
- kimburgess 4y agoAn SMS, email, or even physical mail only agent would be pretty interesting. With the socially accepted inherent limitations (text only, async, some level of misinterpretation expected) of those interfaces I'd wager it'd be possible to convincingly jump the uncanny valley today. Forcing to a comms path that's not just real-time chat affords a little more suspension of disbelief. Particularly with growing token counts, you can have a pen pal, virtual colleague, or friend to bounce ideas off, return to previous thoughts, and chat to in cases where a real one may not exist or be available. A little ELIZA-esque, but adaptable to different needs. My only concern is this would be be primed for misuse by people already experiencing isolation to further retreat miss opportunity to grow real social connections. Also, any semblance of privacy over those mediums would be a nightmare.
- 4y ago
- Animats 4y agoMost of this is about the base model. How's open-source "fine tuning" coming along? I'd like to add a few hundred questions and answers on a specialist topic. Recommendations?
- amrb 4y agoSeems the fine tuned models I.e. gpt4all and alpaca are trained as LoRa's. Best advice is jump in and try the demos on hugging space!
- deleted 4y ago[deleted]
- code51 4y agoMark Zuckerberg has a historic opportunity to completely reverse public perception in his favor and offer the best bet against OpenAI. He is in a great position to do this because a paradigm shift happening to search business doesn't have the heavy effect on them that Google is subject to. Yes, content and ad business is also experiencing a paradigm shift but Meta is better positioned to rework their platform and cope with AI-generated content through a driving force (not control but support) over the most popular content generation tool out there - LLaMA and any upcoming variants. Historic moment and I think Mark deep down wishes all that VR money went to AI instead.
- bitL 4y agoMeta can already start generating ads inside any video on their platform. Imagine replacing some company logos or empty portions of videos with ads.
- skocznymroczny 4y agoAnyone know if there is a way to run these models on AMD GPUs on Windows? I've been playing around with some offline models but so far it seems AMD GPU support is only for Linux so I am forced to use CPU for that which is very slow.