11 ms·
Run LLMs at home, BitTorrent‑style
- esafak 3y agoThe first question I had was "what are the economics?" From the FAQ: Will Petals incentives be based on crypto, blockchain, etc.? No, we are working on a centralized incentive system similar to the AI Horde kudos, even though Petals is a fully decentralized system in all other aspects. We do not plan to provide a service to exchange these points for money, so you should see these incentives as "game" points designed to be spent inside our system. Petals is an ML-focused project designed for ML researchers and engineers, it does not have anything to do with finance. We decided to make the incentive system centralized because it is much easier to develop and maintain, so we can focus on developing features useful for ML researchers. https://github.com/bigscience-workshop/petals/wiki/FAQ:-Frequently-asked-questions#general https://github.com/bigscience-workshop/petals/wiki/FAQ:-Freq...
- brucethemoose2 3y ago> similar to the AI Horde kudos What they are referencing, which is super cool and (IMO) criminally underused: https://lite.koboldai.net/ https://lite.koboldai.net/ https://tinybots.net/artbot https://tinybots.net/artbot https://aihorde.net/ https://aihorde.net/ In fact, I can host a 13B-70B finetune in the afternoon if anyone on HN wants to test a particular one out: https://huggingface.co/models?sort=modified&search=70B+gguf https://huggingface.co/models?sort=modified&search=70B+gguf
- swyx 3y ago> GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp. GGUF offers numerous advantages over GGML, such as better tokenisation, and support for special tokens. It is also supports metadata, and is designed to be extensible is there a more canonical blogpost or link to learn more about the technical decisions here?
- brucethemoose2 3y agohttps://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md#historical-state-of-affairs https://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md#... It is (IMO) a necessary and good change. I just specified gguf because my 3090 cannot host a 70B model without offloading outside of exLlama's very new ~2 bit quantization. And pre quantized gguf is a much smaller download than raw fp16 for conversion.
- swyx 3y agothanks very much!
- Szpadel 3y agoif that part could be replaced with any third party server it would be a tracker in BitTorrent analogy.
- deleted 3y ago[deleted]
- seydor 3y agoIt's a shame that every decentralized projects needs to be compared to cryptocoins now
- AnthonyMouse 3y agoIt's not the comparison, it's that it's one of the things cryptocoins are actually useful for: You have people all over the world with GPUs, some of them want to pay the others for use of them, but their countries use different payment networks or the developers want to be able to automate it without forcing the users to all sign up with the same mercurial payment processor who could screw over any of the users at random.
- littlestymaar 3y ago> it's that it's one of the things cryptocoins are actually useful for It's what their proponent claim that they are useful for, yet there's no single instance of a successful blockchain project actually achieving this kind of resource-sharing goal. > You have people all over the world with GPUs, some of them want to pay the others for use of them The gigantic success of bitTorrent shows that humans as a group don't need to have monetary incentives to share their spare hardware. In fact, it's likely that trying to add money into the mix will just break the system instead of improving it: https://en.wikipedia.org/wiki/Overjustification_effect https://en.wikipedia.org/wiki/Overjustification_effect
- joshuaissac 3y ago> It's what their proponent claim that they are useful for, yet there's no single instance of a successful blockchain project actually achieving this kind of resource-sharing goal Sia and Filecoin already work in this way to for people to share storage. > In fact, it's likely that trying to add money into the mix will just break the system instead of improving it This depends on the amount of money people are willing to pay for processing power. Volunteer contributions would be reduced, but the paid contributions could make up for it if the people who want to train their model pay enough to attract more people into the system and if those people can compete with conventional commercial offerings.
- kordlessagain 3y agoThe logical conclusion is that they (the models) will eventually be linked to crypto payments though. This is where Lightning becomes important... Edit: To clarify, I'm not suggesting linking these Petal "tokens" to any payment system. I'm talking about, in general, calls to clusters of machine learning models, decentralized or not, will likely use crypto payments because it gives you auth and a means of payment. I do think Petal is a good implementation of using decentralized compute for model use and will likely be valuable long term.
- vorpalhex 3y agoI mean, I can sell you Eve or Runescape currency but we don't need any crypto to execute on it. "Gold sellers" existed well before crypto.
- AnthonyMouse 3y agoIs there an API for that which doesn't require each of the users to create a separate account on something else?
- sn0wf1re 3y agoSimilarly there have been distributed render farms for graphic design for a long time. No incentives other than higher points means your jobs are prioritized. https://www.sheepit-renderfarm.com/home https://www.sheepit-renderfarm.com/home
- beardog 3y ago>What's the motivation for people to host model layers in the public swarm? >People who run inference and fine-tuning themselves get a certain speedup if they host a part of the model locally. Some may be also motivated to "give back" to the community helping them to run the model (similarly to how BitTorrent users help others by sharing data they have already downloaded). >Since it may be not enough for everyone, we are also working on introducing explicit incentives ("bloom points") for people donating their GPU time to the public swarm. Once this system is ready, we will display the top contributors on our website. People who earned these points will be able to spend them on inference/fine-tuning with higher priority or increased security guarantees, or (maybe) exchange them for other rewards. It does seem like they want a sort of centralized token however.
- nextaccountic 3y agoCan they actually prevent people from trading petals for money though?
- swyx 3y agoso given that GGML can serve like 100 tok/s on an M2 Max, and this thing advertises 6 tok/s distributed, is this basically for people with lower end devices?
- version_five 3y agoIt's talking about 70B and 160B models. Even heavily quantized can ggml run those that fast? (I'm guessing possibly). So maybe this is for people that dont have a high end computer? I have a decent linux laptop a couple years old and there's no way I could run those models that fast. I get a few tokens per second on a quantized 7B model.
- brucethemoose2 3y agoYeah. My 3090 gets like ~5 tokens/s on 70B Q3KL. This is a good idea, as splitting up llms is actually pretty efficient with pipelined requests.
- russellbeattie 3y ago> ...lower end devices So, pretty much every other consumer PC available? Those losers.
- deleted 3y ago[deleted]
- jmorgan 3y agoThis is neat. Model weights are split into their layers and distributed across several machines who then report themselves in a big hash table when they are ready to perform inference or fine tuning "as a team" over their subset of the layers. It's early but I've been working on hosting model weights in a Docker registry for https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama. Mainly for the content addressability (Ollama will verify the correct weights are downloaded every time) and ultimately weights can be fetched by their content instead of by their name or url (which may change!). Perhaps a good next step might be to split the models by layers and store each layer independently for use cases like this (or even just for downloading + running larger models over several "local" machines).
- mkii 3y agoAh, is it possible to tone down the self-promotion? I've been seeing your comments for ollama on many LLM-related posts here. > Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity. Surely in this case it would've been possible to comment about OP's work while leaving out the free backlink to your project. Just my 0.02
- herewego 3y agoThere is nothing wrong with self-promotion if, as in this case, it is relevant to the discussion.
- brucethemoose2 3y ago> and fine‑tune them for your tasks This is the part that raised my eyebrows. Finetuning 70B is not just hard, its literally impossible without renting a very expensive cloud instance or buying a PC the price of a house, no matter how long you are willing to wait. I would absolutely contribute to a "llama training horde"
- Zetobal 3y agoAn H100 is maybe a car but not nearly close to a house...
- KomoD 3y agoMaybe not in your area, but it's very doable in other places, like where I live.
- ShamelessC 3y agoYou expect me to believe there are other places than where I live?!
- nextaccountic 3y agoIs a single H100 enough?
- brucethemoose2 3y ago80GB is enough, yeah. I'm not sure what exact LORA/quantization settings would be ideal, but check out https://github.com/OpenAccess-AI-Collective/axolotl#config https://github.com/OpenAccess-AI-Collective/axolotl#config
- ioedward 3y ago8 H100s would have enough VRAM to finetune a 70B model.
- AaronFriel 3y agoThat's true for conventional fine-tuning, but is it the case for parameter efficient fine tuning and qLORA? My understanding is that for a N billion parameter model, fine tuning can occur with a slightly-less-than-N gigabyte of VRAM GPU. For that 70B parameter model: an A100?
- nico 3y agoThis is so cool. Hopefully this will give access to thousands or millions more developers in the space
- teaearlgraycold 3y agoWould love to share my 3080 Ti, but after running the commands in the getting started guide (https://github.com/bigscience-workshop/petals/wiki/Run-Petals-server-on-Windows https://github.com/bigscience-workshop/petals/wiki/Run-Petal...) it looks like there's a dependency versioning issue: ImportError: cannot import name 'get_full_repo_name' from 'huggingface_hub' (~/.local/lib/python3.8/site-packages/huggingface_hub/__init__.py)
- behnamoh 3y agolooking at the list of contributors, way more people need to donate their GPU time for the betterment of all. maybe we finally have a good use for decentralized computing that doesn't calculate meaningless hashes for crypto, but helps the humanity by keeping these open source LLMs alive.
- Obscurity4340 3y agoThis way too nobody can copyright-cancel the LLM like OpenAI or whatever
- alextheparrot 3y agoExactly, litigation has never been applied to content delivered over BitTorrent-style networks
- Tostino 3y agoLitigation is one thing, totally erasing it from the public internet if it were hosted centrally is something else.
- Obscurity4340 3y agoAha, touché salesman :)
- judge2020 3y agoIt can cost a lot to run a GPU, especially at full load. The 4090 stock pulls 500 watts of power under full load[0], which is 12 kWh/day or just under 4380 kWh a year, or over $450 in a year assuming $0.10-$0.11/kWh for average residential rates. The only variable is whether or not training requires the same power draw as hitting it with furmark. 0: https://youtu.be/j9vC9NBL8zo?t=983 https://youtu.be/j9vC9NBL8zo?t=983
- pavelstoev 3y agoImagine someone paid you 25c/hour for 4090 compute sharing.
- __rito__ 3y agoI have used Petals at a past project. I share my GPU as well as wrote code for the project. The Petals part was abstracted away from me. I had a normal experience writing code. I don't have the project listed anywhere. Don't really know what happened to it. But, it was mainly some five or so guys spearheading the thing.
- senectus1 3y agoso how long until "tokens" are used to pay for GPU cycles.. people will stop "mining" and just donate their GPU cycles for distributed LLM usages.... in fact, if they did this so that it followed the sun so that the vast majority of it was powered by daylight Solar PV energy I wouldn't even be upset by that.
- quickthrower2 3y agoI got a lurid NSFW comment, just asking for the time (using the Colab), so I assume some people are trolling the network? Human: what is the time? The time is 12:30 PM. Human: are you sure? Yes, I am sure. The time is 12:30 PM.^</s>^<s> I'm a young {...}
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- borzunov 3y agoHi, a Petals dev here. </s> means "end of sequence" for LLMs. If a model generates it, it forgets everything and continues with an unrelated random text (I'm sorry to hear that the model generated a disturbing text in this case). Still, I doubt that malicious actors are involved here. Apparently, the Colab code snippet is just too simplified and does not handle </s> correctly. This is not the case with the full chatbot app at https://chat.petals.dev https://chat.petals.dev - you can try it out instead.
- quickthrower2 3y agoThanks for the reply. One way to guard against that would be if the LLM architecture refused to serve against just <s> as a token?
- brucethemoose2 3y agoBase llama has lots of lurid in it already.
- malwrar 3y agoHow does this defend against a malicious participant altering the output of their share of the larger computation? Even without some kind of method for e.g. producing attacker-determined network output, this system seems vulnerable to lots of nodes joining and simply returning junk results, effectively DoSing the system.
- borzunov 3y agoHi, a Petals dev here. We're developing validators that periodically go over all servers and ban the ones that return incorrect results. Additionally, clients can run data through multiple disjoint routes in the network and check that the results match. This catches frequent attackers but doesn't provide 100% protection - so we expect people to set up a _private_ swarm if they want full correctness guarantees. For example, if you don't have enough GPUs to run an LLM yourself but have some hardware owners you trust to, you can set up a private Petals swarm and jointly run the LLM on geo-distributed hardware to process your data.
- dleeftink 3y agoHow about tried and tested reputation systems for GPUs/providers to join certain swarms? Yes, this can also be gamed (and I do not wish to bring yet another scoring system into this world), but it might just work for users wanting to choose between various levels of LLM security. You might be able to even tie this into 'energy per compute unit' spent, enticing users to opt for more energy efficient offerings. Potentially, an all-round metric (or multiple metrics) for the viability of a GPU provider.
- sumo43 3y agoCool service. It's worth noting that, with quantization/QLORA, models as big as llama2-70b can be run on consumer hardware (2xRTX 3090) at acceptable speeds (~20t/s) using frameworks like llama.cpp. Doing this avoids the significant latency from parallelism schemes across different servers. p.s. from experience instruct-finetuning falcon180b, it's not worth using over llama2-70b as it's significantly undertrained.
- borzunov 3y agoHi, a Petals dev here. You're right, there's no point in using Petals if your machine has enough GPU memory to fit the model and you're okay with the quantization quality. We developed Petals for people who have less GPU memory than needed. Also, there's still a chance of larger open models being released in the future.
- brucethemoose2 3y agoAFAIK you cannot train 70B on 2x 3090, even with GPTQ/qlora. And the inference is pretty inefficient. Pooling the hardware would achieve much better GPU utilization and (theoretically) faster responses for the host's requests
- sumo43 3y agoFor training you would need more memory. As for the pooling, Theoretically yes but wouldn't latency play as much, if not a greater part in the response time here? Imagine a tensor-parallel gather where the other nodes are in different parts of the country. Here I'm assuming that Petal uses a large number of small, heterogenous nodes like consumer gpus. It might as well be something much simpler.
- brucethemoose2 3y ago> Theoretically yes but wouldn't latency play as much, if not a greater part in the response time here? For inference? Yeah, but its still better than nothing if your hardware can't run the full model, or run it extremely slowly. I think frameworks like MLC-LLM and llama.cpp kinda throw a wrench in this though, as you can get very acceptable throughput on an IGP or split across a CPU/dGPU, without that huge networking penalty. And pooling complete hosts (like AI Horde) is much cheaper. I'm not sure what the training requirements are, but ultimately throughput is all that matters for training, especially if you can "buy" training time with otherwise idle GPU time.
- vanillax 3y agoVery cool.
- wwwtyro 3y agoI love this direction. I hope that WebGPU can be leveraged for this purpose in the future so that I can feel somewhat mollified about security and to promote adoption.
- cphoover 3y agoLogo is both mesmerizing and distracting.
- timost 3y agoYou can host your own swarm of servers apparently [0]. I would be curious to have a ballpark estimate of the finetunning performance of a "private" petals cluster. [0] https://github.com/bigscience-workshop/petals/wiki/Launch-your-own-swarm https://github.com/bigscience-workshop/petals/wiki/Launch-yo...
- 0x008 3y agoI think if you run a cluster in a trusted environment it should be more efficient to use ray or something similar
- Double_a_92 3y agoAm I the only one that really really hates pages like google Colab? I never know what is going on there. Is it free? Is it running on my machine, or is it running on googles Cloud? If the latter, again is it really free?! Also everytime I still give it a try, I only get some kind of error at the end. Edit: Here we go. Literally the first line that it wanted to execute: "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. tensorflow-metadata 1.14.0 requires protobuf<4.21,>=3.20.3, but you have protobuf 4.24.3 which is incompatible."
- thathndude 3y agoI’ve always thought crowdsourcing is the future. Crowdsourcing information or compute. The fact is we have the “resources” already. It’s a matter of deployment.
- __MatrixMan__ 3y agoAre trained LLM's composable in any way? Like if you and I trust 99% of the same data, but each have 1% where we disagree, must we have two entirely separate models, or can we pool compute in the 99% case (along with the others who agree) and then create a derivative model for ourselves which covers for the differences in our trust models? I have only a rudimentary understanding of neural nets but it doesn't seem crazy that the weights could be manipulated in such a way while preserving the utility of the model. I ask because I think it would be useful to know which statements two LLMs of equal power agree on and which they disagree on. You could then map that backwards to differences in their training data (only feasible if the differences are small). If instead two LLMs of equal power represent a missed opportunity to have one of greater power, and the disagreement analysis is prohibitively expensive to do, then that's a bit of a different world.
- hnfong 3y agoSomewhat yes. See "LoRA": https://arxiv.org/abs/2106.09685 https://arxiv.org/abs/2106.09685 They're not composable in the sense that you can take these adaptation layers and arbitrarily combine them, but training different models while sharing a common base of weights is a solved problem.
- esafak 3y agoThis is called ensembling. https://blog.allenai.org/llm-blender-a-simple-ensemble-learning-framework-for-llms-9e4bc57af23e https://blog.allenai.org/llm-blender-a-simple-ensemble-learn...
- bennyschmidt 3y agoIf AI does decentralization better than crypto I'm about to laugh