13 ms·
Petals: Run 100B+ language models at home bit-torrent style
- 29athrowaway 4y agoPeople nowadays will use 100 GB in VRAM to run a model that taught itself how to do quicksort.
- tommica 4y agoWhat an interesting concept - also makes me wonder how BitTorrent could be used for more de-centralizing of data, while keeping it accessible on-demand.
- _joel 4y agoSounds just like https://ipfs.tech/ https://ipfs.tech/
- jazzyjackson 4y agorather, ipfs sounds just like bittorrent
- neiman 4y agoIt has a unique identifier of data (their main feature), a naming system and some other features, that makes it quite different than bittorrent (as far as two p2p sharing data networks can be different, obviously).
- ShamelessC 4y agoRight, but one works and has widespread adoption. The other does not. And they certainly cover similar ground.
- Alifatisk 4y agoI’ve always wanted to download an ipfs-node and run it on my pc in the background but I’m worried if it will wear down my hard drives?
- simongray 4y agoWhat an fascinating concept. I guess this won't be useful for any kind of realtime feedback system, though?
- colordrops 4y agoWhy not?
- simongray 4y agoHow would one make a reliable realtime system that depends entirely on unknown network conditions? Perhaps inside a closed network it is possible.
- _joel 4y agoThat's orthoganal to a realtime system. You can infer at a fair speed so realtime would be possible.
- simongray 4y agoGuarantees are not orthogonal to realtime feedback, they are essential. If I write a query, it is not irrelevant whether it takes 1 second or 1 minute to return at any given moment. You write that speed can be inferred, but the analogy that was used here is BitTorrent—and my experience with BitTorrent tells me that it certainly cannot be inferred.
- _joel 4y agoIf you read the article text and the response from the dev then yes, inference can happen at 1/s or if parallelised, more. I'm not sure what your parameters are for a realtime system. If you're talking about network reliability, that's a different issue. Yes it can infer quickly, can it do it reliably is another matter.
- borzunov 4y agoA Petals dev here. It is not real-time, but we think the speed of ~1 token/sec may be enough for some interactive apps such as chat bots (especially, if you show tokens to a user once they are generated). You can try one at http://chat.petals.ml http://chat.petals.ml (heads-up: it may be laggy right now due to lots of HN users trying out the system). Of course, you could do better if you have enough high-end GPUs to host the entire model yourself (3x A100 or 8x 3090). But if you don't, 1 token/sec is much faster than what you get with other existing methods.
- Labo333 4y agoI would love for most of the Blockchain trend to be converted in efforts towards BitTorrent style projects. Distributed File Sharing or computation without the whole tokenomics that, while interesting, creates too much attention from scammers.
- comboy 4y agoOne can't scale without the other.
- deleted 4y ago[deleted]
- Pilottwave 4y agoWhile I understand where you are coming from, this argument is basically "money attracts attention from scammers". As a counter I would say "Money attracts attention, period" and attention is an important resource to foster growth. Decentralized tech would never be where it is today if it weren't for investor attention and the potential for gains. We just have to separate the wheat from the chaff, and remain vigilant for bad actors.
- 2fast4you 4y ago*blockchain tech would never be where it is today… I’ll bet blockchain is only as popular as it is because of the money. But other forms of decentralization like Mastodon or Matrix are pretty separate from the whole crypto sphere
- dist1ll 4y agoMatrix is separate from the crypto sphere, because it solves a different problem. Federated platforms appeal to the privacy-oriented "f** big tech" mindset, which is pretty common in the hacker & FOSS crowds. I'd put it in the same category as VPNs, E2E messengers and TOR.
- Oras 4y agoI think it is a great start, and I suppose there will be many iterations to ensure fair usage. It would be interesting to reach a point to be similar to docker where you don't need to load each layer again and you only need your specific layer. The shared models layers would be already loaded, and running multiple models at once would consume less GPU memory.
- tjoff 4y ago> Fine-tuning and inference up to 10x faster than offloading What is "offloading" in this context?
- taink 4y agoIt's mentioned in their paper: https://arxiv.org/pdf/2209.01188.pdf https://arxiv.org/pdf/2209.01188.pdf Several recent works aim to democratize LLMs by “offloading” model parameters to slower but cheaper memory (RAM or SSD), then running them on the accelerator layer by layer (Pudipeddi et al., 2020; Ren et al., 2021). This method allows running LLMs with a single low-end accelerator by loading parameters from RAM justin-time for each forward pass. Offloading can be efficient for processing many tokens in parallel, but it has inher- ently high latency: for example, generating one to- ken with BLOOM-176B takes at least 5.5 seconds for the fastest RAM offloading setup and 22 sec- onds for the fastest SSD offloading. In addition, many computers do not have enough RAM to of- fload 175B parameters.
- dpflan 4y agoIs a mobile device / edge device a possible participant / source of resources?
- borzunov 4y agoOffloading is another popular method for running large LMs when you don't have the GPU memory to fit the entire model. Imagine you have an A100 GPU with 80 GB memory and want to generate text with BLOOM, a 70-block transformer model with ~2.5 GB of weights per block. For each token, offloading will load the first 1/3 of the model (~27 blocks) from RAM/SSD to your GPU memory, run a forward pass through them, then free the memory and load the next 2/3, and so on. It turns out, Petals is faster than offloading even though it communicates over the Internet (possible, with servers far away from you). That's because Petals only sends NN activations between servers (a small amount of data), while offloading copies hundreds of GB of NN weights to GPU VRAM to generate each new token.
- madisonmay 4y ago
- rightbyte 4y agoCan anyone get it to write code? It just says it has written the code to the file system when I prompt it.
- borzunov 4y agoYou can switch the chat bot (http://chat.petals.ml http://chat.petals.ml) to the "few-shot mode" and provide a couple of "task description & code" examples. Then you can add a new task description and it'll respond with code. The underlying LM, BLOOM, had a few programming languages in its dataset, so it works at least with Python and C++.
- technocratius 4y agoThis sounds (very narrowly) similar to the Enigma network, a blockchain-based technology that can be used for fully encrypted multi-party computation (MPC). It was one of the earlier blockchain projects that actually had an interesting use case and technology in this quite "overhyped" space. They rebranded to the Secret network [0] a few years back and somehow I don't find this use case/promise back nowadays...website screams all of the Web3 BS buzzwords it seems :( [0] https://scrt.network https://scrt.network
- ShamelessC 4y agoWell yeah, the whole movement is founded in deliberate ignorance of all the existing, _working_ solutions we already have. Also, apparently none of them watched the HBO comedy Silicon Valley.
- hatenberg 4y agoJust shows nobody actually needs blockchain even for decentralized systems
- Kerbonut 4y agoAnyone participating in the swarm is able to potentially log the tokens that get processed by their node. Obviously a security concern. Is there any way to implement homomorphic computing to securely process the tokens?
- borzunov 4y agoA Petals dev here. Indeed, the public swarm should not be used for any kind of sensitive data (we have warnings about that in the instructions). If someone wants to process such data, we recommend to set up a private swarm among the orgs they trust (e.g., a couple of labs/small companies who don't have many GPUs themselves may set up a private swarm and collaborate to process their datasets). Regarding homomorphic encryption (HE), I'm afraid the current methods to run neural networks in the HE fashion involve 10-100x slowdown, since they are mostly not designed for floating-point operations. We'd love to find a way to do it faster though, since privacy is obviously an important issue for many tasks.
- Kerbonut 4y agoHi there, thanks for taking the time to answer questions! There are numerous use cases where even a 100x slowdown would be acceptable if it was demonstrable able to process sensitive data. Can you help me understand what kind of a slowdown that is? Could the 10-100x slowdown be overcome by more compute nodes, or would it require the nodes themselves to be 10-100x faster for e.g.?
- borzunov 4y agoIf someone wants to process sensitive data and is okay with 10x slowdown, it's better to use offloading. This is another, slower method for running large LMs locally without high-end GPUs, see details here: https://news.ycombinator.com/item?id=34216213 https://news.ycombinator.com/item?id=34216213 In other words, if Petals nodes became 10-100x slower, Petals would lose its competitive advantage over simpler methods that don't communicate over the Internet.
- oersted 4y agoAre there basic stats on real-time contributors and latency?
- pmontra 4y agoI guess that it could be used to create a private swarm, if one has a lot of hardware at home.
- thot_experiment 4y agoIs there a easy way to run a large language model and/or speech synthesis model locally/in colab? Stable Diffusion is easily accessible and has a vibrant community around AUTOMATIC1111. It's super straightforward to run on a Google Colab. Are there similar open source solutions to LLM/TTS? I believe I had GPT2 running locally at one point, as well as ESPNET2? Not 100% sure it's been a while. Wondering what the state of the art for FOSS neural LLMS and TTS is in 2023.
- FireInsight 4y agoNot sure about TTS, but I've trained GPT-2 (a pytorch implementation I think) on my own data and it worked pretty well, also tried eleutherai's 6B model but, couldn't figure out how to run it.. About an "easy way", I don't think such user interface like what Stable Diffusion has got exists as of now.
- borzunov 4y agoReally large (GPT-3-sized) language models have much more parameters than diffusion models, so it's difficult to load them locally unless you have a server with 8x 3090/3x A100 GPUs. Petals is the only way to fine-tune and inference 100B+ parameter models from Colab, as far as I know.
- thot_experiment 4y agoInteresting, how does that work with the multiple GPUs? I'm not familiar with the internal workings of these models, is there anywhere where I can get a brief rundown of how the processing is split. I imagine there can't me much swapping between GPUs as that seems prohibitively slow? How is the model split such that it can be worked on in parallel by multiple GPUs w/o being bottlenecked by IO?
- borzunov 4y agoI think this is a relevant link for you: https://huggingface.co/transformers/v4.9.0/parallelism.html https://huggingface.co/transformers/v4.9.0/parallelism.html For large LMs, people usually use tensor-parallelism (TP) or pipeline-parallelism (PP). TP involves lots of communication, but uses all GPUs 100% of the time and works faster. PP requires much less communication, but may keep some GPUs idle while they are waiting for data from others. Usually, TP is used when you have good communication channels between GPUs (e.g., they are in one data center and connected with NVLink), while PP is used when communication is a bottleneck (like in Petals, where the data is sent over the Internet, which is much slower than NVLink).
- Roark66 4y agoThis is a nice effort if it allows you to run bloom 170B in 1s per token. Just for comparison sake. With a last gen Ryzen cpu (16core) it takes me about 90s to run the model with 32gb ram (the entire model uses few GB of nvme storage too, as 32gb isn't enough ram). However, I wonder how they prevent abuse. The main page doesn't mention it. As they mentioned block chain I suspect there will be some sort of credits implemented. I'll definitely be watching where this project goes. Edit:just to clarify the 90s is not the 170b parameter model. It is 7b bloom version. I forgot to mention it and it puts the ability to run a 170B model in 1s in better perspective.
- borzunov 4y agoA Petals dev here. At the moment, we're working on a centralized incentive system, no blockchain involved. It will award points if someone is running a server that consistently stays online and returns correct results. Then, users will be able to spend these points for prioritized inference and (maybe) extra features like increased sequence length/batch size limits. This way, the swarm will prioritize people who actually contribute compute and serve others in the remaining idle time.
- narrator 4y agoMaybe you could get Bram Cohen to work on this. Seriously, reach out to him, he loves to work on these game theory sorts of things.
- oefrha 4y agoI’d say his reputation suffered quite a bit after the whole Chia (that proof of SSD thrashing coin) BS.
- prettyStandard 4y agoIs it possible to have the server shutdown predictably when it finishes tasks periodically, and not get penalized? I would like my machine to run while I'm not using it.
- m00dy 4y agowho is working on blockchain + web3 + AI inference => Decentralized AI besides me ?
- prettyStandard 4y agoI have been interested in 2 of those 3. Where are you working on them?
- dpflan 4y agoCan you elaborate on what you are doing?
- ShamelessC 4y agoIncreasingly fewer people are interested in scamming people after the FTX debacle, from what I can tell
- vegabook 4y agoAny plans for releasing an API spec that would allow for access from languages other than Python?
- borzunov 4y agoThere's a lightweight HTTP API for inference: https://github.com/borzunov/chat.petals.ml#http-api-methods https://github.com/borzunov/chat.petals.ml#http-api-methods
- artist_eren 4y agoexciting, will surely check the git repo
- bogwog 4y agoSo NVLink/NVSwitch pools multiple GPU resources on a single (very expensive) system. A cheaper alternative to that is "offloading", which is a technique that splits the inference process into smaller steps, so it can run on systems with much less resources available... and Petals is a 10x faster alternative to that. Did I get that right? This AI stuff is moving very fast and it's hard to keep up, but it's all fascinating.
- parentheses 4y agoMatches my understanding also. Can someone in the know confirm?
- nmitchko 4y agoYour understanding is correct, but I can't vouch for the claim's accuracy. This could make the execution of models much more accessible to people who don't have a 4x / RTX3090 or better in a ML or mining rig ..
- cardine 4y agoOffloading is when the computation is done on the CPU instead of the GPU. DeepSpeed is an example of this.
- borzunov 4y agoIn case of offloading, the computations are usually still performed on GPU, but the model is hosted in RAM/SSD instead of the GPU memory (and its chunks are copied to the GPU memory when necessary).
- cardine 4y agoA lot of computation is offloaded to the CPU, such as gradients and optimizer states. You are right though that quite a bit of computation is still done on the GPU.
- rolenthedeep 4y ago
- deleted 4y ago[deleted]
- sva_ 4y agoI tried the chat on http://chat.petals.ml http://chat.petals.ml, and it seems to struggle with the current load (as per the disclaimer at the top) Human: How is the weather today? AI: the AI theAI)aultAIAI ) course ) . can?esterday to people? ? is to think thatified ) Really cool project though, I wanted to work on something similar.
- throwaway743 4y agoWon't even load for me atm
- metadat 4y agoI replied to me with: It is nice today. Not garbled, but also extremely shallow.
- Alifatisk 4y agoI sometimes tend to forget that you can use decentralization without using all these crypto stuff.
- ShamelessC 4y agoSo does literally the entire “web 3” movement.
- deleted 4y ago[deleted]
- NicoleJO 4y agoWhat is the copyright status of these models?
- borzunov 4y agoPetals runs BLOOM, an open-source, publicly released model of the same size as GPT-3. Here's a description of the data used to train this model: https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom (the "Training" section)
- alexb_ 4y agoBLOOM is not open source. It has the RAIL license, which exists solely to place restrictions on the use of the software, as well as forcing people to update. Read more: https://bigscience.huggingface.co/blog/the-bigscience-rail-license https://bigscience.huggingface.co/blog/the-bigscience-rail-l...
- NicoleJO 4y agoThanks. It contains copyrighted material and is therefore illegal to use.
- alexb_ 4y agoIs the MIT license that this uses compatible with the RAIL license that Bloom uses? Or are there not issues with that?
- szundi 4y agoHm, so the emlolyees never turn off their personal computers please and I have a chatbot? Makes sense.
- deleted 4y ago[deleted]
- thomastjeffery 4y agoWhat's the point? So you can get predicted text that looks "coherent". Then what? There is literally no place to add logic. Neural net-based language models are impressive, sure, but it's not hard to see how useless they are. The only time their output is logically coherent is when they are lucky, and that seems to happen often because most of their input was logically coherent to begin with.
- qznc 4y agoPlay with https://chat.openai.com/ https://chat.openai.com/ to experience how powerful predicting text is.
- thomastjeffery 4y agoI have. And as I said, it's very impressive. And it has some usefulness: essentially it's an alternative to reading through many pages/posts of StackOverflow and Wikipedia. But it doesn't know anything. It has no clue whatsoever whether it is correct or incorrect. It only makes guesses. The only reason there is useful output is because that output is a transformation of useful input. There is no logic. There is no way to introduce logic. There is no way to filter it through logic. If some coherent mixture of the ML's training datasets already contains the answer to your question - like literary or code examples, definitions, etc. - then the output will be useful. Otherwise, it's just wrong, and sometimes unexpectedly so. The output of chatGPT (or any other ML-based NLP) can only be as correct or knowledgeable as the data it is trained on; and it will practically never even match that level, because it is only mixing words by semantic popularity, never by logical relationship.
- fasterik 4y agoWhether or not the current technology is useless is an empirical question. How many people are using ChatGPT, Stable Diffusion, etc. for economically or personally valuable activities? We actually don't know. Even if we assume the technology is useless in its current state, it is still incremental progress. Could we have predicted 10 years ago what neural networks would be capable of today? Now, tell me what neural networks will be doing in 10 years. If you think you know the answer with any degree of certainty, you're probably deluded.
- Seattle3503 4y agoCould this approach be used on other types of models, such as image models eg stable diffusion?
- ShamelessC 4y agoThere is probably less motivation, as these models are much smaller. An M1 Mac can run inference in under a minute. An GPU in as little as 3-5 seconds. This is supposed to be as much as 20x faster when the distilled Stable diffusion model is released.
- rolenthedeep 4y agoSo this sounds like BOINC but specifically for language neural nets? It's a very interesting concept, and I quite like the idea of a public, open compute cloud. I'd like to see more detail on security: if I'm going to donate time on my personal machine, I'd like some assurance that the workload is properly sandboxed and can't reasonably access my network or data. Mostly out of interest, what's the advantage to this over just using the existing BOINC network? I've been running BOINC on and off since the dialup days, it's an extremely mature platform with all kinds of workload capabilities.
- borzunov 4y agoDuring the training, participants only exchange tensors (embeddings, gradients) and never send code to each other. No other peer can execute arbitrary code on your computer - they can only request you to run one of pre-defined BLOOM layers. You can further isolate the Petals server from your machine by running it a Docker container (see the command in the repo). A client needs to communicate with multiple servers in a specific way to run the model, I'm not sure our communication model can be implemented with BOINC.
- KETpXDDzR 4y agoSo... How can I mine crypto with this? :)