16 ms·
Run 100B+ language models at home, BitTorrent‑style
- alchemist1e9 4y agoI was waiting for this! This is exactly where we are headed. Excellent.
- mdgrech23 4y agodis gonna be big
- behnamoh 4y agoWhat if all the computation power that's being wasted by crypto (Proof of Work, etc.) could be shifted to powering AI models in a decentralized way, such as this project? You keep the AI "alive" and in return, you get paid in crypto. What a wonderful time to be alive!
- eclipxe 4y agoLet's do it.
- hhoodie 4y agoI'm in
- nullsense 4y agoLet's watch the world burn!
- antibasilisk 4y agoit's all fun and games until a bunch of kids die
- int_19h 4y agoThe world is capable of burning itself just fine without such assistance. It would be much neater to turn it all into paperclips instead.
- alchemist1e9 4y agome too
- lxgr 4y agoProof-of-work only works with a particular set of computational problems, i.e. those in NP. I'm not sure if running a LLM fits that bill. I suppose you could combine proof-of-stake with it in some way (e.g. you commit to an input/output tuple and get your stake slashed if it can be shown to not reproduce on a "canonical" execution), though? That's not nearly as simple as "normal" PoW, though – you'd need to encode some reputational aspect into the system.
- Enginerrrd 4y agoYou can definitely do arbitrary work as a sort of proof of work. Not quite the same mathematically, but pragmatically similar. The key is building in some redundancy/error-correction and ensuring that a single node can't by itself define "correctness" of a solution. You do that by duplicating work across nodes, distributing chunks randomly and rejecting/rerunning disagreeing results. It's also pretty easy to spot bad actors trying to cheat on their computational work in this scenario.
- deleted 4y ago[deleted]
- pshc 4y agoI don't think it's that easy at all. The work function must be cheap and canonical to check, and the consensus algorithm has to be rigorous, or else it's too easy to attack the security of the network. DoS, Sybil, 51%, social takeover via hard fork, the list goes on...
- alchemist1e9 4y agoIt has a well define loss function with a numerical value. The improvement of this value can be a type of difficulty. Check some other comments I’ve made on this post for how it might work.
- pshc 4y agoIt's an interesting idea for sure, but loss doesn't go down forever. I think this ends with a highly overfitted network that grinds to a halt as the loss function hits local minima. Even if you get past that, there's no consensus mechanism or finalization as it stands, and validating solutions is relatively expensive.
- saulpw 4y agoAnd now we have an AI that is impossible to turn off.
- Taek 4y agoThat type of work is not useful for securing a decentralized system. To secure a decentralized blockchain, you need the work to be provably wasteful. That said, there's a new paradigm of Blockchain brewing called 'Proof of Physical Work', which accepts the centralized limitations and leverages the incentive scheme to produce useful outcomes for society. Perhaps the most interesting one right now is HiveMapper. It won't secure a decentralized blockchain, but it might still be a great value add for society.
- antibasilisk 4y agoby definition that makes it not a blockchain
- alchemist1e9 4y agoI don’t believe it’s accurate what you claim. The training of LLMs I believe has a well defined loss function with a measurable value on a specific input set. I actually already discussed this problem extensively with GPT-4 itself so I’ll leave you with parts of that discussion: > Me: Do LLMs have a well defined loss function that they optimize over? > GPT-4 Assistant: Yes, large language models (LLMs) have a well-defined loss function that they optimize over during the training process. The choice of loss function depends on the specific task the model is designed for and the architecture used. For LLMs like GPT-3, which are based on the transformer architecture and designed for natural language processing tasks, the most commonly used loss function is the cross-entropy loss. > Cross-entropy loss measures the difference between the model's predicted probability distribution over the possible output tokens (words, subwords, or characters, depending on the tokenization used) and the true probability distribution, which assigns a probability of 1 to the correct output token and 0 to all other tokens. The goal of the training process is to minimize the cross-entropy loss, which essentially means making the model's predicted probabilities as close as possible to the true probabilities. > During training, the LLM processes input sequences (e.g., sentences or paragraphs) and generates output token probabilities based on its current weights. The cross-entropy loss is computed using these probabilities and the true target tokens, and then the gradients of the loss with respect to the model's weights are calculated using backpropagation. The weights are updated using an optimization algorithm, such as stochastic gradient descent (SGD) or one of its variants like Adam or Adagrad, in order to minimize the loss. > The process of iteratively updating the model's weights to minimize the loss function is what allows LLMs to learn patterns and structure in the training data, ultimately enabling them to generate coherent text, answer questions, or perform other language-related tasks. > Me: Is it possible to restrict the training with a rules that says only these weights, say specific slices, are allowed to change, keep all other weights fixed? Is this how LLMs can be trained in a distributed fashion? > GPT-4 Assistant: Yes, it is possible to restrict the training of an LLM to specific slices or subsets of weights while keeping other weights fixed. This is a common approach used in distributed and parallel training scenarios, where multiple devices or nodes work on different parts of the model simultaneously. This approach is known as model parallelism. > In model parallelism, the model's weights are divided into smaller subsets or partitions, with each partition being assigned to a different device or node. During training, each device works only on updating the weights in its assigned partition while keeping the weights in the other partitions fixed. The gradients are computed for each partition separately, and the updates are performed locally on each device. > Once all devices complete their local updates, they communicate their updated weights with one another to synchronize the model's state. This communication can be done through various methods, such as parameter server architecture or all-reduce algorithms. The key to effective model parallelism is balancing the computational workload among the devices while minimizing the communication overhead. > Adapting this concept for the proposed proof of work and LLM training combination could involve assigning different subsets of weights to miners and having them work on updating these subsets. This would essentially turn the mining process into a form of distributed LLM training, with each miner contributing to a portion of the model's weights. However, coordinating and synchronizing the updates among miners in a decentralized environment, while maintaining the integrity and security of the blockchain, would be a significant challenge that needs to be addressed in the design.
- deleted 4y ago[deleted]
- alchemist1e9 4y agoPortion of a discussion I had on this topic with GPT-4: > Your idea of having all miners work on the same section of the model and compete to find better weights with a lower loss function value is an interesting approach. This would essentially treat the LLM training process like a proof-of-work mining competition. It is true that this approach involves some degree of duplicate work, but the simultaneous, randomized exploration of the weight space could still leverage the distributed computing power of the network. > Here's a high-level overview of how this approach could work: > Miners receive a specific section of the LLM's weights and a subset of training data. The training data could be determined based on information from the previous block (e.g., by hashing the previous block). > Each miner initializes their local copy of the weight section with random perturbations to introduce diversity in the optimization process. > Miners perform optimization steps (e.g., gradient descent) on their local weight section to minimize the loss function on the given training data. > Miners search for a solution that satisfies both the proof of training (improvement in the loss function) and the proof of work (hash of the block meeting the difficulty target). > The first miner to find a solution meeting both criteria broadcasts their updated weight section and the new block to the network. > Other miners verify the validity of the proposed solution (i.e., checking the improvement in the loss function and the hash meeting the difficulty target) and add the new block to their local copy of the blockchain. > This approach would turn the mining process into a competitive LLM training process, where miners contribute their computing power towards improving the model. It maintains some of the core properties of proof-of-work mining while directing the computational resources towards a productive goal. However, this approach still needs to address potential issues related to data privacy, intellectual property, and the synchronization of the model's weights across the entire network.
- vore 4y agoWhat's the point of this hashing the block business if you can already prove your work by presenting a set of of weights that reduces the loss function? And even then, you run into the risk of overfitting if you just blindly optimize for loss like that.
- inimino 4y agoEventually it will be common knowledge that asking a text compressor trained on all available text to design a new technology doesn't work.
- lgas 4y agoI believe that's pretty close to what https://bittensor.com/ https://bittensor.com/ does.
- amelius 4y agoWhat if we moved to "Proof-of-Carbon-capture" instead?
- xotom20390 4y ago[dead]
- lxgr 4y agoImagine if it was possible to combine this with homomorphic encryption into something like AirDrop for LLMs! ("Sorry, I don't know how to answer that – but you can try getting closer to a bunch of other people running the app on their device and ask again!")
- chatmasta 4y agoSounds like a literal hive mind!
- techwizrd 4y agoThe library they're using is literally called Hivemind [0]. I'm interested to see how the approach they're using differs from what we use in federated learning or gossip learning. > Hivemind is a PyTorch library for decentralized deep learning across the Internet. 0: https://github.com/learning-at-home/hivemind https://github.com/learning-at-home/hivemind
- borzunov 4y agoA Hivemind/Petals dev here. As far as I understand, most federated learning methods can't efficiently train very large models (with billions of parameters) because they repeat some calculations on many peers and/or involve excess communication. In contrast, the training methods implemented in Hivemind struggle to minimize compute and communication but don't provide data privacy guarantees. This is mostly okay for LLMs, since they are trained on public data scraped from the Internet anyway.
- sebzim4500 4y agoHomomorphic encryption has such an enormous overhead that it would never be faster than just running the model locally. Or probably on your wristwatch for that matter.
- firstfewshells 4y agoML/AI moving too fast.
- ashokpoudel 4y agoI would be very concerned about sending my data over to a swarm https://ashokpoudel.medium.com/understanding-security-and-privacy-in-petals-what-you-need-to-know-before-using-this-breakthrough-4c68bb9b6cad https://ashokpoudel.medium.com/understanding-security-and-pr...
- sillysaurusx 4y ago> Parallel inference reaches hundreds of tokens/sec. Marketing claims, meh. It gives normal people the wrong impression. You can’t parallelize your query because it’s sequential. I think people will be willing to wait the ~200 sec necessary to get 200 words, but it’s best to be up front about this limitation. Also abuse is a problem. Once 4chan realizes they can poison the distributed model, they’ll have a field day. But maybe it’s too much effort for too little reward that trolls won’t bother.
- Etheryte 4y agoThere is no poisoning vector, you can only update prompts and adapters hosted locally.
- sillysaurusx 4y agoFrom the site: > you load a small part of the model, then team up with people serving the other parts to run inference or fine-tuning. If multiple people participate in a fine tuning session, you have to trust all of them. You also have to trust everybody for inference too, but at least one of them can’t scramble the model.
- pshc 4y agoWouldn't untrusted weird input increase loss and be rejected?
- chatmasta 4y agoMaybe this could be solved with opt-in (or opt-out via banning) federation similar to Mastodon. Instead of one network you could have a bunch of different networks each focused on the interests of a different community. Or maybe as someone with a node, you could "subscribe" to different communities that use different filtering and prioritization mechanisms for task assignments. I do love the general direction, and I think it's inevitable that training will move to be more decentralized like this. It's also the best chance we have at disrupting the centralization of "Open"AI and their ilk. I say the earlier we figure this out, the better, but it's not an easy problem to solve cleanly. And, not to be that guy, but maybe we could add some cryptocurrency incentives to the mix... conveniently enough, the crypto miners already have the GPUs ready to go!
- smrtinsert 4y agoIf nodes drop in and out how does that impact the inferences I wonder
- frankreyes 4y agoSkynet.
- ftxbro 4y agoThis link just goes to their website. Last I looked at this project, I was happy that it existed but I was disappointed (given my over-optimistic expectations) for two reasons: 1) It's for the BLOOM model which isn't great compared to somewhat recent gpts. Like I think I read that it's worse than the openai models on a per parameter basis. 2) It's faster than using RAM/SSD as faux VRAM but 'only' by 10x. That was even before LLaMA or its improvements had existed for running locally. So by my old understanding, bloom/petals wouldn't be even as good as those ones even though it technically has more parameters. I wonder are these interpretations still true (assuming they ever were true lol), or did something happen where bloom/petals is much better than that now? Edit: The petals/bloom publication that I read for the information I put above was https://arxiv.org/abs/2209.01188 https://arxiv.org/abs/2209.01188 published to arxiv on September 2 2022.
- deleted 4y ago[deleted]
- winddude 4y agoI got worse than 1 token/sec, and yes, wasn't impressed with bloom results, but I believe it's also very foreign language heavy. I haven't tried it yet but I believe flexGen benchmarked faster as well.
- borzunov 4y agoA Petals dev here. FlexGen is good at high-throughput inference (generating multiple sequences in parallel). During single-batch inference, it spends more than 5 sec/token in case of GPT-3/BLOOM-sized models. So, I believe 1 sec/token with Petals is the best you can get for the models of this size, unless you have enough GPUs to fit the entire model into the GPU memory (you'd need 3x A100 or 8x 3090 for the 8-bit quantized model).
- mzz80 4y agoIt does appear to only support Bloom, which makes it currently useless since there are much better models with fewer parameters that you can run on a single machine. However, the project has a lot of appeal. Not sure how different architectures will get impacted by network latency but presumably you could turn this into a HuggingFace type library where different models are plug-n-play. The wording of their webpage hints that they’re planning on adding support for other models soon.
- devoq 4y agoVery cool. Had been wondering when we would see real "cloud" database and model computation without some silly token attached.
- WithinReason 4y agoFrom the table, a collection of 14 servers is equivalent to a single A100 when using a batch size of 64. So what if you used 1 computer but did smart offloading to RAM or SSD? Would that be more than 14 times slower?
- MagicMoonlight 4y agoI feel like it makes much more sense to just run it in the CPU instead. CPUs have access to far more memory so you could fit the entire model at its original size. Instead of messing around with inefficient nonsense like this, figure out a way to prune and modify the models so that they run efficiently on a CPU.
- borzunov 4y agoRight now most CPUs are orders of magnitude slower than GPUs for doing forward/backward passes, so you're unlikely to get a similar speed. Some kind of pruning may help though.
- amrb 4y agoWhat I want to see is a bunch of RTX3060 mining cards being used to run ML models. They did talk about bandwidth issues with under 100mbit for servers tho we're doing this as a local cluster you could run some ex-server network gear and be gold.
- jhoelzel 4y agoi just thought this through building my new PC and house. The bottleneck is always the NIC because the most you can do right now is like 10gbe for conventional motherboards. After that you also run into cabeling issues. Cat 8 for instance also only does 40gbe max, which means for any more you need to bundle up connections which comes with its own problems. Another point is that while mining, gpus still are independent and not connected to each other. so each of them are restricted to the max your PCIe port will give you too. PCIe 4.0 has a maximum data transfer rate of 16 GT/s (gigatransfers per second) per lane, which translates to 2 GB/s (gigabytes per second) per lane. PCIe 4.0 can support up to 16 lanes, which means that it can provide a maximum data transfer rate of 32 GB/s (gigabytes per second) in each direction (upstream and downstream) on a x16 slot.
- quickthrower2 4y agoDumb question from someone who knows not to much yet about LLMs. How can you trust the other computers? Will I end up with a bunch of swear words coming back from the other nodes that are playing a prank?
- cjbprime 4y agoI don't think that's a dumb question! I don't know if this project has an answer to that, but there are some techniques (Merkle tree hashes etc) that might work depending on how much of the model you want to download locally. I don't see how to securely scale the inference step, though.
- sillysaurusx 4y agoSending tensors to two or more nodes is a partial solution, since you can reject ones that don’t match. But fundamentally the answer is yes — bad actors can overwhelm the swarm, and there’s not much you can do about it. I think the project should embrace this limitation. eBay had the same problem, but it’s in a seller’s interest to deliver correct items quickly. Make a social incentive and the rest will work itself out.
- flangola7 4y agoIf you get banned from eBay your SSN/phone number/email/browser fingerprint/address/etc are prevented from coming back. What system would enforce that for computation nodes?
- 6510 4y ago> bad actors can overwhelm the swarm I don't think so. To simplify: You send out 1000 tasks, you perform them yourself, now you have 999 bad flags and 1 good one, you send out 10 tasks to the same including the good one, now you have 990 with 1 bad flag, 9 with 2 and 1 with 2 good ones, you continue sending tasks to the bad nodes and drop their response, if they send you a task you return garbage, you ask the good nodes (with say 100+ good flags) for their list of good nodes and test them one by one. You could build a system where bad nodes have to return so many good responses before getting booted that the joke is on them.
- nirav72 4y agoI wonder how close we are before someone comes up with a peer to peer malware using similar concept to train their model. Kind of like how trojan crypto miners were found in public package repos and apps just couple of years. (Probably still an issue)
- jimrandomh 4y agoThis works by taking a language model that won't fit in a single consumer GPU's memory, partitioning it layerwise, and running it distributed across a bunch of different people's computers. If I'm understanding correctly, then any single node acting dishonestly can replace the output out its portion with whatever they want, and (if every other node is honest), this is sufficient to fully control the output. So, probably okay to use for prompts like "rewrite Rick Astley lyrics in the style of Shakespeare", but not something you'd want to use in a way that feeds into another automated system. Meta-level, I think it's bad for the world if there's good technology for running neural nets on distributed consumer GPUs. From a cybersecurity perspective, Windows gaming PCs are easy pickings compared to datacenters, and I think there's a risk that after a few more iterations of AI development, we'll start getting systems that figure out they can increase their own power level by building a botnet that runs additional copies of themselves.
- polyomino 4y agoHow can this be decentralized with no ICO?
- whoisjuan 4y agoI like the idea behind this because large AI seems to be highly constrained by co-located computation and the costs associated with it (GPUs and energy). There are many delivery and cost advantages to running a massive LLM in a distributed P2P fashion. Weirdly enough, I see this as a real "web 3" opportunity. Corporations running large LLMs could run their models on a decentralized network and pay participants for their contributed computing capacity. AI most significant headwinds are cost and the pace at which GPU capacity is being built. This seems like a good model to tackle both issues.
- alwayslikethis 4y agoThe average computer is not realistically capable of running LLMs effectively (because VRAM or RAM does not fit the full model).
- hanniabu 4y agoYou simply reward based on performance
- whoisjuan 4y ago“Run large language models like BLOOM-176B collaboratively — you load a small part of the model, then team up with people serving the other parts to run inference or fine-tuning.” According to this excerpt, a node in the network doesn’t need load the entire model. Only a part.
- vineyardmike 4y ago> Weirdly enough, I see this as a real "web 3" opportunity. Corporations running large LLMs could run their models on a decentralized network and pay participants for their contributed computing capacity. The same problem we saw with "web3" is here. If I were a "miner" in this case, why would I not go commercial-scale to gain efficiencies here. I could just build a real datacenter, and offer real contracts to real companies instead. It'd be cheaper for everyone. Unless the expectation is that we literally can't get enough GPUs for all the datacenters, and we rely on the aggregate of consumers' integrated GPUs in their laptops? I think we'd just see companies not using LLMs before they got desperate enough to pay rando's for LLM processing.
- slg 4y agoI guess this starts the countdown clock to the first botnet running a LLM to generate spam content. Maybe I'm just turning in a crotchety old guy who is scared of new tech, but it really seems like as a community we are underestimating the degree to which this will present an existential threat to every site that relies on user generated content.
- ShamelessC 4y agoPerhaps you’re overstating the importance of those sites.
- bobthepanda 4y agoI mean, everyone ultimately reads content written by a person. Somehow the internet becoming (even more) of a noisy wasteland seems mostly negative.
- OJFord 4y agoBut generated nonsense is already possible and already exists. If all that crap becomes higher quality crap... Isn't that... It's not bad?
- bobthepanda 4y agoHigher quality sounding, and higher quality, are two different things, since generative AIs don’t really care about truth. Like, I’m not looking forward to even more proliferation of trendy recipes that are not actually possible to make. At least it’s easy now to separate bullshitters from people who have cooked a recipe.
- OJFord 4y agoI suppose the question is is there an incentive to do that? A crappy sounding crappy quality spam recipe already gets a page hit and no goodwill. Does better sounding but still crappy do better in any way that translates to money for the author (or author's operator)?
- jeffmjack 4y agoSo in the old model you could: 1. pay for compute 2. charge the customers to pay for compute, and now you can instead: 1. pay your customers to pay for compute 2. charge the customers to pay for the customers to pay for compute Is there something I'm not understanding in the business logic of this? Is it the fact that this would be running on computers that are essentially free, since it would just be like the desktop in someone's home office, so the infrastructure costs are already paid for (e.g. externalized)? Or like would the value here be accessing the LLM service for 'free'? But isn't just paying for a service like OpenAI relatively inexpensive and already nicely set up?
- causality0 4y agoBut isn't just paying for a service like OpenAI relatively inexpensive and already nicely set up? Sure, but OpenAI is never going to offer you a raw product. Their offerings will always be the heavily restricted, corporatized product they offer now. That works for many, maybe most, people but there's definitely a market for a "power to the users" LLM AI with no rules.
- ilyt 4y ago> Is there something I'm not understanding in the business logic of this? That people would rather give away some of the GPU time they aren't using at this moment than pay subscription. And presumably also not wanting to be beholden to whatever filters the "big AI cluster owner" puts in place
- fswd 4y agoIt's super slow. 1 token per second, if that. Half a word a second.
- deleted 4y ago[deleted]
- varunkmohan 4y agoCurious if anyone has actually used this. It's quite slow for me and feels more like a cute idea rather than a useful product.
- nothrowaways 4y agoThis is pure genius if it works.
- TeeMassive 4y agoAm I the only one excited about when 4chan will train its own AI by collectively pooling their power levels?
- geepytee 4y agoKinda reminds me of the BIONIC system
- eternityforest 4y agoYou made a real skynet!!! Jokes aside it's pretty cool!
- ar9av 4y agoRunning LM on two gpus on a single system comes with 10x speed penalty. Getting layers across network will be in general even slower. They talk about 1 token per second, with images it will be even less due to larger amount of sequential steps. It can be useful... if it's even possible. But there is quite slim amount of possible use cases. Generation will be slower, so why bother? For high amounts of batches? Maybe. But why use it if we have Swarm by db0? Training theoretically can be worth it, but something like Kickstarter and gpu renting can be both more cost-effective and quicker.
- visarga 4y agoSpeculative sampling to the rescue - you decode locally with a smaller-LLM, and only check from time to time with a large model, like every few tokens. This guarantees exactly the same quality with a big speedup, as you don't need to predict with the large model each individual token. Accelerating Large Language Model Decoding with Speculative Sampling https://arxiv.org/abs/2302.01318 https://arxiv.org/abs/2302.01318
- Havoc 4y agoThis seems to be inference side. Surely for distributed building a license free model similar to say 3.5 chatGPT would be more useful? ie rebuild the alpaca work minus legal issues
- chrisMyzel 4y agoPetals is an impressive feat but be aware it is very slow with 1-4 sec / token (depending on the hardware you have). I find it too slow for even experimenting, as a developer I want faster feedback cycles. Super interesting to see the project evolve over time, onboarding could not be easier.
- MuffinFlavored 4y agoWhat is the rate of tokens per second when you are talking to ChatGPT on GPT-4?
- FrameworkFred 4y agoMy interest in AI has just gone 10X, thanks and cheers!