5 ms·
Mesh LLM: distributed AI computing on iroh
- jing09928 3mo ago[dead]
- josefrichter 3mo agoIs there a catch? If not, this would be super useful.
- stymaar 3mo agoThe catch is that the token generation speed is going to be limited by network latency, making it unbearably slow to run over the internet. It can be great on a local network though, especially if your workload is prefill-heavy (more text input to process than output tokens to emit).
- turtleyacht 3mo agoIt sounds like iroh enables distributed compute without having to finangle custom hardware.
- jmercouris 3mo agoI thought about this too, but the throughput over a network is incredibly slow. It’s not usable for interactive use.
- i386 3mo agoThat isn’t true. llama RPC is incredibly slow but staged splits in skippy are orders of magnitude faster.
- stymaar 3mo agoThroughput is not a problem as you just share relatively small vectors (a few kilobytes in size), the key issue is network latency.
- jmercouris 3mo agoI didn't mean the throughput of the network, I meant the throughput of the calculations. Yes, the latency is the issue.
- nttylock 3mo ago[flagged]
- darkpicnic 3mo agococompute.ai is already doing this really well.
- SwellJoe 3mo agoIs it? I don't see anything on the website about splitting a model across multiple devices, only about putting local models on the internet, a wholly orthogonal problem (which is already easy with existing tools, since models use an http API).
- darkpicnic 3mo agoGood point. I know cocompute is working on splitting, but it’s not there yet; I was referring to the round-robin delegation within a trusted pool. Mesh LLM looks great too!
- dnoberon 3mo agoCool, always good to have more in the ecosystem. I love Iroh and hope this continues to succeed.
- SwellJoe 3mo agoI note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks. Are we talking 1 token per second for a split model? Less? Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.
- woadwarrior01 3mo agoPerf should be fairly straightforward to ballpark. You'll need to transfer roughly 2 . hidden_size . num_shards bytes over the network per token during autoregressive decoding. And divide that number by chunk size during prefill.
- i386 3mo agoThis was done on my home lab simulating 5ms latency and jitter between machines. Splits work quite well if you your nodes are over WAN at metro latency’s but not super fast on global WAN. The idea is that you could take several machines without dedicated RDMA or NVLINK fabric and use them to serve a large model on hardware you own then share it with others. I’m currently working on GLM 5.2 on my lab environment with around 10 tok/s on the same split.
- zdw 3mo agoWhat hardware (CPU/GPU/memory) and network was used for this? What quantization for GLM 5.2? How much tuning of the split was needed?
- i386 3mo agoThe lab features two Mac Studios: an Apple M3 Ultra (32 CPU cores, 80 GPU cores, 256 GB unified memory) and an Apple M1 Ultra (20 CPU cores, 48 GPU cores, 128 GB unified memory), both connected via 1Gbit Ethernet. We use a customized Q2 quantization that preserves sensitive tensors at Q8. To reduce compute time per layer, we are developing a custom GLM DSA Metal graph. While we are not yet approaching MTP, we plan to port our existing MTP implementations from versions 4.7 and 5.1 to 5.2. Since GLM's MTP acceptance rate is very high for a single predicted token, we are exploring token prediction techniques to widen the predicted tokens and utilize parallelism for verification.
- darkpicnic 3mo agoDoes Mesh LLM encrypt the payload between nodes? Is it possible to read requests from other users?
- tekacs 3mo agoI'm not affiliated, but yes – the main 'point' of iroh is that it's 'dial-a-key', QUIC with encryption based on the keys of the endpoints.
- metadat 3mo agoJust wondering, why do you care about encryption in this context?
- darkpicnic 3mo agoIf payloads to LLMs are being passed around to various nodes, even trusted ones (like friends and family), it gets awkward if you send something very personal. Think sending a medical question to medgemma:27b.
- oezi 3mo agoEven if transport is encrypted, the LLM computations will always be clear text, right?
- embedding-shape 3mo agoIndeed, it's in-transit-encrypted so snoopers won't be able to see it, but it's not E2E encrypted nor in-process encrypted, the one's doing the inference could obviously see the input/output.
- i386 3mo agoI’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
- deleted 3mo ago[deleted]
- iotapi322 3mo agoThis is super impressive, We have a lab with lots of different epycs and different models - to bring them together this way is amazing. Well done!
- i386 3mo agoThank you! AMD is a weak spot in our testing right now. If you’re willing to contribute or let us borrow some compute time, drop in on the Discord.
- Lerc 3mo agoI have never really delved into kv cache implementation, do they run effectively separate caches per layer? If so I can see it all dividing nicely, computation and data size wise and the only slowdown would be in search layer waiting for it's turn. If you pipelined it you could run multiple queries. Is anyone doing best-of-n with a n stage pipeline running each query offset by one?
- i386 3mo agoEach stage has its own KV for the layers it hosts. You are on the money there, when one stage is waiting it's free for more parallelism. I am planning on exploiting this for more token verification through ngram spec decoding.
- DerivativeBS 3mo agoCurious about: does it have fault tolerance if one of the machines goes down mid-inference? Can it dynamically reroute, or does it just retry?
- tlongwell-block 3mo ago[dead]
- _superposition_ 3mo agoI just wish I had the hardware to try it out!
- dwoosley 3mo agoI’ve been curious what a polymorphic botnet that runs one (or multiple) distributed LLMs would be capable of doing. The idea would be to evolve the botnet delivery and payload using the clustered compute of all hosts in the botnet to run LLMs that guides the evolution of various botnet clusters. Bad cluster morphs get caught and cleaned off and bad delivery methods never spread, but the best versions survive to continue to grow. What I envisioned for how it works is fairly similar to this, QUIC can actually be more difficult to detect than it seems since it’s very dynamic.
- nullc 3mo agoDoes this have intelligent expert handling for high parallelism MOE? You can get very high throughput for highly parallel MOE if you can mix different queries at each expert stage, but if the batch has to run together for the whole pipeline you get a parallelism loss instead of gain.
- michaelneale 3mo ago@null not as yet, but did a tiny bit of research before. If i understand it this would help mainly with batching (ie concurrent sessions/users/turns etc) to keep the pipeline more busy (vs a batch of a fixed size going together), which is interesting, but most work so far has been on latency hiding for straight line performance to start. We did have a version of things which used expert islands for MoE parallel alternative (I think that has been mostly scrubbed from the code). It showed early promise by having trunk+hot experts together, but as models got larger, it made both that very large on its own but also didn't seem to work as well (or we just weren't good at grouping experts), the idea being to really side step latency and route sessions to those islands. It felt like as models scaled it didn't stay smart (as diverse experts activated more than I thought they would). Could be our mistake (was an exciting possibility though - if you don't mind accepting some loss). tldr; yes this seems a very nice enhancement for smarter batching/keeping things busy and seems like most larger models we look at are MoE!
- whatjustin 3mo agoThe real test is throughput. I'd like to see tokens/sec at higher concurrency and with uneven hardware.
- Abishek_Muthian 3mo agoI'm more interested in running distributed inference for purpose built small language models than these coding LLMs. Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output. Nicely done OP.
- unrvl22 3mo agoSomething like this is nice, where instead of having 1 model with X active experts, you have 10 different models, all small and dense, trained on specific information. and loaded on 10 different servers, with one router.
- Abishek_Muthian 3mo agoTrue, I've been telling that coding LLMs have been detrimental to local AI scene. What happened to those nifty ML projects which can run on raspberry pi zero.
- sig_kill 3mo agoThis is actually how I develop and use the mesh at home. Rather than splitting models, I aggregate disparate compute behind one endpoint, without having separate inference providers on each host and a gateway like LiteLLM
- downrightmike 3mo agodifference between this and Exo?
- sig_kill 3mo agoWrite up here: https://meshllm.cloud/docs/pages/exo-comparison/ https://meshllm.cloud/docs/pages/exo-comparison/
- deleted 3mo ago[deleted]
- MattPerry 3mo agoThe first picture "gpu rig", "laptop", "server", "cloud node, etc made me realize how little compute I have. I don't have a laptop with 24GB VRAM or a workstation with 96GB. I think if I convinced all of my friends to run LLMs on their gaming PCs, I don't I would have the total VRAM in the picture. As an aside, I saw this post mentions a public mesh, but I couldn't find any more information.
- kennywinker 3mo agohttps://public.meshllm.cloud/ https://public.meshllm.cloud/
- whs 3mo agoI've been looking for similar distributed computing style LLM, and I found AI Horde and a few other smaller efforts like one from Aphrodite people and distributed training from Nous Research. AI Horde seems to be the biggest of them all. Their API speaks KoboldCPP text completion (not even chat completion). It seems that the community (or at least the active people) strongly prefer it this way because the API exposes more tunables than chat completions, which for roleplay use seems to result in better result. I don't know what else you can use AI Horde for anyway since all other use cases likely will require tool use. Just this week I was set out to improve their OpenAI bridge to support chat templates and response parsing. We'll see if I could get it deployed officially then you might be able to use it to code, although you'll have to use RP models. I think Horde do have a lot more abuse prevention. Workers needs to have 1 week of cumulative uptime to be considered trusted to prevent brigading - users can opt into trusted workers only. Running a worker give you kudos which is required for >512 max tokens generations and also free requests get bumped to last.
- SubiculumCode 3mo agoAll these ASICS being designed and specialized for AI but none seem to be being built for consumers. Reason?
- kennywinker 3mo agoI spent a while trying to get mesh-llm running, but none of the installable llama.cpp builds worked with my older gpu. It looks like it should be able to be used to proxy an external llama.cpp service, but I had no luck setting that up either. Seems very cool, but definitely some rough edges.
- i386 3mo agoI’d love a bug report - we can get it working for you!
- bubbi 3mo ago[flagged]
- luciana1u 3mo ago[flagged]
- jkwang 3mo ago[dead]
- vigsterkr 3mo agothe https://query.mt/ https://query.mt/ project has been using iroh based mesh for a while. maybe give it a go, especially if you wanna use your mesh models on your mobile phone as well.
- derdi 3mo agoThey might want to make their docs less confusing. https://query.mt/showcase/gpu-sharing/ https://query.mt/showcase/gpu-sharing/ says "no API key needed", and talks about running llama.cpp on a friend's GPU. No API keys are needed for this. But then the page goes on about who "holds" API keys, and where API keys "stay", where API keys are "locked", etc.
- tangsoupgallery 3mo ago[flagged]
- deleted 3mo ago[deleted]
- dana321 3mo agoI knew this was possible, i asked chatgpt about a year ago and it said no the latency would be too big of a problem. I spent the best part of a year learning libp2p and was looking for a project to do with it at the time.
- roger_ 3mo agoDoes this support Qwen 3.6 (e.g. 27B) and the myriad of llama.cpp options (batch sizes, quantization, etc.)? I'd love to see some performance data.
- sig_kill 3mo agoYes! I’ve worked on the settings interface between our runtime and llamacpp, these are documented and available via our config.toml file
- hydrogenbon007 3mo ago[dead]
- Onavo 3mo agoIs this truly more secure though? The host can still see your data.
- maccam912 3mo agoI have a macbook pro, figured I'd see how easy it was to contribute some vram... And I can't overstate how easy it was. The swarm page thing had a little "join" button and said to run "mesh-llm --auto". And I did. And it worked first try. That is such an uncommon experience I had to report back. It handled picking a model to serve, downloading it from peers, and to test it I chatted with the model I was hosting, I could see the GPU doing work, etc. It might be more of an endorsement for iroh than mesh-llm, although I'm sure getting it to all work seamlessly took work on both sides. But to whoever spent the time and energy trying to make it seamless, consider the effort recognized!
- jjheuaao 3mo ago[flagged]
- postpress 3mo agoThis got me thinking about experiments with models talking to each other over WebRTC: https://xt-ml.github.io/shadow-claw/ https://xt-ml.github.io/shadow-claw/ Its sort of a "P2P mesh" :-) Watch four instances of the harness running together and collaborating on checking the weather: https://www.youtube.com/watch?v=h1les1A3gcg https://www.youtube.com/watch?v=h1les1A3gcg
- lenerdenator 3mo agoLet's say I have a bunch of computers (Raspberry Pi 3+, MBP from 2017, Lenovo T420) sitting around that are on my local network. All are from the last 15 years and my main driver is a M2 MBP with 32 gb of RAM. Would this let me utilize that spare hardware to do anything meaningful with a self-hosted coding-focused LLM at a somewhat-decent token generating speed, or is this still a pipe dream?
- s4saif 3mo agoSo, how is this different then exo ? it does the same job as well
- i386 3mo agoexo is Mac only.
- throw1234567891 3mo agono it isn't
- brcmthrowaway 3mo agoHow does this differ to darkbloom?
- enbarca 3mo agoI'm looking at adopting a mesh network architecture for self-sovereign inference across longitudinal health records.
- guillaume_code 2mo ago[dead]