11 ms·
Exo: Run your own AI cluster at home with everyday devices
- iJohnDoe 2y agoAnyone run this? Works?
- tdubhro1 2y agoThe readme shows how to run it assuming you can run a python program on the device, so I expect it works with laptops and PCs but there's a note at the end of the page saying that the iOS app has fallen behind the python version so it's not clear to me how to get this running on your iphone or other such devices.
- orsorna 2y agoThe "device" in question must be Apple Silicon because the `mlx` package is a hard dependency, or at least an ARM machine (I do not have any Apple Silicon Macbooks or ARM machines to run this). I tried tweaking this before realizing calls to this library is littered all over the repo. I don't really understand the AI ecosystem very well but it seems that the use of the `mlx` library should be supplanted by some other library depending on the platform machine. Until then, and the actual release of the iOS code somewhere, "everyday devices" is limited to premium devices that almost no one has more than one of. I'm looking forward to run this on other machine platforms and squeeze out what I can from old hardware laying around. Otherwise I doubt the tagline of the project. Edit: to add on, the only evidence that this runs anywhere but Apple Silicon is the maintainer's Twitter where they show it running on two Macbook Pros as well as other devices. I'm not sure how many of those devices are not ARM. I'm not throwing shade at the concept the author is presenting, but I'd appreciate if they could slow down functional commits (he is writing them right now as I type) and truthfully modify the documentation to state which targets are actually able to run this.
- acosmism 2y agowhy ask? try it!
- tvshtr 2y agosone people value their time
- hagope 2y agoI used to be excited about running models locally (LLM, stable diffusion etc) on my Mac, PC, etc. But now I have resigned to the fact that most useful AI compute will mostly be in the cloud. Sure, I can run some slow Llama3 models on my home network, but why bother when it is so cheap or free to run it on a cloud service? I know Apple is pushing local AI models; however, I have serious reservations about the impact on battery performance.
- fouc 2y agohttps://x.com/karpathy/status/1814038096218083497 https://x.com/karpathy/status/1814038096218083497 LLMs will start shrinking massively in size soon, without any loss in performance.
- Cantinflas 2y agoWhy bother running models locally? Privacy, for once, or censorship resistance.
- bongodongobob 2y agoI have a 2 year old Thinkpad and I wouldn't necessarily call llama3 slow on it. It's not as fast as ChatGPT but certainly serviceable. This should only help. Not sure why your throwing your hands up because this is a step towards solving your problem.
- nhod 2y agoIs this a hunch, or do you know of some data to back up your reservations? Copilot+ PC’s, which all run models locally, have the best battery life of any portable PC devices, ever. These devices have in turn taken a page out of Apple Silicon’s playbook. Apple has the benefit of deep hardware and software integration that no one else has, and is obsessive about battery life. It is reasonable to think that battery life will not be impacted much.
- mg 2y agoThis enables you to run larger models than you would be able to on any single device. No further explanation on how this is supposed to work? If some layers of the neural network are on deviceA and some layers are on deviceB, wouldn't that mean that for every token generated, all output data from the last layer on deviceA have to be transferred to deviceB?
- steeve 2y agoYes, that’s how it works (pipeline parallelism)
- mg 2y agoInteresting. Let's do the math ... Let's say the model has 50B parameters and 50 layers. That would mean about one billion values have to travel through the wifi for every generated token? I wonder how much data that is in bytes and how long it takes to transfer them.
- blackbear_ 2y agoIt's not the parameters that are sent, it's the layer outputs. That makes for a few thousands floats per token
- mikewarot 2y agoYes, so you would have a vector about 8k values long to be transferred on each token generated. You could do that easily with any modern network.
- thom 2y agoBexowulf.
- tarasglek 2y agoThis is the first timer i've seen tinygrad backend in the wild. Amusing that it's supposedly more stable than llama.cpp for this project.
- alexandercheema 2y agoRepo author here. Tinygrad changes rapidly so wouldn't it say it's "more" stable, but it certainly supports more accelerators than llama.cpp. As George Hotz likes to say, it sits somewhere on the spectrum between llama.cpp and Mojo. No hand-written kernels, optimal kernels are generated and found by beam search.
- matyaskzs 2y agoCloud cannot be beaten on compute / price, but moving to local could solve privacy issues and the world needs a second amendment for compute anyway.
- CuriouslyC 2y agoYou can beat gpt4/claude in terms of price/performance for most things by a mile using fine tuned models running in a colo. Those extra parameters give the chatbots the ability to understand malformed input and to provide off the cuff answers about almost anything, but small models can be just as smart about limited domains.
- ComputerGuru 2y agoThe problem is that once you say “fine tuned” then you have immediately slashed the user base down to virtually nothing. You need to fine-tune per-task and usually per-user (or org). There is no good way to scale that. Apple can fine-tune a local LLM to respond to a catalog of common interactions and requests but it’s hard to see anyone else deploying fine-tuned models for non-technical audiences or even for their own purposes when most of their needs are one-off and not recurring cases of the same thing.
- CuriouslyC 2y agoNot necessarily, you can fine tune on a general domain of knowledge (people already do this and open source the results) then use on device RAG to give it specific knowledge in the domain.
- dijit 2y ago> Cloud cannot be beaten on compute / price Sorry, I can't let misinformation like that slide. Cloud cost/benefit ratio is not good in many circumstances. For hobbyists it works well because you run your job for very brief periods and renting is much cheaper than buying in those cases. Similarly, if your business usage is so low as to be effectively run once per day then cloud has major benefits. However, if you are doing any kind of work that consumes more than 8hrs of computer time in a day, cloud is going to start being much more expensive. The exact cost/benefit depends on the SKU and I'm mostly talking about CPU/Memory/Storage- for managed services like databases it's significantly worse, and I'm comparing to rented servers not self-hosting at home, which is significantly cheaper still. Local hardware has downsides (availability, inflexibility), but it's faster and cheaper in almost all real workload scenarios where the compute would otherwise be completely idle/turned off >90% of the time.
- pierrefermat1 2y agoWould be great if we could get some benchmarks on commonly available hardware setups.
- pharrington 2y agoI'm sure someone will show their benchmarks in a couple years!
- festive-minsky 2y agoSo I just tried with 2x macbook pros (M2 64GB & M3 128GB) and it was exactly the same speed as with just 1 macbook pro (M2 64GB) Not exactly a common setup but at least it's something
- alexandercheema 2y agoCould you create a GitHub issue? There's a lot of work we'd like to do to improve this.
- pierrefermat1 2y agoThis is so hilariously bad, how does something like this end up needing to be a user created Github issue and not being caught when you guys launch.
- T3RMINATED 2y ago[dead]
- whoami730 2y agoIs it possible to use this for image recognition and like? Not sure what can be the usage of this apart from as a chatbot.
- tama_sala 2y agoYou can use other models like a vision LLM, or use AI agents as well
- jononor 2y agoImage recognition can generally be done very efficiently on a single commodity PC. Even a phone that is a few years olds can do quite a lot. Or a Raspberry PI. So it generally does not need distributed computing solutions. I am talking about models like YOLO, ResNet, MobileNets, etc.
- ajnin 2y agoIt requires mlx but it is an Apple silicon-only library as far as I can tell. How is it supposed to be (I quote) "iPhone, iPad, Android, Mac, Linux, pretty much any device" ? Has it been tested on anything else than the author's MacBook ?
- orsorna 2y agoOne of the maintainers has a video demo on his twitter claiming iOS, android and Linux. Some of the code is not released and I wish they were advertising that properly.
- tama_sala 2y agoThe library already has tinygrad support it seems, so it's not limited to Apple & MLX
- orsorna 2y agoThat is true. However (as of two days ago, it may have rapidly changed since then) the python program did not differentiate based on your architecture and would try to import mlx regardless if it's installable on your system or not, causing import errors.
- alexandercheema 2y agoThis is fixed now, with these commits: - https://github.com/exo-explore/exo/commit/dbbc7be57fb1871d2b8f72ad345088ed86aec1fc https://github.com/exo-explore/exo/commit/dbbc7be57fb1871d2b... - https://github.com/exo-explore/exo/commit/ce46f000591d8d59c1c21fbec7b668ada65c5969 https://github.com/exo-explore/exo/commit/ce46f000591d8d59c1... Please keep the bug reports coming, we're moving fast to get this stable on all platforms.
- lopuhin 2y agoThe README says they plan to add llama.cpp support which should cover a lot of targets, also they have tinygrad already integrated I think.
- pyinstallwoes 2y agoSwarm compute should be the norm for all compute - so much unused cpu across all the devices we collectively own.
- KronisLV 2y agoThis might not work for use cases where you need low latency, but for longer winded processing it would be amazing if possible. For example, if I have a few servers, laptop (connected to power) as well as a desktop PC and they’re all connected to a fast local network, it’d be great to distribute the task of rendering a video or working with archive files across all of them.
- greggsy 2y agoThose are two precise examples that benefit from single core compute power, and are wholly unsuited to distributed computing…
- _factor 2y agoThis exists: https://aihorde.net/ https://aihorde.net/ I haven’t tried it, and not the norm, but I agree it should be more common. We have a global supercomputer with higher latency, but still a supercomputer.
- 2y ago
- christkv 2y agoIs apple silicon with a lot of memory 32Gb and up still considered a cheapish way to run models or are there other options now?
- talldayo 2y agoA good Apple Silicon Mac with 32gb of RAM will cost you over $2,000 on-sale. For that price you might as well buy an Nvidia machine instead, either two 3090s or a 64gb Jetson Orin board would be both cheaper and faster. The markup on Apple hardware is so big that I just don't think "cheapish" will ever be a way to describe the position they hold in the AI market. Apple's current budget lineup gets smoked by an RTX 3060 in a cheap Linux homeserver; the bar for high-value AI has been raised pretty high.
- makmanalp 2y agoQuestion - if large clusters are reporting that they're seeing gains from using RDMA networks because communication overhead is a bottleneck, how is it possible that this thing is not massively bottlenecked running over a home network?
- DistractionRect 2y agoI suspect that most of the devices you'd expect to find in your consumer cluster are too small/slow to saturate the link. Edit: it's also a matter of scale. You probably have a small number of small/slow devices in a consumer network versus a lot of large/fast devices in your enterprise cluster.
- derefr 2y agoI haven't looked into exactly what this project is doing, but here's my understanding: Inference across O(N) pre-trained hidden layers isn't exactly an "embarrassingly parallel" problem, but it is an "embarrassingly pipeline-able" problem (in the CPU sense of "pipelining.") Each device can keep just one or a few layers hot in their own VRAM; and also only needs to send and receive one small embedding (<1MB) vector per timestep — which is so trivial that it's easily achievable in realtime even if all the devices are on wi-fi, talking to the same router, in your "noisy" apartment where 100 other neighbours are on the same bands. (To put it another way: running a single inference job, has more forgiving realtime latency+throughput requirements than game streaming!) Assuming that you have a model that's too big for any of your home machines to individually hold; and that all you care about is performance for single-concurrent-request inference on that model — then in theory, you just need one GPU of one node of your homespun Beowulf GPU cluster to have enough VRAM to keep the single largest layer of your model always-hot; and then other smaller devices can handle keeping the smaller layers always-hot. And the result should be faster than "overloading" that same model on that single largest-VRAM device and having some layers spill to CPU, or worse yet, having the GPU have to swap layers in and out repeatedly with each inference step. (Also, if you're wondering, in the case where a single machine/node has multiple GPUs — or a GPU+VRAM and also a CPU+RAM! — you can treat this as no different than if these were multiple independent nodes, that just-so-happen to have a very efficient pipeline communication channel between them. As the VRAM+computation cost of running inference far outweighs the communication overhead of forward propagation during inference, a home-network inference-pipelining cluster scheduler like this project, would still likely "schedule" the model's layers purely in consideration of the properties of the individual GPU+VRAM (or CPU+RAM), rather than bothering to care about placement.) --- That being said, AFAIK training is "pipeline parallelizable" exactly as inference is. And people training models do do this — but almost always only across multiple top-of-the-line GPUs in one machine; not across multiple machines. When you think about what pipelining achieves for training — all you get is either: 1. the ability to use a bunch of small-aggregate-VRAM nodes to achieve the aggregate training capacity of fewer, larger-aggregate-VRAM nodes — but with more power consumption = higher OpEx; and where also, if you scale this to O(N), then you're dumping a quadratic amount of layer-propagation data (which is now both forward-prop and backprop data, and backprop data is bigger!) over what would likely be a shared network just to make this work. (If it's not a shared network — i.e. if it's Infiniband/other RDMA — then why did you spend all that CapEx for your network and not on your GPUs!?) 2. the ability to pipeline a bunch of large-aggregate-VRAM nodes together to train a model that will then never be able to be deployed onto any single node in existence, but can instead only exist as a "pipelined inference model" that hogs O(log N) nodes of your cluster at a time for any inference run. Which makes cluster scheduling hell (if you aren't just permanently wiring the scheduler to treat O(log N)-node groups as single "hyper-nodes"); makes it so that you'll never be able to practically open-source the model in a way anybody but other bigcorps could ever run it (if that's something you care about); and very likely means you're cutting the concurrent-inference-request-serving capacity of your huge expensive GPU cluster by O(log N)... which the product team that allowed that cluster to be budgeted is really not gonna like. That being said, I imagine at some point one of these proprietary "Inference-as-a-Service" models has been trained at a layer size that puts it into pipelined-inference-only territory, temporarily. Doing so would be the ML engineer's equivalent to the CPU engineer's "we have no fundamentally clever advance, so this quarter we'll just crank up the clock frequency and deal with the higher TDP." (Heck, maybe GPT-4o is one of these.) --- What people with GPU clusters want, is 1. for the output of the process to be a model that runs on a single (perhaps multi-GPU) node; and 2. for the process itself to be mostly-shared-nothing with as little cross-node communication burden as possible (such that it's just a question of building highly internally communication-efficient nodes, not so much highly-communication-efficient clusters.) And both of those goals are achieved by sizing models so that they fit within a single node; continuously fanning out streams of training data to those nodes; and then periodically fanning back in model-weights (or model-weight deltas) in an AllReduce operation, to merge the learning of O(N) independently-training nodes to become the new baseline for those nodes. (If you'll note, this architecture doesn't put any latency requirements on the network, only some monstrous throughput requirements [at the fan-in step] — which makes it a lot easier to design for.)
- ulrischa 2y agoDoes somebody know if it runs on a raspberry?
- alexandercheema 2y agoIt *should* but I haven't tried it. I will try it. Updated in this issue: We could also try raspberry pi + coral usb tpu (https://coral.ai/products/ https://coral.ai/products/) - that might be a killer combo for super cheap home ai cluster.
- alexandercheema 2y agoIssue link: https://github.com/exo-explore/exo/issues/11 https://github.com/exo-explore/exo/issues/11
- yjftsjthsd-h 2y ago> coral usb tpu I thought those were so memory limited that there was no useful way to run an LLM on them?
- dcreater 2y agoThis is a great ideal and user friendly as well. Has the potential of converting multiple old devices overnight from being useless. However, I wish they had provided some results on tok/s, latency with some example setups.
- alexandercheema 2y agoWe didn't expect this to blow up so quickly. A lot of work needs to be done on getting different setups working. I have made an issue here: https://github.com/exo-explore/exo/issues/11 https://github.com/exo-explore/exo/issues/11
- DiogoSnows 2y agoThis is great work! I will keep an eye (and maybe even try to contribute). Looking back at the beginning of Google, I think their use of hardware and hardware agnostic platform likely contributed to support growth at lower cost. We need more of that in the AI era
- alexandercheema 2y agoThank you for the support! I agree on the cost point, and personally I don’t want to live in a world where all AI runs on H100s in a giant datacenter controlled by one company.
- throwawaymaths 2y agoIs this sensible? Transformers are memory bandwidth bound. Schlepping activations around your home network (which is liable to be lossy) seems like it would result in atrocious TPS.
- alexandercheema 2y ago"Transformers are memory bandwidth bound" - this is the precise reason why this makes sense. If a model doesn't fit into memory on a single device, it needs to be incrementally loaded into memory (offloading), which is bottlenecked by memory bandwidth. Splitting the model over multiple devices avoids this, instead trading off for latency of communicating between nodes. The network bandwidth requirements are minimal since only the activations (intermediary embeddings) are passed between devices. For Llama-3-8B these are ~10KB, for Llama-3-70B these are ~32KB.
- boroboro4 2y agoIt worth noticing that number you're quoting is for embeddings between layers. If you split your model between 5 nodes you will need to send this 32kb 5 times. Also it's per token. Meaning if you process 1K tokens it turns to be 32 MB of data, 1M tokens - 32 GB...
- cess11 2y agoI look forward to something similar being developed on top of Bumblebee and Axon, which I expect is just around the corner. Because, for me, Python does not spark joy.
- alexandercheema 2y agoRepo author here. This sounds interesting. Could you elaborate on the benefits of Bumblebee / Axon?
- cess11 2y agoThey run on the BEAM, and there are related IoT-platforms like Nerves. If find that to be a much nicer runtime than (C)Python. Edit: I don't know where else to begin. It's a runtime that has lightweight processes, excellent observability, absurdly good fault tolerance, really nice programming languages and so on. It's designed for distributed computing.
- alexandercheema 2y agoFascinating, will check this out! I wanted to focus on Python first to build this quickly, test out ideas and iterate. This seems like a good option for a switch. Do you know if any of these can run on Apple/Android devices?
- cess11 2y agoI avoid touching Apple devices but anything that can expose a Linux shell can run the BEAM. There are two main projects for small devices, https://nerves-project.org/ https://nerves-project.org/ for more ordinary SoC-computers and https://www.atomvm.net/ https://www.atomvm.net/ for stuff like ESP32-chips. On Android you've got Termux in F-Droid and can pull in whatever BEAM-setup you want. That's how I first started dabbling with the BEAM, I was using a tablet for most of my recreational programming and happened to try it out and got hooked. Erlang is pretty weird, but it just clicks for some people so it's worth spending some time checking it out. Elixir is a really nice Python-/Ruby-like on the BEAM, but with pattern matching, real macros and all the absurdly powerful stuff in the Open Telecom Platform.
- yjftsjthsd-h 2y agoUnfortunately I don't see any licensing info, without which I'm not touching it. Which is too bad since the idea is really cool.
- deleted 2y ago[deleted]
- alexandercheema 2y agoThanks for pointing out that. Fixed https://github.com/exo-explore/exo/blob/main/LICENSE https://github.com/exo-explore/exo/blob/main/LICENSE
- yjftsjthsd-h 2y agoExcellent, thank you:)
- bufferoverflow 2y ago[flagged]
- deleted 2y ago[deleted]
- Jayakumark 2y agoJust got https://github.com/distantmagic/paddler https://github.com/distantmagic/paddler working across 2 machines on windows, for load balancing, This will be next level and useful for Llama 400B to run across multiple machines. But looks like windows support is not there yet.
- fudged71 2y agoSince this is best over a local network, I wonder how easy you could make the crowdsourcing aspect of this. How could you make it simple enough for everyone that's physically in your office to join a network to train overnight? Or get everyone at a conference to scan a QR code to contribute to a domain specific model.
- alexandercheema 2y agoThat’s where we want to get eventually. There’s a lot of work that needs to be done but I’m confident we’ll get there. Give us 3 months and it’ll be as simple as running Dropbox.
- pkeasjsjd 2y agoIt bothers me that they don't talk about security here, I don't like it at all.
- alexandercheema 2y agoYou’re right. The assumption right now is that you’re running on trusted devices on your own local network. I will add a section in the README.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- Obertr 2y agoOkey, I'll say it. It will not work because of network bottlneckes. You need to be sending gigabytes of Data. so by definition you need (1) good internet 20mb/s+ and (2) good devices. This thing will not go any further than cool demo on twitter. Please prove me wrong.
- alexandercheema 2y agoTry it out - don't trust me! The way this works is that each device holds a partition of the model (for now a continuous set of layers). E.g. let's say you have 3 devices and the model is 32 layers. Device 1 could hold layers 1-10, device 2 holds 11-20 and device 3 holds 21-32. Each device executes the layers it's responsible for and passes on the output of its last layer (the activations) to the next device. The activations are ~8KB for Llama-3-8B and ~32KB for Llama-3-70B (it's linear in the number of parameters in that layer and Llama-3-70B has more layers). Generally the larger the model gets (in terms of parameters), the more layers it ends up having, so we end up with sub-linear scaling so I expect Llama-3-405B to have activations on the order of ~100KB. This is totally acceptable to send over a local network. The main issue you run into is latency, not bandwidth. Since LLMs are autoregressive (tokens are generated serially), additional latency limits throughput. However, over a local network latency is generally very low (<5ms in my experience). And if not, it's still useful depending on the use-case since you can get a lot of throughput with pipeline parallelism (overlapping requests): https://pytorch.org/docs/stable/pipeline.html https://pytorch.org/docs/stable/pipeline.html
- gnicholas 2y agoThis is great! I really wish Apple allowed your device to query a model you host instead of skipping to their cloud (or OpenAI). I'd love to have a Studio Pro running at home, and have my iPhone, iPad, Mac, and HomePod be able to access it instead of going to the cloud. That way I could have even more assured privacy, and I could choose what model I want to run.
- alexandercheema 2y agoDo you mean with Apple Intelligence? You can already query models you host from Apple using exo or even just local on-device inference.
- gnicholas 2y agoDoes this work with Siri? I'm not running the beta so am not familiar with the features and limitations, but I thought that it was either answering based on on-device inference (using a closed model) or Apple's cloud (using a model you can't choose). My understanding is that you can ask OpenAI via an integration they've built, and that in the future you may be able to reach out to other hosted models. But I didn't see anything about being able to seamlessly reach out to your own locally-hosted models, either for Siri backup or anything else. But like I said, I'm not running the beta!
- throwaway2562 2y agoHow long before the accursed crypto kids try to tokenise token generation with Exo clusters?
- rbanffy 2y agoThe all important question: When there’s only one device left on the network, will it sing Daisy Bell?
- alexandercheema 2y agoNot yet, should I make an issue for it?
- rbanffy 2y agoIt'd be nothing but appropriate.
- Aerbil313 2y agoI can't wait to see malware which downloads and runs LLMs on remote C&C server command.