8 ms·
Raspberry Pi's New AI Hat Adds 8GB of RAM for Local LLMs
- dwedge 9mo ago> In practice, it's not as amazing as it sounds. 8GB RAM for AI on a Pi sounds underwhelming even from the headline
- deleted 9mo ago[deleted]
- agent013 9mo agoA good illustration of how “can run LLM” ≠ “makes sense to run LLM”. A prime example of how numbers in specs don’t translate into real UX.
- phito 9mo agoSounds like some PM just wanted to shove AI marketing where it doesn't make sense.
- buran77 9mo agoI think Raspberry lost the magic of the older Pis, they lost that sense of purpose. They basically created a niche with the first Pis, now they're just jumping into segments that others created and are already filled to the brim with perhaps even more qualified competition. Are they seeing a worthwhile niche for the tinkerers (or businesses?) who want to run local LLMs with middling performance but still need full set of GPIOs in a small package? Maybe. But maybe this is just Raspberry jumping on the bandwagon. I don't blame them for looking to expand into new segments, the business needs to survive. But these efforts just look a bit aimless to me. I "blame" them for not having another "Raspberry Pi moment". P.S. I can maybe see Frigate and similar solutions driving the adoption for these, like they boosted Coral TPU sales. Not sure if that's enough of a push to make it successful. The hat just doesn't have any of the unique value proposition that kickstarted the Raspberry wave.
- ulnarkressty 9mo agoThat niche was long taken over by cheap Chinese SBCs, so they have to innovate somehow. Their only advantage that remains is the community.
- mschuster91 9mo agoThe thing with the Chinese SBCs is that (like every other player in the embedded world) they don't give a flying fuck about upstreaming their code to the Linux kernel. Of course, Raspberry Pi just like everyone else has their custom patches, but at least to my knowledge you can use a straight Linux kernel and still have a running system.
- ChromaticPanic 9mo agoOr just updates in general. Need documentation? Good luck.
- joe_mamba 9mo agoYep. RPi foundation lost the plot a long time ago. The original RPi was in a league of its own when it launched since nothing like it existed and it was cheap. But now if I want some low power linux PC replacement with display output, for the price of the latest RPi 5, I can buy on the used market a ~2018 laptop with a 15W quad core CPU, 8GB RAM, 256 NVME and 1080p IPS display, that's orders of magnitude more capable. And if I want a battery powered embedded ARM device for GPIO over WIFI, I can get an ESP32 clone, that's orders of magnitude cheaper. Now RPi at sticker price is only good for commercial users since it's still cheaper than the dedicated industrial embedded boards, which I think is the new market the RPI company caters to. I haven't seen any embedded product company that hasn't incorporate RPis in its products they ship, or at least in their lab/dev/testing stage, so if you can sell your entire production stock to industrial users who will pay top dollar, why bother making less money selling to consumers, just thank them for all the fish. Jensen Huang would approve.
- haunter 9mo ago
- deleted 9mo ago[deleted]
- moffkalast 9mo ago> The Pi's built-in CPU trounces the Hailo 10H. Case closed. And that's extremely slow to begin with, the Pi 5 only gets what, a 32 bit bus? Laughable performance for a purpose built ASIC that costs more than the Pi itself. > In my testing, Hailo's hailo-rpi5-examples were not yet updated for this new HAT, and even if I specified the Hailo 10H manually, model files would not load Laughable levels of support too. As another datapoint, I've recently managed to get the 8L working natively on Ubuntu 24 with ROS, but only after significant shenanigans involving recompiling the kernel module and building their library for python 3.12 that Hailo for some reason does not provide outside 3.11. They only support the Pi OS (like anyone would use that in prod) and even that is very spotty. Like, why would you not target the most popular robotics distro for an AI accelerator? Who else is gonna buy these things exactly?
- renewiltord 9mo agoWhat’s the current state of the art in low power wake word and speech to text? Has anyone written a blog post on this? I was able to run a speech to text on my old Pixel 4 but it’s a bit flaky (the background process loses the audio device occasionally). I just want to take some wake word and then send everything to remote LLM and then get back text that I do TTS on.
- geerlingguy 9mo agoMaybe not SOTA but the HA Voice Preview Edition [1] in tandem with a Pi 5 or some similar low-power host for the Piper / Whisper pipeline is pretty good. I don't use it but was able to get an Alexa/Google Home-like experience going with minimal effort. I was only using it for local Home Assistant tasks, didn't try anything further like retrieving sports scores, managing TODO lists, or anything like that. [1] https://www.home-assistant.io/voice-pe/ https://www.home-assistant.io/voice-pe/
- monocasa 9mo agoWake word can be tiny. Like 10k weights and can run on an esp32 or similar with plenty of compute to spare. TinyML is a book that goes through the process of building a wake word model for such constrained environments.
- folmar 9mo agoWake word is not expensive, you can do it on esp32 https://docs.espressif.com/projects/esp-sr/en/latest/esp32s3/wake_word_engine/README.html https://docs.espressif.com/projects/esp-sr/en/latest/esp32s3... (and then send audio to something more beefy as TTS will be marginal at best).
- huntercaron 9mo agoGlad Jeff was critical here they need a bit of a wake up call it seems.
- speedgoose 9mo agoIs there any usefulness with the small large language models, outside perhaps embeddings and learning? I fail to see the use-case on a Pi. For learning you can have access to much better hardware for cheaper. Perhaps you can use it as a slow and expensive embedding machine, but why?
- kouteiheika 9mo agoA natural language based smart home interface, perhaps? Tiny LLMs are pretty much useless as general purpose workhorses, but where they shine is when you finetune them for a very specific application. (In general this is applicable across the board, where if you have a single, specific usecase and can prepare appropriate training data, then you can often fine-tune a smaller model to match the performance of a general purpose model that is 10x its size.)
- michaelmior 9mo agoI think there's a lot of room to push this further. Of course there are LLMs being used for this case and I guess it's nice to be able to ask your house who the candidates were in the Venezuelan presidential election of 1936, but I'd be happy if I could just consistently control devices locally and a small language model definitely makes that easier.
- cmpxchg8b 9mo ago8GB? What is this, an LLM for ants?
- mlvljr 9mo ago[dead]
- matja 9mo agoIt's more about demonstrating what's possible on a Pi than expecting GPT-4 level performance. It's designed for LLMs that specialize in tiny, incredibly specific tasks. Like, "What's the weather in my ant farm?" ;) The vision processing boost is notable, but not enough to justify the price over existing HATs. The lack of reliable mixed-mode functionality and sparse software support are significant red flags. (This reply generated by an LLM smaller than 8GB, for ants, using the article and comment as context).
- kirurik 9mo agoYou can run some models pretty decently using CPU inference only, things like Gemma 3 that are built for exactly that use case or some tiny speech to text models via llama.cpp that I have tested out (not so good). Although not the best for "heavy" tasks, if you just need a decent text generator that can produce more or less sensible, generic output you are good to go.
- deleted 9mo ago[deleted]
- kotaKat 9mo ago"For example, the Hailo 10H is advertised as being used for a Fujitsu demo of automatic shrink detection for a self-checkout." ... why though? CV in software is good enough for this application and we've already been doing it forever (see also: Everseen). Now we're just wasting silicon.
- Barathkanna 9mo agoAs an edge computing enthusiast, this feels like a meaningful leap for the Raspberry Pi ecosystem. Having a low-power inference accelerator baked into the platform opens up a lot of practical local AI use cases without dragging in the cloud. It’s still early, but this is the right direction for real edge workloads.
- saagarjha 9mo agoWhich ones?
- teekert 9mo agoAt this moment my two questions for these things are: 1. Can I run a local LLM that allows me to control Home Assistant with natural language? Some basic stuff like timers, to do/shopping lists etc would be nice etc. 2. Can I run object/person detection on local video streams? I want some AI stuff, but I want it local. Looks like the answer for this one is: Meh. It can do point 2, but it's not the best option.
- worksonmine 9mo ago1. Probably, but not efficiently. But I'm just guessing I haven't tried local LLMs yet. 2. Has been possible in realtime since the first camera was released and has most likely improved since. I did this years ago on the pi zero and it was surprisingly good.
- noodletheworld 9mo ago> Can I run a local LLM that allows me to control Home Assistant with natural language? Some basic stuff like timers, to do/shopping lists etc would be nice etc. No. Get the larger PI recommended by the article. Quote from the article: > So power holds it back, but the 8 gigs of RAM holds back the LLM use case (vs just running on the Pi's CPU) the most. The Pi 5 can be bought in up to a 16 GB configuration. That's as much as you get in decent consumer graphics cards1. > Because of that, many quantized medium-size models target 10-12 GB of RAM usage (leaving space for context, which eats up another 2+ GB of RAM). … > 8 GB of RAM is useful, but it's not quite enough to give this HAT an advantage over just paying for the bigger 16GB Pi with more RAM, which will be more flexible and run models faster. The model specs shown for this device in the article are small, and not fit for purpose even for the relatively trivial use case you mentioned. I mean, look, lots of people have lots of opinions about this (many of them wrong); it’s cheap, you can buy one and try… but, look. The OP really gave it a shot, and results were kind of shit. The article is pretty clear. Don’t bother. You want a device with more memory to mess around with for what you want to do.
- rballpug 9mo agoCatalog TG 211, 1000 Hz.
- endymion-light 9mo agocan't wait to not be able to buy it, and also for it to be more expensive than a mini-computer I buy a raspberry pi because I need a small workhorse - I understand adding RAM for local LLMs, but it would be like a raspberry pi with a GPU, why do i need it when a normal mini machine will have more ram, more compute capacity and better specs for cheaper?
- rocketvole 9mo agoa lot of people buy rpis because they are the only reasonable option for connectivity with power. I'm not sure what other devices you can get that have gpio and mipi connectivity with the ability to (potentially) run vlms and llms on them. I daresay they could charge more than a comparably specced computer (if they don't already) and they would still be a viable purchase.
- endymion-light 9mo agoSurely with this hat you don't have any access to any GPIO? Unless i'm missing something - which is where i'm like why not just buy a NUC with similiar RAM for far less.
- Rohansi 9mo agoYou do still have access to the GPIO. This HAT [1] stacks on top of the GPIO connector but passes through all the pins so you can still use them. This one is connected through PCIe so it shouldn't be blocking off any pins from use, unless you wanted an NVMe SSD hooked up! [1] https://www.raspberrypi.com/news/introducing-raspberry-pi-hats/ https://www.raspberrypi.com/news/introducing-raspberry-pi-ha...
- endymion-light 9mo agoAh great! Yeah that makes the benefit a lot more clear. I'm used to hats that seem to lock you out of GPIO use
- t43562 9mo agoIn the UK I've never seen the hailo hats (which are quite old BTW) advertised for LLMs. The presented usecase has been object detection from lots of video cameras in realtime. They seem very fast and I certainly want to use that kind of thing in my house and garden - spotting when foxes and cats arrive and dig up my compost pit, or if people come over when I'm away to water the plants etc. [edit: I've just seen the updated version in Pimonori and it does claim usefulness for LLMs but also for VLMs and I suspect this is the best way to use it].
- esskay 9mo agoWhat a pointless product to waste time making.
- wyldfire 9mo agoI wonder -- how does this thing compare against the Rubik Pi [1]? [1] https://rubikpi.ai/ https://rubikpi.ai/
- philipallstar 9mo agoIs that affiliated with raspberry pis in some way, or are they just freeloading on the "pi" suffix to confuse people?
- Elfener 9mo ago> are they just freeloading on the "pi" suffix to confuse people? Yes, but that is normal I guess: - https://banana-pi.org/ https://banana-pi.org/ - http://www.orangepi.org/index.html http://www.orangepi.org/index.html - https://radxa.com/products/rockpi https://radxa.com/products/rockpi
- wyldfire 9mo agoThey are "freeloading" indeed but in this case it is a valuable indicator that it should be physically compatible with many add-ons/attachments that are compatible with the Raspberry Pi.
- joelthelion 9mo ago8GB is really low. That said, perhaps there is a niche for slow LLM inference for non-interactive use. For example, if you use LLMs to triage your emails in the background, you don't care about latency. You just need the throughput to be high enough to handle the load.
- klft 9mo ago[flagged]
- imtringued 9mo agoThis looks pretty nice for what it is. However, the RAM is a bit oversized for the vast majority of applications that will run on this, which is giving a misleading impression of what it is useful for. I once tried to run a segmentation model based on a vision transformer on a PC and that model used somewhere around 1 GB for the parameters and several gigabytes for the KV cache and it was almost entirely compute bound. You couldn't run that type of model on previous AI accelerators because they only supported model sizes in the megabytes range.
- MORPHOICES 9mo ago[dead]
- Lio 9mo agoI've seen the AI-8850 LLM Acceleration M.2 Module advertised as an alternative RPi accellorator (you need an M.2 hat for it). That's also limited to 8Gb RAM so again you might be better off with a larger 16Gb Pi and using the CPU but at least the space is heating up. With a lot of this stuff it seems to come down to how good the software support is. Raspberry Pis generally beat everything else for that.
- Havoc 9mo agoThat seems completely and utterly pointless. A NPU that adds to price but underperforms a rasp cpu? You get SBC with 32gb ram… Nevermind the whole minipc ecosystem which will crush this
- nottorp 9mo agoHmm. Can this "AI" hardware - or any other "AI" hardware that isn't a GPU - be used for anything other than LLMs? YOLO for example.
- dismalpedigree 9mo agoYes. The Hailo chips are mainly for AI vision models. This is the first time I have seen them pushed for LLM. They are very finicky and difficult to setup outside of the examples. Documentation is inconsistent and the models have to be converted to a different format to run. It is possible to run a custom yolo8 model, but is challenging.
- _ea1k 9mo agoI'd expect that kind of thing to be the primary use case, tbh. Maybe even running whisper models? If it could run whisper, it'd be a solid addition to a pi based home assistant setup.
- vander_elst 9mo agoI had a couple of Pis that I wanted to use as a Media center, I always had some small issues that created a suboptimal experience. Went for a regular 2nd hand amd64 with a small form factor and never looked back, much better userspace support and for my use case a much smoother experience, no lags no memory swapping and if needed I can just buy a different memory bank or a different component. I have no plans to use a raspberry pi any time soon. I am not sure these days if they really still have a niche to fill and if yes how large this niche is.
- JustFinishedBSG 9mo agoIt's useless for LLMs and it's actually slower than Hailo 8H for standard vision tasks, so, why ?
- myrmidon 9mo agoAre there significant usecases for the really small LLMs right now (<10b distills and such)? My impression so far was that the resulting models are unusably stupid, but maybe there are some specific tasks where they still perform acceptably?
- arkmm 9mo agoThey're still very good for finetuned classification, often 10-100x cheaper to run at similar or higher accuracy as a large model - but I think most people just prompt the large model unless they have high volume needs or need to self host.
- syntaxing 9mo agoInteresting idea. I think the Jetson Orin Nano is a better purchase for this application. The main downside is the RAM is shared so you lose about 1G from the OS overhead.
- giantg2 9mo agoWhat not use a USB Coral TPU? Seems to do mostly the same stuff and is half the price.
- geerlingguy 9mo agoCoral is many times slower at this point (2 TOPS IIRC), but if it meets your needs, it's okay.
- giantg2 9mo agoYeah, I think it's 4 TOPS, so I see.
- incomingpain 9mo agoI wonder if this is the magic hardware for LiquidAI/LFM2.5-Audio-1.5B Dont need more than 8gb. It'll be enough power. IT can do audio to audio.
- djhworld 9mo agoA few years ago this product would have just been called an ML Accelerator and marketed as helping accelerate ML workloads like object detection in images Hitching their wagon to the AI train comes with different expectations, leading to a mixed bag of reviews like this.
- yawniek 9mo agoi wonder how the Hailo 10H compares to Axera AX8850. add on boards seem to be cheaper and its a full SoC that can also draw much more power.
- yjftsjthsd-h 9mo agoAny chance you can split layers so that some run on the CPU and some run on this board to let you run bigger models and/or get better performance?
- xp84 9mo agoGigabytes?? In THIS economy?
- 1970-01-01 9mo agoIt does seem like this is a marketing gimmick. I doubt it will have worked in terms of sales numbers. "What else does it do?"
- venturecruelty 9mo agoAre computers even for anything else anymore? Is it just AI until the bubble pops? This is exhausting.
- janalsncm 9mo agoExcited to see more hardware competition at this level. Models that can run on this amount of RAM are right in the sweet spot of small enough to train on consumer-grade GPUs (e.g. 4090) but big enough to do something interesting (simple audio/video/text processing). The price point is still a little high for most tasks but I’m sure that will come down.
- rootusrootus 9mo agoWithin minutes of each other I can read a bunch of comments about how an 8GB AI hat for RPi would be grand for LLMs, while another contingent points out that my M2 MAX 96GB MacBook is completely useless for LLMs. All I can say is at least the latter is also a great laptop.
- bigyabai 9mo agoFor server use, the Pi is cheaper, better-supported and lower power. I really cannot recommend that anyone spend $2,500+ to run inference on a laptop that is slower than $300 GPUs.
- sxzygz 9mo agoMy primary concern with this offering is vendor of the enabling silicon. I think it’s important to consider who you do business with. The industrial uptake of RPi boards has poisoned their mandate and made them forget making a world where all children can discover the magic of computing, not just the special chosen ones.
- tonymet 9mo agoCan someone share a non-theoretical LLM application that demands local inference and pocketable hardware? The internet hype for these is sky high, but the actual results are disappointing.
- 5ersi 9mo agoThis AI 8GB HAT is $150, while better performing Pi5 8GB is $100. Makes no sense to buy the AI HAT.
- bdavbdav 9mo agoHave we not all given up on pis at this stage? I’ve got ESP32s for edge I/O, Nucs/VMs for the crunch. What’s the Pi doing?
- scq 9mo agoThere are definitely use cases. Pis have lower power consumption than NUCs. This is the main reason I went for one to run Home Assistant rather than a NUC. I have a NAS/home server that I could put it on but I don't want my home automation going down when I tinker with it.
- bdavbdav 9mo agoI think I’ve got too used to VM snapshotting to turn away from it for HA.
- Western0 9mo agocan You add gemma 3n to benchmark? this is less tahan normal 3 gemma but still good in small memory