10 ms·
The Orange Pi 5 Plus
- user_7832 3y ago> Keeping in mind that the i7 runs at nearly six times the wattage, this is pretty good3, and the key point here is that the Orange Pi 5+ generated the output at a usable pace–slower than what you’d get online from ChatGPT, but still fast enough that it is close to “reading speed”–whereas the Raspberry Pi 4 was too slow to be usable. > This also means I could probably (if I could find a suitable model that fit into 4GB RAM) use the Orange Pi 5+ as a back-end for a “smart speaker” for my home automation setup (and I will probably try that in the future). This is pretty interesting for me. I had (wrongly, I suppose) assumed that hardware requirements for LLMs were “have a recent NVidia GPU” but this proves otherwise.
- rcarmo 3y agoHi! Author here. Mind that I had to test with relatively small, "dumb" LLMs. I have no doubt I can run whisper.cpp on the RK3588 _and_ a tuned LLM to handle intents, but it won't be a very smart one (I am hoping to find a good way to run quantized Mixtral, but given the RAM constraints on the 4GB board I didn't even try). Edited to add: I'm looking for something like https://news.ycombinator.com/item?id=38704982 https://news.ycombinator.com/item?id=38704982 (LLM in a Flash) even if I find something with 16/32GB of RAM, which is why I looked at OnnxStream as well (but of course the inference in LLMs is different, so I can't leverage the NVMe just yet).
- user_7832 3y agoThanks! How "bad" are these LLMs though, especially for smart home-esque basic tasks? I'd imagine them to be alright-ish?
- rcarmo 3y agoWell, the ones I tried that would fit into 4GB RAM have trouble following directions and inconsistent output. I can't really tell you (yet) which would fit into, say, 16GB RAM and work consistently (and fast enough to, say, turn a light off faster than it would take you to get up and reach for the switch), but I'll eventually get to it...
- user_7832 3y agoThank you, that makes sense. I think I'll stick with my google hub for now haha
- FergusArgyll 3y agoDolphin-phi (uncensored tweaked version of msft phi) has been pretty good, I've been testing for only a couple days. It's 2.7B params, so depending on how much RAM the os is using you might be able to run that on 4GB. I run it on 8GB windows using wsl2/ollama and the OS takes up around 5GB (I think), so maybe....
- Havoc 3y agoI’ve run larger models on mine - work but quite slow. You can get a 32 orange off aliexpress today already
- timnetworks 3y agoInferencing can be done in software entirely (e.g. INT8) but it's very slow compared to GPU or APU. nVidia cornered the market because everything (tensorflow and everything after) is optimized for it, but you can get good results on AMD now, and on ARC too in some cases. And slow results entirely in software (CPU-RAM), which for personal and non-constant use may be just fine too.
- user_7832 3y agoThanks! Do you have any guides/websites/github repos for running these models on CPUs?
- rcarmo 3y agoOllama will (nearly always) work provided you have enough RAM. I was actually pretty surprised that it didn't work on my N5105 (which has 16GB) because it relies on AVX instructions...
- user_7832 3y agoThanks! Someone else mentioned llama.cpp but it appears that ollama is just a gui frontend for llama (which is good because I find guis easier). I'll hopefully set it up soon!
- FergusArgyll 3y agoIt's not a GUI it's a cli but very easy to use "ollama run {model}". you can also `ollama serve` which serves an api, and then you can use or build a simple gui.
- user_7832 3y agoThanks, I’ll keep that in mind!
- jmorgan 3y ago
- nabakin 3y agoOver the past year or so various projects have made it possible to run LLMs on just about anything. Some GPUs are still better than others like Nvidia GPUs are still the best for token throughput (via TensorRT-LLM), but AMD GPUs are competitive (via vLLM) and even CPUs can run LLMs at decent speeds (via Llama.cpp).
- user_7832 3y agoThank you!
- 38 3y agois their website dead or something? http://www.orangepi.org/html/hardWare/computerAndMicrocontrollers/details/Orange-Pi-5-plus.html http://www.orangepi.org/html/hardWare/computerAndMicrocontro...
- rcarmo 3y agoWell, it was there when I revised the draft early this morning... I'm betting having the link here won't help their uptime :)
- 38 3y agoI'm not removing it. with all due respect, without a working URL the article is kind of useless to me. ultimately I want to buy or see the official pages on the hardware, which currently doesn't seem possible.
- rcarmo 3y agoI wasn't asking you to, just pointing out the impact :)
- jerrysievert 3y agoorange pi's are officially available on amazon or aliexpress. they do not have their own online shop: https://www.amazon.com/stores/OrangepiCreateNewFuture/page/FB8A3E37-4849-452A-B0BC-036E3DE6991C https://www.amazon.com/stores/OrangepiCreateNewFuture/page/F... unfortunately, there are too many sellers selling them on aliexpress for me to find their store there, so you will have to wait until they recover from the hug. prices are the same on each, though if you are purchasing from the united states. (edited for clarity)
- politelemon 3y agoIs it running on an OrangePi perhaps?
- rcarmo 3y agohttps://archive.is/eLIyr https://archive.is/eLIyr
- ComputerGuru 3y agoI can't take any of the benchmarks seriously when he is using very different hardware across the tests. I can somewhat understand comparing Orange Pi NVMe to the RPi 4 SATA because that's what ships out-of-the-box (but there's an NVMe hat available), even though it'll be rate-limited to USB 3.0 speeds. But I can't understand comparisons to the u59 micro that are actually run on an Intel machine and then not using an NVMe in the Intel for comparison. This abounds across all tests, from the very first I/O tests that show the Orange Pi 5+ beating both Intel configurations to the OnnxStream test that shows Intel beating the Orange Pi 5+ even though the Intel unit has to load/stream the model from its paltry SATA disk while the Orange Pi 5+ is outfit with an NVMe drive.
- deleted 3y ago[deleted]
- rcarmo 3y agoHi, author here. I tested with what I have, and with what I currently use to work and test with that is in the general price/performance range. If that isn't obvious, then I have to apologize and make an explicit note of that. My u59 ships with SATA SSDs, as it happens. I do have an Intel i7 13th Gen with PCIe 4.0 NVMes (and several modern Macs), but that would be so far off base (and so expensive) that it isn't even comparable. The i7-6700 is much closer in "value", if you will. However, you are mis-reading the way the OnnxStream test works. It is still CPU-bound for the most part.
- adrian_b 3y agoI did not understand your complaints, so I have searched the specifications of Beelink u59. This is a small computer with the previous generation of Intel Atom CPUs (Jasper Lake) and it happens to support only SATA SSDs, so your suggestion of using a NVMe SSD would have been impossible. Even with the current generation of cheap Intel CPUs, i.e. Alder Lake N, for instance N100, the CPUs have very few PCIe lanes and most cheap computers do not have an M.2 socket that works at the full PCIe 3 speed of 32 Gb/s like the SSD of the tested OrangePi computer, but they have sockets with only 2 lanes or only 1 lane, which work at half speed or at quarter speed. Most computers with RK3588 have a full-speed M.2 type M SSD socket and this is one of their advantages over most other computers in this price range. Since the OnnxStream performance depends both on SSD and on CPU performance, there is no surprise that an Intel Skylake CPU using AVX2 instructions is so much faster than Cortex-A76 with much lower clock frequency that it wins the benchmark despite the slower SSD. The only benchmarks more informative than these would have included comparisons with a computer using the direct competitor of RK3588, i.e. Intel N100 (which is faster for CPU-limited tasks, but not necessarily for those involving I/O or video), but it appears that the author does not have such a newer computer.
- jerrysievert 3y agoi use an orange pi 5+ as a read replica for all of my home logs, open telemetry, system metrics (gathered via snmp), and environmental data. i have it configured at 16gb of ram, a 2tb nvme, connected to my network at 1gbit, and to my nas to run iscsi at 2.5gbit. it is a very nice little system, and has been rock steady running ubuntu 22.04. i plan on making it my primary database server, but that's a later project. it's been in service for 8 months now, and has been quite impressive. highly recommended for those into small compute home databases.
- rcarmo 3y agoIncidentally, one of the LXC containers I have running on mine has a copy of my IoT metrics database (2 years, roughly 10GB in SQLite, a bit over that when imported into Postgres). Queries are lightning fast compared to the Pi, even doing aggregations--which is why I mention at the bottom of the article that I intend to move my home automation stuff to it.
- applied_heat 3y agoWhat is your IOT metrics stack?
- rcarmo 3y agoHomegrown. Essentially Node-RED and some Go/Python.
- charcircuit 3y agoThere was no mention that this costs >$100 which does limit its usage compared to a $35 rpi.
- jsheard 3y agoAt current rates the best Pi you can get for $35 is a Pi4 with just 1GB of RAM, which isn't in the same league as the OP5+ 4GB.
- charcircuit 3y agoThe article mentions that author wanted to see if the orange pi 5 plus could be the successor to the Pi4.
- rcarmo 3y agoNot so much a successor but an alternate path. I didn't want to get into pricing because a Pi (regardless of number) doesn't ship with a PSU, an NVMe slot, or even two 2.5GbE interfaces. It's not something I want to directly compare with price-wise, really (although I suspect adding up all the bits might be comparable...)
- magicalhippo 3y agoOn the other hand, you can in fact do a lot with a RPi4 with 1GB of RAM. For me, once you reach $100 I'd much rather get one of the NUC-like boxes.
- moffkalast 3y agoThe Pi 5 + heatsink + customs fees also comes up to just over $100.
- tonymet 3y agoI'm a longtime Pi fan but I'm really dubious of the single board computer market . At $5-$10 (pi-zero / W) and $35 (pi3 / pi 4) these little boards made a ton of sense. Pushing into ~ $120-$150 to me doesn't make any sense. You can get 8gb/16gb N100 at that price point complete with a case. I saw his point on the per-watt performance which is valid, but are people running a room full of these things? Why spend so much to save 10 watts ? Someone please enlighten me on how this segment still remains viable.
- stefan_ 3y agoBut the per-watt performance of these chips is usually terrible, made as they are on older silicon processes. Any recent-ish laptop CPU will have no problem beating them.
- nic547 3y agoThe Intel N100 is a Alder-Lake N chip, based on Intel 7. It's four of the small cores you find on 12-Gen Laptop processors, made with the same process. It's successor arch, Meteor Lake has only recently launched, so the N100 shouldn't be too far off in terms of efficiency.
- beebeepka 3y agothese small intel cores aren't known for their power efficiency
- nic547 3y agoMostly because the high end intel processors are run at power limits where they're getting extremely diminishing returns, trying to eek out every last bit of performance. In lower power scenarios the little cores can be more efficient than the large ones. And at 6W TDP the N100 is a low power scenario. https://chipsandcheese.com/2022/01/28/alder-lakes-power-efficiency-a-complicated-picture/ https://chipsandcheese.com/2022/01/28/alder-lakes-power-effi...
- 404mm 3y agoPi 5 is the first Pi I did not buy. I was always the early adopter of all Pi’s but that ended with 4. For me, the killer was a combo of 3 things: 1. Too expensive relative to performance. 2. Availability of quite decent N100 (and similar) boxes with expandable memory and storage. 3. Not interested in Pi that should be actively cooled. I always wished the Pi had eMMC storage but it never happened for the non-compute module versions. I still have a few Pi’s around the house and they are plenty powerful for their purpose.
- hammyhavoc 3y agoChecked out RISC-V yet? That's becoming more interesting than ARM for me.
- justin66 3y agoWhat's the "I have less than a hundred dollars and I want a reasonably performing Linux desktop" RISC-V SBC option?
- UncleOxidant 3y agoI think this is a good question. I don't think you're going to find a RISC-V board with comparable performance to an rPI 4 or 5 in that price range currently.
- justin66 3y agoInteresting. Using the RPi 5 as a benchmark, what if I was willing to spend $200? I guess what I'm wondering is how much you have to spend to get to that point.
- bpye 3y agoI’m in a similar boat with the Pi 5 being the first I haven’t bought. I do have one of the VisionFive 2 boards, but I’m more interested in the Milk-V Oasis board that’s supposedly coming near the end of the year. Software support for RISC-V isn’t as good yet either. I don’t know of any hardware that supports the hypervisor extension, there is no IOMMU spec today, many distributions don’t give it the same support as ARM64, etc. It is definitely improving rapidly, but I don’t think there have been any really compelling RISC-V SoCs just yet.
- nosebear 3y agoThis thing would be great for a low-power NAS, but no mainline kernel support, no buy :/
- squarefoot 3y agoThese links might be of interest. https://forum.armbian.com/topic/33306-trying-to-use-opi5plus-as-nas-with-nvme-to-sata-adapter-for-storage/#comment-179431 https://forum.armbian.com/topic/33306-trying-to-use-opi5plus... https://www.armbian.com/orangepi-5/ https://www.armbian.com/orangepi-5/
- Havoc 3y agoI couldn’t get armbian to boot on mine
- rcarmo 3y agoYeah, well, I do mention near the end that I would like a couple of SATA ports for it. I suppose one could stick a controller into the NVMe or Wi-Fi slot...
- cedws 3y agoDamn, that's a shame. Why are these hardware companies too lazy to upstream?
- roughly 3y agoThe idle power draw at 5W is a little surprising to me, actually - the M1 mac mini draws ~7 idle. The max is a whole lot higher, but I'd also suspect the max performance you're getting is a whole lot higher, too. I really like the pi zeros for low power budget computing, but I think once you're getting into this kind of power envelope, you're kicking into "real computer" territory and I'm not sure how much benefit the SBCs are giving you.
- eropple 3y agoYou can bring it down to around 3W with a few config tweaks. Mine averages about that. (edit: wrong board! I was thinking of my OPi5, not the Plus. My OPi5+ looks like it's around the same as the author.) An M1 Mac is more powerful for sure, but for my use cases (my OPi5+ is being used as a video capture relay with its onboard HDMI capture and my two OPi5's are k8s nodes running Github Actions jobs) it would also be a lot more expensive.
- rcarmo 3y agoDid you change the CPU governor, or were there any more tweaks? I did measure this off the wall and there's always a bit more overhead in that situation, but I'm curious.
- eropple 3y agoNope, I just remembered the wrong one. 3W on the OPi5, not the Plus. Mea culpa.
- roughly 3y agoI'm curious how much more expensive, though - the low-end mini is $600. I don't know how much each of the OPis is once you've added all the accessories to make them functional, but it wouldn't surprise me if all of performance/$, performance/watt, and actual total cost winds up being better with the mini once you've got more than one or two of the OPis running.
- 3y ago
- squarefoot 3y agoBefore someone starts the usual yadda yadda about the RPi biger community, the OS not having long time support etc. I would repeat one more time: do not rely on board vendor supplied images; this is valid for pretty much all boards. Just go to Armbian or DietPi pages and you'll almost certainly find one or more images that work on your board and forums to discuss about them with very knowledgeable people. https://www.armbian.com/download/ https://www.armbian.com/download/ https://dietpi.com/#download https://dietpi.com/#download Those projects are well worth a contribution, as they don't have a giant like Broadcom behind them.
- rcarmo 3y agoYep. That is why I went with Armbian (even though I did test the OrangePi images while I waited for my NVMe). Can't wait for them to ship kernel 6.x support for this board.
- hrldcpr 3y agoMind elaborating on this? I've always used Raspbian and am interested in hearing about the downsides.
- bemusedthrow75 3y agoNot to mention that if you really want to tinker you can use pi-gen to customise builds from a desktop without all that much difficulty: https://github.com/RPi-Distro/pi-gen https://github.com/RPi-Distro/pi-gen Worth a play.
- squarefoot 3y agoI was referring to other boards, not the Raspberries which have a well supported OS, apologies for not being clear about that. Other board manufacturers often publish distros which have been cobbled together with old kernels and proprietary blobs, then they abandon them when the board is declared obsolete. This is not the case of the Raspberry Pi of course, but for other boards I would check first the above mentioned distros before installing anything by the vendors.
- qwertox 3y agoWhich reminds me to mention that the Radxa ZERO 3E [0], which was announced last November, is now for sale [1] (since last week). It's basically a Raspberry Pi Zero with the difference that it has a gigabit ethernet port instead of WiFi+Bluetooth. This is not an ad, I've ordered two because my OpenVPN server which runs on a Raspberry Pi B+ (1st gen, 9 MBit/s throughput on Bookworm) needs upgraded hardware. In that context, it's remarkable that Bookworm still runs on an device as old and weak as the 1st-gen Raspi. [0] https://radxa.com/products/zeros/zero3e/ https://radxa.com/products/zeros/zero3e/ [1] https://arace.tech/products/radxa-zero-3e https://arace.tech/products/radxa-zero-3e
- rcarmo 3y agoI got one for review in December. Had a few initial issues with it (this was way before release), hope to be able to test it fully soon.
- bicebird 3y agoCan't remember the exact name but I remember seeing a similar pi0 sized board with POE ethernet and an m.2 slot for wifi, and thinking it'd be perfect for a WAP. Unfortunately it was a industrial vendor so don't think you can buy it in low quantities and the price is probably way too high for what it is. I feel like there must be a market for something like that tough, a board with the bare essentials to make it cheap enough to have a few around the house / office and leave it up to the customer to find a wireless card that could be upgraded down the line.
- doublepg23 3y ago
- fisian 3y agoI was interested in this board because of the HDMI input. However, I couldn't find anyone reviewing/testing that (last searched a few weeks ago). I have another board (Khadas Vim4) with HDMI input. But the HDMI input only recently got support in their vendor provided Linux image and is finnicky. In the Armbian image I couldn't get it to work for more than a few frames of input video (tried with gstreamer and ffmpeg). Additionally, I couldn't find any information on HDMI input in Linux (seems like everyone uses USB capture cards that use uvc with v4l2).
- rcarmo 3y agoHi, author here. That is something I intend to test. I know it works under Android (which I've yet to test), and under Linux I can see the audio part of it using lshw: *-sound:3 description: rockchiphdmiin physical id: 6 logical name: card3 logical name: /dev/snd/controlC3 logical name: /dev/snd/pcmC3D0c I can't see anything interesting in the USB bus: $ lsusb -t /: Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M /: Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=ohci-platform/1p, 12M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=ohci-platform/1p, 12M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-platform/1p, 480M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-platform/1p, 480M (other than the ludicrous bandwidth available, that is) ...but I am using the Armbian 5.x image, so maybe I am missing some driver or ARM DTD.
- chazeon 3y agoCan I use a regular m.2 PC Wi-Fi card on these ARM SBCs? Suppose the SBC's CPU is mainline supported / armbian supported, and the PC Wi-Fi card works on x86 machines under Linux.
- ThatPlayer 3y agoGenerally yes. M.2 wifi cards are just PCI-E (except for Intel CNVi). Jeff Geerling tried a bunch of different PCI-E cards with the Raspberry Pi 4 Compute Module which does expose the PCI-E interface: https://pipci.jeffgeerling.com/ https://pipci.jeffgeerling.com/
- bemusedthrow75 3y agoPeople who think the Raspberry Pi 5 is underwhelming a) don't understand the market or its needs particularly well b) aren't really paying attention to the underlying trends from the foundation or the trading company c) are willing to write off any absurdly arcane, poorly-documented things they had to do to get a competing board to offer a stable, supported alternative to the Pi 5. But the most interesting thing about the Pi 5 is what it tells you about what is coming. Look at the (astonishingly) successful RP2040, and then look at the Pi 5's RP1 Southbridge, and then scratch your chin and think for a bit. It's not really incremental. Something quite big has happened here, we just don't see the product of it yet.
- hackernudes 3y agoCare to share your opinion more clearly? PCIE is eating the world? More modular computing systems? None of that seems "big" to me so I am probably missing your point.
- 542458 3y agoBoth the 2040 and the Pi5 south bridge are in-house custom silicon, and fairly good at that. I think the parent post is alluding to the Pi foundation eventually building their own processors, and in the process hopefully shrugging off many of the pi’s longstanding limitations.
- SeasonalEnnui 3y agoI agree with what the parent is alluding to; the introduction of the RP1 is very understated but perhaps it's more interesting to SBC engineers rather than the end users. In other words: 1. The RP1 (implemented on TSMC 40LP) contains all the power hungry/high bandwidth IO that is difficult to do on smaller process nodes. This allows the main processor to be moved to smaller nodes or even a different vendor/architecture in future boards. Easier to target better power efficiency in the future. 2. Going forwards, the IO feature set will now be consistent and reliable, by reusing RP1. It is no longer a requirement to try to get these peripherals on the main processor.
- TheChaplain 3y agoI always found it odd that many seem to belittle the RPi and/or say it's lacking in power or features. As I understood it since the beginning, the RPi is a teaching and learning tool, not your 32gb home server running git, nextcloud, plex, portainer and 15 other services. So faulting it for something it was never intended to be seems a bit unfair?
- NikkiA 3y agoThe RK3588 is great, but I really want a Dimensity 9300 SBC
- FL33TW00D 3y agoWow this chip does look good - when can we get our hands on it?
- NikkiA 3y agoISTR seeing that a couple of phones with it released in china just before christmas, so it's probably already available, here's one of them (albeit pretty meh apart from the CPU): https://www.gsmarena.com/vivo_x100_pro-12694.php https://www.gsmarena.com/vivo_x100_pro-12694.php
- Havoc 3y ago>there is going to be a 32GB version as well I've got two in my k3s cluster so very much a thing already
- cedws 3y agoAnyone know if the CPU is fast enough to saturate those 2.5GbE ports?
- Havoc 3y agoSaturate probably. Switching fw style over the two ports not so sure
- theodric 3y ago> in general, right now Rockchip images are mostly kernel 5.x based No, they're mystery meat-based, merely pretending to be 5.x, like most of those Chinese SoCs. I'll care when it can run mainline and not Armbian.
- apatheticonion 3y agoI just want something with sata ports and 10G ethernet so I can make a cheap, power efficient home NAS. Right now I'm using an old office PC that costs $7/m to run. Using the USB port on my router is too slow (maybe I need more expensive HDD enclosures) and can't RAID
- xyzzy123 3y agoODROID-H3/H3+ has 2 sata ports, emmc and an m.2 but you have to be willing to do a tiny bit of hacking to get 10gbe out of it. Supports a fair bit of ram (64GB) but no ECC which is a dealbreaker for some. The "hacks" with these are to use the m.2 port with an m.2 to PCIE adapter to get 10gbe or several more SATA ports. NAS's are tricky to spec because people have such a wide range of requirements. For example, a lot of people want their NAS to be able to do video transcoding which wouldn't work well on this hardware.
- apatheticonion 3y agoI just need a fast file host. The idea of video transcoding is pretty strange to me. What happened to the days where the client had the necessary codecs to play a video? Even if you store the video in the destination format, adding subtitles immediately requires Plex (and similar) to transcode, rendering the subtitles into the video. Personally, I have VLC installed on my Chromecast and, while it's not as pretty or convenient as Plex, the format of my files is irrelevant.
- xchip 3y agoWarning, suspiciously the article doesn't mention the USB ports are only 2.0. Was this a coincidence or was the article biased?