7 ms·
The Orange Pi 6 Plus
- Neywiny 6mo agoDisappointing on the NPU. I have found it's a point where industry wide improvement is necessary. People talk tokens/sec, model sizes, what formats are supported... But I rarely see an objective accuracy comparison. I repeatedly see that AI models are resilient to errors and reduced precision which is what allows the 1 bit quantization and whatnot. But at a certain point I guess it just breaks? And they need an objective "I gave these tokens, I got out those tokens". But I guess that would need an objective gold standard ground truth that's maybe hard to come by.
- cyanydeez 6mo agojust try to find some benchmark top_k, temp, etc parameters for llama.cpp. There's no consistent framing of any of these things. Temp should be effectively 0 so it's atleast deterministic in it's random probabilities.
- Neywiny 6mo agoRight. There are countless parameters and seeds and whatnots to tweak. But theoretically if all the inputs are the same the outputs should be within Epsilon of a known good. I wouldn't even mandate temperature or any other parameter be a specific value, just that it's the same. That way you can make sure even the pseudorandom processes are the same, so long as nothing pulls from a hardware rng or something like that. Which seems reasonable for them to do so idk maybe an "insecure rng" mode
- andai 6mo ago>Temp should be effectively 0 so it's atleast deterministic in it's random probabilities. Is this a thing? I read an article about how due to some implementation detail of GPUs, you don't actually get deterministic outputs even with temp 0. But I don't understand that, and haven't experimented with it myself.
- kingstnap 6mo agoBy default CUDA isn't deterministic because of thread scheduling. The main difference comes from rounding order of reduction difference. It does make a small difference. Unless you have an unstable floating point algorithm, but if you have an unstable floating point algorithm on a GPU at low precision you were doomed from the start.
- geerlingguy 6mo agoThe even more confounding factor is there are specific builds provided by every vendor of these Cix P1 systems: Radxa, Orange Pi, Minisforum, now MetaComputing... it is painful to try to sort it out, as someone who knows where to look. I couldn't imagine recommending any of these boards to people who aren't already SBC tinkerers.
- coredog64 6mo agoI was also onboard until he got to the NPU downsides. I don't care about use for an LLM, but I would like to see the ability to run smallish ONNX models generated from a classical ML workflow. Not only is a GPU overkill for the tasks I'm considering, but I'm also concerned that unattended GPUs out on the edge will be repurposed for something else (video games, crypto mining, or just straight up ganked)
- Havoc 6mo ago>But I rarely see an objective accuracy comparison. There are some perplexity comparison numbers for the previous gen - Orange pi 5 in link below. Bit of a mixed bag, but doesn't seem catastrophic across the board. Some models are showing minimal perplexity loss at Q8... https://github.com/invisiofficial/rk-llama.cpp/blob/rknpu2/ggml/src/ggml-rknpu2/README.md https://github.com/invisiofficial/rk-llama.cpp/blob/rknpu2/g...
- topspin 6mo agoThere are a couple outfits making M.2 AI accelerators. Recently I noticed this one: DeepX DX-M1M 25 TOPS (INT8) M.2 module from Radxa[1]: https://radxa.com/products/aicore/dx-m1m https://radxa.com/products/aicore/dx-m1m If you're in the business of selling unbundled edge accelerators, you're strongly incentivized to modularize your NPU software stack for arbitrary hosts, which increases the likelihood that it actually works, and for more than one particular kernel. If I had an embedded AI use case, this is something I'd look at hard.
- jerf 6mo agoSo, this is slightly off topic, but out of curiousity, what are NPUs good for right this very second? What software uses them? What would this NPU be able to run if it was in fact accessible? This is an honest, neutral question, and it's specifically about what can concretely be done with them right now. Their theoretical use is clear to me. I'm explicitly asking only about their practical use, in the present time. (One of the reasons I am asking is I am wondering if this is a classic case of the hardware running too far ahead of the actual needs and the result is hardware that badly mismatches the actual needs, e.g., an "NPU" that blazingly accelerates a 100 million parameter model because that was "large" when someone wrote the specs down, but is uselessly small in practice. Sometimes this sort of thing happens. However I'm still honestly interested just in what can be done with them right now.)
- BirAdam 6mo agoI love that OrangePi is making good hardware, but after my experience with the OrangePi 5 Max, I won’t be buying more hardware from them again. The device is largely useless due to a lack of software support. This also happened with the MangoPi MQ-Pro. I’ll just stick with RPi. I may not get as much hardware for the money, but the software support is fantastic.
- daymanstep 6mo agoYeah that's the problem with ARM devices. Better just buy a N100
- simlevesque 6mo agoThe N100 is way larger than a OrangePi 5 Max.
- geerlingguy 6mo agoAlso about half as efficient, if that matters, and 1.5-2x higher idle power consumption (again, if that matters). Sometimes easier to acquire, but usually the same price or more expensive.
- spockz 6mo agoI can run my N100 nuc at 4W wall socket power draw idle. If I keep turbo boost off, it also stays there under normal load up to 6W full power. Then it is also terribly slow. With turbo boost enabled power draw can go to 8-10W on full load. Not sure how this compares to the OrangePI in terms of performance per watt but it is already pretty far into the area of marginal gains for me at the cost of having to deal with ARM, custom housing, adapters to ensure the wall socket draw to be efficient etc. Having an efficient pico psu power a pi or orange pi is also not cheap.
- daymanstep 6mo agoWhich NUC do you have? A lot of the nameless brands on aliexpress draw 10 watts on idle.
- adrianwaj 6mo agoOne or two USB-C 3.2 Gen2 ports are all that's required - can then plug in a hub or dock. eg: https://us.ugreen.com/collections/usb-hub?sort_by=price-descending https://us.ugreen.com/collections/usb-hub?sort_by=price-desc... Can also plug in a power bank. https://us.ugreen.com/collections/power-bank?sort_by=price-descending https://us.ugreen.com/collections/power-bank?sort_by=price-d... The advantage is that if the machine breaks or is upgraded, the dock and pb can be retained. Would also distribute the price. The dock and pb can also be kept away to lower heat to avoid a fan in the housing, ideally. Better hardware should end up leading to better software - its main problem right now. This 10-in-1 dock even has an SSD enclosure for $80 https://us.ugreen.com/products/ugreen-10-in-1-usb-c-hub-ssd https://us.ugreen.com/products/ugreen-10-in-1-usb-c-hub-ssd (no affiliation) (no drivers required) I'd have another dock/power/screen combo for traveling and portable use.
- spwa4 6mo agoUnfortunately most of my attempts to power both embedded PC and robot motors from the same power bank result in unexpected reboots (even on what is effectively an RC car). And yes, I'm a CS Msc, not an electrical engineer, but ...
- adrianwaj 6mo agoYeah, I've got a new mini PC that reboots at random. It kind of "drops out." Works better with Ubuntu than Manjaro. Failed device. I was thinking of setting up a forum of some sort to discuss various devices that people may be tinkering with. The domain is tinkeriDOTng - just need to do something with it - could end up looking like Discogs with all the variations on gadgets and devices out there - sort of like music. And then there's the whole "builds" dimension. How much otherwise highly useful stuff are people sitting on that they can't get going (or going well) for one reason or another? Also, recycling I expect will end up important at some point, and knowing who has what (and where) could expedite this process.
- james-clef 6mo agoSomething in me wants to buy every SBC and/or microcontroller that is advertised to me.
- hypercube33 6mo agoYeah I have this problem (?) too. They are just so neat. I also really like tiny laptops and recreations of classic computers.
- junon 6mo agoClockwork Pi if you haven't seen it. Beautiful little constructions.
- james-clef 6mo agoThese are awesome.
- 3abiton 6mo agoEven though all can be replaced by a decent mini pc with beefy memory, with lots of VMs.
- nomel 6mo agoYeah, I ended up using an old mac mini for my Home Assistant needs. It draws a whopping 7W from the wall at idle (and it's near always idle), but the price of a new RPi is the same as 13k hours of electric usage for this. Using whatever compute you have sitting in a drawer usually makes the most sense (including an old phone).
- thesh4d0w 6mo agoSBC is good for usbip endpoints for those vms. I use them to push devices around my home network.
- zzzoom 6mo agoAt some point SBCs that require a custom linux image will become unacceptable, right? Right?
- jagged-chisel 6mo ago“Custom”? No. Proprietary and closed? One can hope.
- Gigachad 6mo agoI think even custom is unacceptable. It’s too much of a pain being limited in your distro choice because you are limited to specific builds. On x86 you can run anything.
- joshuaissac 6mo agoThere are some projects to port UEFI to boards like Orange Pi and Raspberry Pi. You can install a normal OS once you have flashed that. https://github.com/tianocore/edk2-platforms/tree/master/Platform/RaspberryPi/RPi4 https://github.com/tianocore/edk2-platforms/tree/master/Plat... https://github.com/edk2-porting/edk2-rk3588 https://github.com/edk2-porting/edk2-rk3588
- ajb 6mo agoThere also seems to be a plan to add uefi support to u-boot[1]. Many of these kinds of boards have u-boot implementations, so could then boot uefi kernel. However many of these ARM chips have their own sub-architecture in the Linux source tree, I'm not sure that it's possible today to build a single image with them all built in and choose the subarchitecture at runtime. Theoretically it could be done, of course, but who has the incentive to do that work? (I seem to remember Linus complaining about this situation to the Arm maintainer, maybe 10-20 years ago) [1] https://docs.u-boot.org/en/v2021.04/uefi/uefi.html https://docs.u-boot.org/en/v2021.04/uefi/uefi.html
- ekianjo 6mo agoThe orange pi 6 plus supports UEFI from the get go.
- jonpalmisc 6mo agoLooks like the SoC (CIX P1) has Cortex-A720/A520 cores which are Armv9.2, nice. I've still been on the hunt for a cheap Arm board with a Armv8.3+ or Arvm9.0+ SoC for OSDev stuff, but it's hard to find them in hobbyist price range (this board included, $700-900 USD from what I see). The NVIDIA Jetson Orin Nanos looked good but unfortunately SWD/JTAG is disabled unless you pay for the $2k model...
- timschumi 6mo agoAnd it doesn't seem like anything newer than ARMv9.2 is available either, no matter the price point.
- preisschild 6mo agoUnfortunately only available atm for extremely high prices. I'd like to pick some up to create a ceph cluster (with 1x 18tb hdd osd per node in an 8 node cluster with 4+2 erasure coding)
- ggdxwz 6mo agoThis seems to be an overkill for most of my workloads that require an SBC. I would choose Jetson for anything computationally intensive, as Orange Pi 6 Plus's NPU is not even utilized due to lack of software support. For other workloads, this one seems a bit too large in terms of formfactor and power consumption, and older RK3588 should still be sufficient
- preisschild 6mo agothis has much better IO than equivalent RK3588 compute modules / boards though. Useful if you want to build a distributed storage system.
- youngNed 6mo agoI'm a big fan of raspberry pi, I have many, in fact I have so many I have: ``` alias findpi='sudo nmap -sP 192.168.1.0/24 | awk '\''/^Nmap/{ip=$NF}/B8:27:EB|DC:A6:32|E4:5F:01|28:CD:C1/{print ip}'\''' ``` On every `.bashrc` i have. But I just don't get... everything, I don't get the org, I don't get the users on hn, I'm like skinner in the 'no the kids are wrong' meme. It's a lambda. It's a cheap, plug in, ssh, forget. And it's bloody wonderful. If you buy a 1 or 2 off ebay, ok maybe a 3. After that? Get a damn computer. Want more bandwidth on the rj45? Get a computer. Want faster usb? Get a computer. Want ssd? Get a computer Want a retro computing device? Get a computer. Want a computer experience? Etc etc etc, i don't need to labour this. Want something that will sit there, have ssh and run python scripts for years without a reboot? Spend 20 quid on ebay. People demanded faster horses. And the raspi org, for some, damn fool, reason, tried to give them. There are people bemoaning the fact that raspberry pi's aren't able to run LLM's. And will then, without irony, complain that the prices are too high. For the love of God, raspi org, stop listening to dickheads on the Internet. Stop paying youtubers to shill. Stop and focus. You won't win this game
- geerlingguy 6mo agoTFA is about an Orange Pi, with a 12-core Arm chip, a bit more than a Raspberry Pi.
- thegdsks 6mo agoThis resonates. I still have a Pi 3B running pihole and it's been up for years. No updates needed, just works. The newer boards trying to compete with mini PCs feels like a different product category entirely.
- Lerc 6mo agoYou have to go in with your eyes open wth SBCs. If you have a specific task for it and you can see that it either already supports it or all the required software is there and it just needs to be gathered, then they can be great gadgets. Often they can go their entire lifespan without some hardware feature being usable because of lack of software. The blunt truth is that someone has to make that software, and you can't expect someone to make it for you. They may make it for you, and that's great, but really if you want a feature supported, it either has to already be supported, or you have to make the support. It will be interesting to see if AI gets to the point that more people are capable of developing their own resources. It's a hard task and a lot of devices means the hackers are spread thin. It would be nice to see more people able to meaningfully contribute.
- madduci 6mo agoCan you please upload your work with the custom image on GitHub?
- abc123abc123 6mo agoExcellent! There is an OrangePi Zero 3W. That means my radxa zero tv-computer now has some competition. It is sad that rasspberry pi abandoned the small, zero computer. Will keep the OrangePi Zero 3W in mind next time I need to cobble together a new tv computer.
- dd_xplore 6mo agoI bought a NanoPi R6C in the past in the hope that it's going to be a nice mini pc to run all my containers with super low power usage or router. But the software was bad, really bad. I found https://github.com/Joshua-Riek/ubuntu-rockchip/ https://github.com/Joshua-Riek/ubuntu-rockchip/ , it was godsend but still had some shortcomings. after 2 years, it's bit stable but I just keep it around as a backup route to access my homelab incase the main machines go down.
- quadruple 6mo agoIt should be noted that the CIX P1(this board's SoC) has ongoing efforts to be upstreamed. Last I checked, the GPU drivers were still not available(due to them not supporting ACPI? I may be wrong on this) and power draw being weird and stuck at 10-15ish watts. It seems like this blog confirms nothing has changed on those 2 points. With that being said, CIX and their main board partner, Radxa, have been open with the UEFI. I am not an expert in low-level environments such as the kernel or the UEFI, but if these tidbits sound interesting I would encourage anyone who is to look further into the CIX P1. To my untrained eyes, CIX looks like a company that is working towards a desktop/laptop chip with real UEFI/ACPI support. I look forward to the day it is polished up a bit.
- freedomben 6mo ago> and you end up diving far more into boot chains, vendor GPU blobs and inference runtimes than you ever intended. Yep, I'll pass. I'm done dealing with that kind of crap that is spread like a nasty STD through the ARM world. I'm sticking with x64 unless/until ARM gets this crap together.
- eqvinox 6mo agoI can for the love of whatever is holy not understand why they put 5GbE on that thing. Massively simplified, 2.5G is 1G sped up while 5G is 10G slowed down. It makes no sense and the market agrees. The ladder of popularity goes: 1000base-T, <long break>, 10Gbase-T, 2.5Gbase-T, <long break>, 5Gbase-T. (Depends on context ofc, 2.5G is quite popular on APs for example.) And note a lot of 10Gbase-T hardware is not Nbase-T compatible, and there are chips that do only 1G, 2.5G and 10G - no 5G. I guess if your design doesn't work at 10GbT you try with 5? Ugh.
- eqvinox 6mo ago> lspci is a bit more revealing, especially because you get to see where the dual 5GbE setup and Wi-Fi controller are placed–each seems to get its own PCI bridge: That's how PCIe works. A PCIe port - both upstream and downstream - is a "PCI bridge". The link is one bus. A switch chip's "interior" is another bus. The next links are each their own bus again. One per port. There's no switch here, bus 0 ( / 30 / 60)is "in" the CPU, each port is it's own bus. The more interesting thing is the PCI domain, the first 4 digits: 0000:60:00.0 PCI bridge: CIX Technology Group Co., Ltd. CIX P1 CD8180 PCI Express Root Port 0000:61:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8126 5GbE Controller 0001:30:00.0 PCI bridge: CIX Technology Group Co., Ltd. CIX P1 CD8180 PCI Express Root Port 0001:31:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8126 5GbE Controller 0002:00:00.0 PCI bridge: CIX Technology Group Co., Ltd. CIX P1 CD8180 PCI Express Root Port 0002:01:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8852BE PCIe 802.11ax Wireless Network Controller This generally (caveat emptor) means the ports aren't handled in some common PCIe subsystem, rather each port is independently connected to the CPU crossbar. The ports may also not be able to access each other, or non-transparent mapping rules apply. Doesn't have to, though; it might be due to some technicality, driver bug, misunderstanding, whatever else.
- bitfilped 6mo agoComplaining about vendor blobs while showing a picture of a machine hooked up to a China produced RAT box is certainly a statement.