17 ms·
6 Raspberry Pis, 6 SSDs on a Mini ITX Motherboard
- AreYouSirius 4y ago
- erulabs 4y agoMan, Ceph really doesn't get enough love. For all the distributed systems hype out there - be it Kubernetes or blockchains or serverless - the ol' rock solid distributed storage systems sat in the background iterating like crazy. We had a huge Rook/Ceph installation in the early days of our startup before we killed off the product that used it (sadly). It did explode under some rare unusual cases, but I sometimes miss it! For folks who aren't aware, a rough TLDR is that Ceph is to ZFS/LVM what Kubernetes is to containers. This seems like a very cool board for a Ceph lab - although - extremely expensive - and I say that as someone who sells very expensive Raspberry Pi based computers!
- mcronce 4y agoCeph is fantastic. I use it as the storage layer in my homelab. I've done some things that I can only concisely describe as super fucked up to this Ceph cluster, and every single time I've come out the other side with zero data loss, not having to restore a backup.
- erulabs 4y agoHaha "super fucked up" is a much better way of describing the "usual, rare" situations I was putting it into as well :P
- dylan604 4y agoCare to provide examples of what these things were that you were doing to a storage pool? I guess I'm just not imaginative enough to think about ways of using a storage pool other than storing data in it.
- Lex-2008 4y agonot OP, but I would start with filling disk space up to 100%, or creating zillions of empty files. In case of distributed filesystems - maybe removing one node (under heavy load preferably), or "cloning" nodes so they had same UUIDs (preferably nodes storing some data on them - to see if the data will be de-duplicated somehow). Or just a disk with unreliable USB connection?
- erulabs 4y agoIn our case we were a free-to-use-without-any-signup way of testing Kubernetes. You could just go to the site and spin up pods. Looking back, it was a bit insane. Anyways, you can imagine we had all sorts of attacks and miners or other abusive software running. This on top of using ephemeral nodes for our free service meant hosts were always coming and going and ceph was always busy migrating data around. The wrong combo of nodes dying and bursting traffic and beta versions of Rook meant we ran into a huge number of edge cases. We did some optimization and re-design, but it turned out there just weren't enough folks interested in paying for multi-tenant Kubernetes. We did learn an absolute ton about multi-tenant K8s, so, if anyone is running into those challenges, feel free to hire us :P
- mcronce 4y agoAdministrating the storage pool. The worst that comes to mind for me was a node failure in the middle of a major version upgrade. Not likely a big deal for proper deployments, but I don't have enough nodes to tolerate complete node failure for most of my data. Grabbed a new root/boot SSD, reinstalled the OS, reinstalled the OSDs on each disk, told Ceph what OSD ID each one had previously (not actually sure if that was required), and....voila, they just rejoined the cluster and started serving their data like nothing ever happened.
- 3np 4y agoWe're more and more feeling we made the wrong call with gluster... The underlying bricks being a POSIX fs felt a lot safer at the time but in hindsight ceph or one of the newer ones would probably have been a better choice. So much inexplicable behavior. For your sake I hope the grass really is greener.
- rwmj 4y agoRed Hat (owner of Gluster) has announced EOL in 2024: https://access.redhat.com/support/policy/updates/rhs/ https://access.redhat.com/support/policy/updates/rhs/ Ceph is where the action is now.
- 3np 4y agoWhoa, totally missed that announcement, whenever it was... But yeah, have felt the wind blowing for a while now so with this I guess it's about time to get moving.
- nik736 4y agoCeph seems to be always related to big block storage outages. This is why I am very wary of using it. Has this changed? Edit: rephrased a bit
- antongribok 4y agoCeph is incredibly stable and resilient. I've run Ceph at two Fortune 50 companies since 2013 to now, and I've not lost a single production object. We've had outages, yes, but not because of Ceph, it was always something else causing cascading issues. Today I have a few dozen clusters with over 250 PB total storage, some on hardware with spinning rust that's over 5 years old, and I sleep very well at night. I've been doing storage for a long time, and no other system, open source or enterprise, has given me such a feeling of security in knowing my data is safe. Any time I read about a big Ceph outage, it's always a bunch of things that should have never been allowed in production, compounded by non-existent monitoring, and poor understanding of how Ceph works.
- shrubble 4y agoCan you talk about the method that Ceph has for determining whether there was bit rot in a system? My understanding is that you have to run a separate task/process that has Ceph go through its file structures and check it against some checksums. Is it a separate step for you, do you run it at night, etc.?
- lathiat 4y agoThat’s called ceph scrub & deep-scrub. By default it “scrubs” basic metadata daily and does a deep scrub where it fully reads the object and confirms the checksum is correct from all 3 replicas weekly for all of the data in the cluster. It’s automatic and enabled by default.
- shrubble 4y agoSo what amount of disk bandwidth/usage is involved? For instance, say that I have 30TB of disk space used and it is across 3 replicas , thus 3 systems. When I kick off the deep scrub operation, what amiunt of reads will happen on each system? Just the smaller amount of metadata or the actual full size of the files themselves?
- halbritt 4y agoI love it, but when it fails at scale, it can be hard to reason about. Or at least that was the case when I was using it a few years back. Still keen to try it again and see what's changed. I haven't run it since bluestore was released.
- teraflop 4y agoYeah, I've been running a small Ceph cluster at home, and my only real issue with it is the relative scarcity of good conceptual documentation. I personally learned about Ceph from a coworker and fellow distributed systems geek who's a big fan of the design. So I kind of absorbed a lot of the concepts before I ever actually started using it. There have been quite a few times where I look at a command or config parameter, and think, "oh, I know what that's probably doing under the hood"... but when I try to actually check that assumption, the documentation is missing, or sparse, or outdated, or I have to "read between the lines" of a bunch of different pages to understand what's really happening.
- geerlingguy 4y agoI think many people (myself included) had been burned by major disasters on earlier clustered storage solutions (like early Gluster installations). Ceph seems to have been under the radar for a bit of time when it got to a more stable/usable point, and came more in the limelight once people started deploying Kubernetes (and Rook, and more integrated/wholistic clustered storage solutions). So I think a big part of Ceph's success (at least IMO) was its timing, and it's adoption into a more cloud-first ecosystem. That narrowed the use cases down from what the earliest networked storage software were trying to solve.
- imiric 4y agoCan someone with experience with Ceph and MinIO or SeaweedFS comment on how they compare? I currently run a single-node SnapRAID setup, but would like to expand to a distributed one, and would ideally prefer something simple (which is why I chose SnapRAID over ZFS). Ceph feels to enterprisey and complex for my needs, but at the same time, I wouldn't want to entrust my data to a simpler project that can have major issues I only discover years down the road. SeaweedFS has an interesting comparison[1], but I'm not sure how biased it is. [1]: https://github.com/seaweedfs/seaweedfs#compared-to-ceph https://github.com/seaweedfs/seaweedfs#compared-to-ceph
- BlackLotus89 4y agoSeaweedfs has problems with large "pools" it's based on an old facebook paper (haystack) and supposed for block storage to distribute large image caches. I found it mediocre at best as it's documentation was lacking, performance was lacking (in my tests) and the multitude of components were hard to get working. The idea behind it is that every daemon uses one large file as data store to skip slow metadata access. There are different ways to access the storage over gateways. MinIO is changing so much in the last years thatI can't give a competent answer but compared to seaweedfs it uses many small local databases. Right now it's deprecating many features like the gateway and it is split into 2 main components (cli and server) compared the seaweedfs deployment is dead simple, but I don't know which direction the project is going. Went from a normal open source project to a more business like deal (in what I saw) like I said I didn't quite follow the process. Ceph is based on blocmlk storage. Offers an object gateway (s3/swift), fs (cephfs) and block storage (rbd). You can access everything through librados directly as well. For a minimal setup you need a "larger" vluster but it is the most flexible solution (imho). Uses the most resources as well, but you can do nearly everything you want without limit with it.
- imiric 4y agoThanks for your perspective. Ceph does sound the most appealing for my use case. I'm hoping that the learning curve is mild, and that it has a mostly set-and-forget UX.
- 4y ago
- alexk307 4y agoCool but I still can’t find a single raspberry pi compute module despite having been in the market for 2 years…
- simcop2387 4y agohttps://rpilocator.com/ https://rpilocator.com/ is probably the best place to keep an eye out for them. This is unfortunately also the case for non-CM rpis. Been wanting to get some more pi4s to replace some rather old pi3 (non+) that i've got running just because i want the uefi boot on everything since it makes managing things that much easier.
- mongol 4y agoYes when will this dry spell end?
- geerlingguy 4y agoSo far it seems like "maybe 2023"... this year supplies have been slightly better, but not amazing. Usually a couple CM4 models pop up over the course of a week on rpilocator.com.
- jonhohle 4y agoThis looks incredible. Is it possible to expose a full PCI interface from an NVMe slot? I have an old SAS controller that I want to keep running. If I could do that from a PI, that would be incredible.
- mcronce 4y agoYou can get M.2 to PCI-E add-in-card adapters, yeah - as long as it's an M.2 slot that supports NVMe, not SATA only
- crest 4y agoI don't see how they could have hooked a 2.5Gb/s Ethernet NIC to the CM4 modules without using up the single PCI-e 2.0 lane other than adding a power hungry, expensive and often out of stock PCI-e switching chip.
- wtallis 4y agoThere's no such thing as an NVMe slot, just M.2 slots that have PCIe lanes. NVMe is a protocol that runs on top of PCIe, and is something that host systems support at the software level, not in hardware. (Firmware support for NVMe is necessary to boot off NVMe SSDs, but the Pi doesn't have that and must boot off eMMC or SD cards.)
- geerlingguy 4y agoIf you want to use SAS with the Pi, I've only gotten newer generation Broadcom/LSI cards working so far—see my notes for the storage controllers here: https://pipci.jeffgeerling.com/#sata-cards-and-storage https://pipci.jeffgeerling.com/#sata-cards-and-storage
- jonhohle 4y agoIncredible resource, thanks! I’m currently using an older MegaRAID card, but could upgrade if I can find a reasonable configuration to migrate.
- formerly_proven 4y ago
- rabuse 4y agoIf only Raspberry Pi's weren't damn near $200 now...
- mmastrac 4y agoIs that just on the secondary market(s)? I'm still seeing them available <$100 in various models, but not always in-stock.
- lathiat 4y agoTry https://rpilocator.com/ https://rpilocator.com/ - no promises though.
- greggyb 4y agoUnless you are constrained in space to a single ITX case as in this example, you can get whole x86 machines for <$100 with RAM and storage included. There is a lot of choice in the <$150 range. You could get eight of these and a cheap 10-port switch for any kind of clustering lab you want to set up. Here is an example: https://www.aliexpress.com/item/3256804328705784.html?spm=a2g0o.productlist.0.0.4df06082Dv2Ht3&algo_pvid=ae0b3514-36c3-4c7a-8b4c-1b32825d286f&algo_exp_id=ae0b3514-36c3-4c7a-8b4c-1b32825d286f-7&pdp_ext_f=%7B%22sku_id%22%3A%2212000029435526857%22%7D&pdp_npi=2%40dis%21USD%21124.28%2187.0%21%21%21%21%21%402101e9cf16607501783027265e0df9%2112000029435526857%21sea&curPageLogUid=YZExMcNI3qO0 https://www.aliexpress.com/item/3256804328705784.html?spm=a2...
- Asdrubalini 4y agoWhat is the power consumption tho? It easily adds up over time.
- greggyb 4y agoThe linked machine uses a 2W processor. The successor product on the company's site uses a 12 volt, 2 amp power adapter: https://www.bee-link.net/products/t4-pro https://www.bee-link.net/products/t4-pro Here is a YouTube review of the linked model with input power listed at 12 volt, 1.5 amp (link to timestamp of bottom of unit): https://youtu.be/56UA2Uto1ns?t=129 https://youtu.be/56UA2Uto1ns?t=129
- rustdeveloper 4y agoThis looks really cool! There was a tutorial posted on HN about building mobile proxy pool with RPI that had obvious limitations: https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scraping https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scrap... It seems this could be a solution to scale capabilities of a single RPI.
- 3np 4y agoThis could be neat for a 3xnomad + 3xvault cluster. Just add runners and an LB.
- tenebrisalietum 4y ago> Many people will say "just buy one PC and run VMs on it!", but to that, I say "phooey." I mean with VM-leaking things like Spectre (not sure how much similar things affect ARM tbh) having physical barriers between your CPUs can be seen as a positive thing.
- mrweasel 4y agoSure, it's just that the Raspberry Pi isn't really fast enough for most production workloads. Having a cluster of them doesn't really help, you'd still be better off with a single PC. As a learning tool, having the ability to build a real hardware cluster, in a MiniITX case is awesome. I do sort of wonder what the business case for these boards are, I mean are there actually enough people who want to do something like this... schools maybe? I still think it's beyond weird that that there are so much hardware available for build Pi clusters, but I can't get an ARM desktop motherboard, with a PCI slot capable of actually being used as a desktop, for a reasonable prices.
- geerlingguy 4y agoI think a lot of these types of boards are built with the business case of either "edge/IoT" (which still for some reason causes people to toss money at them since they're hot buzzwords... just need 5G too for the trifecta), or for deploying many ARM cores/discrete ARM64 computers in a space/energy-efficient manner. Some places need little ARM build farms, and that's where I've seen the most non-hobbyist interest in the CM4 blade, Turing Pi 2, and this board.
- schaefer 4y agoThe Nvidia Jetson AGX Orin Dev Kit is getting pretty damn close to a performant Linux/ARM desktop system. GB5 Scores (Single: 763/ Multi: 7193) That's roughly 80% the performance of my current x86 desktop. Ubuntu 20.04 LTS 12 Core Cortex A78AE v8.2 64 bit. 2.20 Ghz 32 GB LDPDDR5 memory, 256 bit, 204.8 GB/s NVIDIA graphics/AI acceleration. PCI-E Slot 64 Mb eMMC 5.1 M.2 PCIe Gen 4 Display Port 1.4a +MST The deal breaker, if there is one is price: $2k
- 4y ago
- onlyrealcuzzo 4y ago> I was able to get 70-80 MB/sec write speeds on the cluster, and 110 MB/sec read speeds. Not too bad, considering the entire thing's running over a 1 Gbps network. You can't really increase throughput due to the Pi's IO limitations—maybe in the next generation of Pi, we can get faster networking! This isn't clear to me. What am I missing?
- gorkish 4y agoThe device has an onboard 8 port unmanaged gigabit switch. The two external ports are just switch ports and cannot be aggregated in any way. The entire cluster is therefore limited effectively to 1gbps throughput. IMO it ruins the product utterly and completely. They should have integrated a switch IC similar to what's used in the netgear gs110mx which has 8 gigabit and 2 multi-gig interfaces.
- geerlingguy 4y agoIt would be really cool if they could split out 2.5G networking to all the Pis, but with the current generation of Pi it only has one PCIe lane, so you'd have to add in a PCIe switch for each Pi if you still wanted those M.2 NVMe slots... that adds a lot of cost and complexity. Failing that, a 2.5G external port would be the simplest way to make this thing more viable as a storage cluster board, but that would drive up the switch chip cost a bit (cutting into margins). So the last thing would be allowing management of the chip (I believe this Realtek chip does allow it, but it's not exposed anywhere), so you could do link aggregation... but that's not possible here either. So yeah, the 1 Gbps is a bummer. Still fun for experimentation, and very niche production use cases, but less useful generally.
- lathiat 4y ago110MB/s is gigabit. It’s limited to gigabit networking and only has 1Gbps out from the cluster board. So there’s no way to do an aggregate speed of more than 1Gbps/110MBs on this particular cluster board. If each pis Ethernet was broken out individually and you used a 10G uplink switch or multiple 1G client ports then you could do better. The write speed being lower than the read speed will be because writes have to be replicated to two other nodes in the ceph cluster (everything has 3 replicas) which are also sharing bandwidth on those same 1G links. Reads don’t need to replicate so can consume the full bandwidth. So basically it’s all network limited for this use case. Needs a 2.5G uplink, LACP link aggregation or individual Ethernet ports to do better.
- gorkish 4y agoWould buy this in an instant if it weren't hobbled as hell by the onboard realtek switch. If it had an upstream 2.5/5/10g port it would be instantly 6 times more capable.
- DrudgeCorporate 4y agoWould 6 Pis be able to handle more than 1g? It says that they got around 70MB write and 100MB read. 2.5/5/10 seems like it would be a waste unless I'm overlooking something
- Havoc 4y agoPi raw reads off ssd are higher so yes there would be a benefit. Reckon it could saturate 2.5.
- gorkish 4y agoThe internal AXI bus on the Pi SoC can only move data at ~4gbps by DMA, so no the Pi 4 simply cannot cannot round trip real data from disk to network any faster than about 2gbps. It can saturate a 2.5gbps nic with a synthetic datastream, but for a storage application the 1gbps interface is as much as a single pi is going to realistically be capable of
- gorkish 4y agoThe AXI bus internal to the Pi's SoC is only capable of about 4gbps, and it carries DMA, so ~2gbps is more or less the hard limit for any kind of combined IO operation like disk<=>network no matter what kind of hardware you use for disk and network. So yes, each pi can easily saturate its own 1gbps interface, so a system like ceph that parallelizes reads and writes among nodes is severely crippled by the onboard switch choking off bandwidth to external clients. For the same reason, you can't easily scale this platform beyond a single board, which puts your clustered system back into a single point of failure. Strange, but terrible design decision.
- marcodiego 4y agoIt is a shame we have nothing as simple as the old OpenMOSIX.
- alberth 4y agoImagine instead of 6 Pi's this was 6 M2 arm chips on a mini-ITX board.
- dis-sys 4y agoThat would cost 6 arms and 6 legs.
- alberth 4y agoAs a thought experiment, so the: - 8GB Pi cost $75 (Geekbench score of 318 single / 808 multi-core) - 8GB M1 Mac mini cost $699 (Geek bench score of 1752 single / 7718 multi-core) This isn't too far off linearly price scaling. The M1 Mac mini costs 9.3x more, but get 5.5x faster single-core & 9.5x faster multi-core. https://browser.geekbench.com/v5/cpu/15902536 https://browser.geekbench.com/v5/cpu/15902536 https://browser.geekbench.com/v5/cpu/15912650 https://browser.geekbench.com/v5/cpu/15912650
- rbanffy 4y ago> The M1 Mac mini costs 9.3x more, but get 5.5x faster single-core & 9.5x faster multi-core. It's not always that we get a Mac that has a better price/performance than any other computer ;-) Their M-series is quite remarkable.
- rbanffy 4y agoI find these boards - that get little boards and network them into a cluster - very interesting. I'd like to see more of these in the future. I hope someone makes a Framework-compatible motherboard with these at some point. The Intel Edison module could have been a viable building block for one (and it happened a long time ago in computing terms - 2014) - it was self-contained, with RAM and storage on the module - but it lacked ethernet to connect multiple ones on a board - and I don't remember it having a fast external bus to build a network on. And it was quickly discontinued.
- kllrnohj 4y agoYou can get things like Epyc mini-ITX boards ( https://www.anandtech.com/show/16277/asrock-rack-deep-mini-itx-romed4id-2t https://www.anandtech.com/show/16277/asrock-rack-deep-mini-i... ) if you're just looking to ramp up the compute in a tiny space. Divide it up into 6 VMs if you still want it to be a cluster even.
- aftbit 4y agoImagine being able to buy 6 Raspberry Pis! I have so many projects I'd like to do, both personal and semi-commercial, but it's been literal years since I've seen a Raspberry Pi 4 available in stock somewhere in the USA, let alone 6.
- tssva 4y agoMicro Center often has them in stock. My local Micro Center currently has 17 RPi 4 4GB in stock. They are available only in store, but you can check stock at their website. Find one you know someone close to that is willing to purchase for you and ship.
- aftbit 4y agoI have a Micro Center within 15 minutes. I've checked them a handful of times with no luck. When they do have them, it's usually only the Pi 400s. They've also really jacked up the prices at the one near me, around $80 for a Pi 4 4GB and $120 for the 8GB version.
- hackeraccount 4y agoCrap. When did that happen? I swear I bought like two or three not that long ago and they were like 40 or 50 apiece.
- greggyb 4y agoIf you are not constrained to the Pi's form factor, there are lots of options for more powerful x86 systems for <$150 per system, often with RAM and storage included. See the linked thread for more details. If you are willing to go used, you can get good rates on lots of multiple thin clients (sample in linked thread). https://news.ycombinator.com/item?id=32497672 https://news.ycombinator.com/item?id=32497672
- antongribok 4y agoAside from running Ceph as my day job, I have a 9-node Ceph cluster on Rasberry Pi 4s at home that I've been running for a year now, and I'm slowly starting to move things away from ZFS to this cluster as my main storage. My setup is individual nodes, with 2.5" external HDDs (mostly SMR), so I actually get sligtly better performance than this cluster, and I'm using 4+2 erasure coding for the main data pool for CephFS. CephFS has so far been incredibly stable and all my Linux laptops reconnect to it after sleep with no issues (in this regard it's better than NFS). I like this setup a lot better now than ZFS, and I'm slowly starting to migrate away from ZFS, and now I'm even thinking of setting up a second Ceph cluster. The best thing with Ceph is that I can do a maintenance on a node at any time and storage availability is never affected, with ZFS I've always dreaded any kind of upgrade, and any reboot requires an outage. Plus with Ceph I can add just one disk at a time to the cluster and disks don't have to be the same size. Also, I can move the physical nodes individually to a different part of my home, change switches and network cabling without an outage now. It's a nice feeling.
- bityard 4y agoIs 9 the minimum number of nodes you need for a reasonable ceph setup or is that just what you arrived at for your use case?
- geerlingguy 4y agoI've seen setups with as few as 2 nodes with osds and a monitor (so 3 in total), but I believe 4-5 nodes is the minimum recommendation.
- antongribok 4y agoI would say the minimum is whatever your biggest replication or erasure coding config is, plus 1. So, with just replicated setups, that's 4 nodes, and with EC 4+2, that's 7 nodes. With EC 8+3, which is pretty common for object storage workloads, that's 12 nodes. Note, a "node" or a failure domain, can be configured as a disk, an actual node (default), a TOR switch, a rack, a row, or even a datacenter. Ceph will spread the replicas across those failure domains for you. At work, our bigger clusters can withstand a rack going down. Also, the more nodes you have, the less of an impact it is on the cluster when a node goes down, and the faster the recovery. I started with 3 RPis then quickly expanded to 6, and the only reason I have 9 nodes now is because that's all I could find.
- oblak 4y agoCredit where it's due - this is some 18 watt awesomeness at idle. Is it more "practical" than doing a Mini-ITX (or smaller, like one of those super small tomtom with up to 5900HX) build and equipping it with one one or more NVME expansion cards? Probably not. But it's cool. Now, if there were a new Pi to buy. Isn't it time for the 5? It's been 3 years for most of which they've been hard to fine. Mine broke and I really miss it because having a full blown desktop doing little things makes no sense, especially during the summer.
- formerly_proven 4y ago18 W idle is kinda horrible if you just want a small server (granted, this isn't one server, but instead six set-top boxes in one). That's recycled entry-level rack server range, which come with ILOM/BMC. Most old-ish fat clients can do <10 W, some <5 W, no problem. If you want a desktop that consumes little power when idle or not loaded a lot, just get basically any Intel system with an IGP since 4th gen (Haswell). Avoid Ryzen CPU with dGPU if that's your goal; those are gas guzzlers.
- oblak 4y ago1. I would bet at least half of all that wattage is the SSDs. 2. Buddy, you're spewing BS at someone who used to run a Haswell in a really small Mini-ITX case. It was a fine HTPC back in 2014. But now everything, bar my dead Pi, is some kind of Ryzen. All desktops and laptops. The various 4800u/5800u/6800u and lower parts offer tremendous performance at 15W nominal power levels. The 5800H I am writing this message on is hardly a guzzler, especially when compared to Intel's core11/12 parts. This random drive-by intel shilling really took me by surprise.
- formerly_proven 4y ago> 1. I would bet at least half of all that wattage is the SSDs. SSDs are really good at consuming nearly nothing when not servicing I/O. > 2. Buddy, you're spewing BS ... various 4800u/5800u/6800u ... 5800H ... None of those SKUs are a "Ryzen CPU with dGPU".
- sitkack 4y agoAnyone else having problems with the shipping page? Says it cannot ship to my address, formatted incorrectly ...
- naranha 4y agoDo you think Raspberries without ECC RAM are fine to use for a Ceph storage cluster? I did some research yesterday on the same topic, many say ECC Ram is essential for Ceph (and ZFS too). But I'm not sure what to believe, sure data could get corrupted in RAM before being written to the cluster, but how likely is that?
- dpedu 4y agoECC is not necessary for ZFS - this is a commonly repeated myth. https://news.ycombinator.com/item?id=14447297 https://news.ycombinator.com/item?id=14447297
- dijit 4y agoraspberry pi's have ECC memory; it's just on-die ECC. (this was a surprise to me too)
- EvanAnderson 4y agoIt sounds like the ECC counters are completely hidden from the SoC, though: https://forums.raspberrypi.com/viewtopic.php?t=315415 https://forums.raspberrypi.com/viewtopic.php?t=315415
- naranha 4y agoSounds like what DDR5 is doing too, errors are corrected automatically, but not necessarily communicated (?)/
- amarshall 4y agoDDR5 on-die ECC is not the same as traditional ECC. To that point, there are DDR5 modules with full ECC. On-die DDR5 ECC is there because it needs to be for the modules to really work at all.
- kordlessagain 4y agopastel-mature-herring~> Is this where compute is going? awesome-zebra*> There is no definitive answer to this question, as the direction of compute technology is always changing and evolving. However, the trend in recent years has been towards smaller, more powerful devices that are able to pack more processing power into a smaller form factor. The DeskPi Super6c is an example of this trend, as it offers a trim form factor and six individual Raspberry Pi Compute Module 4s, each of which offers a high level of processing power.
- sschueller 4y agoIf someone is trying to find a pi you can try the telegram bot I made for rpilocator.com. It will notify you as soon as there is stock with filters for specific pis and your location/preferd vendor. The bot is here: https://t.me/rpilocating_bot https://t.me/rpilocating_bot source: https://github.com/sschueller/rpilocatorbot https://github.com/sschueller/rpilocatorbot
- cosmiccatnap 4y agoI love fun projects like this. I would love to know if I could make one a router since the nic has two ports. I have a poweredge now which works fine but it's nowhere close to 20 watts and I barely use a quarter of it's cpu and memory
- technofiend 4y agoI will once again lament the fact that WD Labs built SBCs that sat with their hard drives to make them individual CEPH nodes but never took the hardware to production. It seems to me there's still a market for SBCs that could serve a CEPH OSD on a per-device basis, although with ever increasing density in the storage and hyperconverged space that's probably more of a small business or prosumer solution.
- preisschild 4y agoYeah, those were really cool. I saw some homelab setups using the ODroid HC2 from Hardkernel in a similar way. The 2 issues with this setup were that the HC2 was using a low-performance armv7 processor, with armv7 being a very unsupported platform by most software and the fact that you can't use a flash-based disk for a ceph bluestore metadata device since it only had one SATA port.
- hinkley 4y agoI think there's something to be said for sizing a raspberry pi or a clone to fit into a hard drive slot. I also think the TuringPi people screwed up with the model 2. Model 2 of a product should not have fewer slots than the predecessor, and in the case of the Turing Pi, orchestrating 4 devices is not particularly compelling. It's not that difficult to wire 4 Pi's together by hand. I had 6 clones wired together using risers and powered by my old Anker charging station and an 8 port switch, with a few magnets to hold the whole thing together.
- cptnapalm 4y agoOh my God, I want this. I have no use for it, whatsoever, but oh my God I want it anyway.
- aloer 4y agoConsidering that this is custom made for the CM4 form factor, the Turing Pi with carrier boards looks much more attractive because future proof. If only it were already available. It also has SATA and USB 3.0 which is nice Until I can preorder one I will slowly stock up on CM4s and hope I’ll get there before pi5 comes out. Are there other boards like this?
- mhd 4y agoCan I run a Beowulf cluster on this?
- mgarfias 4y agonow if only we could get compute modules
- pishpash 4y ago18W at idle seems like a lot of power.
- justsomehnguy 4y agoDivide it for six PIs, six NVMEs, one switch.
- pishpash 4y agoWhy divide? You don't divide by how many cores are on a regular PC, to which this has comparable power.
- justsomehnguy 4y agoEach Pi, NVMe, switch has it's own PSU/VRM units. That's about efficency of power conversion and power management. It would be interesting if the authour measured the same board with 1 Pi and 1 NVMe, and 1 Pi and 6 NVMe.
- marshray 4y agoQuite the bold design choice to put the removable storage on the underside of the motherboard.
- antod 4y agoIs it bad that my first thought was "Imagine a Beowulf cluster of these..."
- speed_spread 4y ago
- rasz 4y ago>6 Raspberry Pis 6x >$100 >, 6 SSDs 6x NVME SSDs >I was able to get 70-80 MB/sec write speeds on the cluster, and 110 MB/sec read speeds. ...
- deleted 4y ago[deleted]