6 ms·
Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.
by matheusmoreira 3y ago
Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.
- nullwarp 3y agoThere still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been rock solid for me for years in both small and huge scales. we actually moved our production ceph setup to this. - longhorn was solid for me when i was in the k8s world - glusterfs is still fine as long as you know what you are going into.
- reactordev 3y agoI'd throw minio [1] in the list there as well for homelab k8s object storage. [1] https://min.io/ https://min.io/
- speedgoose 3y agoAlso garage. https://garagehq.deuxfleurs.fr/ https://garagehq.deuxfleurs.fr/
- BlackLotus89 3y agoGarage seems to only to duplication https://garagehq.deuxfleurs.fr/documentation/design/goals/ https://garagehq.deuxfleurs.fr/documentation/design/goals/ > Storage optimizations: erasure coding or any other coding technique both increase the difficulty of placing data and synchronizing; we limit ourselves to duplication. This is probably a nogo for most use cases where you work with large datasets....
- speedgoose 3y agoYes I mentioned it in the context of homelabs, it’s not an alternative to Ceph or MinIO for large installations. But the simplicity is appreciated. You can recover all the data using rsync.
- plagiarist 3y agoMinio doesn't make any sense to me in a homelab. Unless I'm reading it wrong it sounds like a giant pain to add more capacity while it is already in use. There's basically no situation where I'm more likely to add capacity over time than a homelab.
- reactordev 3y agoYou get a new nas (minio server pool) and you plug it into your home lab (site replication) and now it's part of the distributed minio storage layer (k8s are happy). How is that hard? It's the same basic thing for Ceph or any distributed JBOD mass storage engine. Minio has some funkiness with how you add more storage but it's totally capable of doing it while in use. Everything is atomic.
- dataangel 3y agoI really wish there was a benchmark comparing all of these + MinIO and S3. I'm in the market for a key value store, using S3 for now but eyeing moving to my own hardware in the future and having to do all the work to compare these is one of the major things making me procrastinate.
- woopwoop24 3y agominio is good but you really need fast disks. They also really don't like, when you want to change the size of your cluster setup. No plan to add cache disks, they just say use faster disks. I have it running, goes smoothly but not really user friendly to optimize
- deleted 3y ago[deleted]
- rglullis 3y agoMinio gives you "only" S3 object storage. I've setup a 3-node Minio cluster for object storage on Hetzner, each server having 4x10TB, for ~50€/month each. This means 80TB usable data for ~150€/month. It can be worth it if you are trying to avoid egress fees, but if I were building a data lake or anything where the data was used mostly for internal services, I'd just stick with S3.
- rglullis 3y ago> glusterfs is still fine as long as you know what you are going into. Does that include storage volumes for databases? I was using glusterFS as a way to scale my swarm cluster horizontally and I am reasonably sure that it corrupted one database to the point I lost more than a few hours of data. I was quite satisfied with the setup until I hit that. I know that I am considered crazy for sticking with Docker Swarm until now, but aside from this lingering issue with how to manage stateful services, I've honestly don't feel the need to move yet to k8s. My clusters is ~10 nodes running < 30 stacks and it's not like I have tens of people working with me on it.
- camkego 3y agoDocker Swarm seems to be underrated, from a simplicity and reliability perspective, IMHO.
- bityard 3y agoCeph is sort of a storage all-in-one: it provides object storage, block storage, and network file storage. May I ask, which of these are you using seaweedfs for? Is it as performant as Ceph claims to be?
- asadhaider 3y agoI thought it was popular for people running Proxmox clusters
- geerlingguy 3y agoIt is, and if you have a few nodes with at least 10 GbE networking, it's certainly the best clustered storage option I can think of.
- matheusmoreira 3y agoI just want to hoard data. I hate having to delete stuff to make space. Things disappear from the web every day. I should hold onto them. My requirements for a storage solution are: > Single root file system > Storage device failure tolerance > Gradual expansion capability The problem with every storage solution I've ever seen is the lack of gradual expandability. I'm not a corporation, I'm just a guy. I don't have the money to buy 200 hard disks all at once. I need to gradually expand capacity as needed. I was attracted to this ceph because it apparently allows you to throw a bunch of drives of any make and model at it and it just pools them all up without complaining. The complexity is nightmarish though. ZFS is nearly perfect but when it comes to expanding capacity it's just as bad as RAID. Expansion features seem to be just about to land for quite a few years now. I remember getting excited about it after seeing news here only for people to deflate my expectations. Btrfs has a flexible block allocator which is just what I need but... It's btrfs.
- chromatin 3y ago> ZFS is nearly perfect but when it comes to expanding capacity it's just as bad as RAID. if you don't mind the overhead of a "pool of mirrors" approach [1], then it is easy to expand storage by adding pairs of disks! This is how my home NAS is configured. [1] https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...
- roygbiv2 3y agoThis is also exactly how mine is done. Started off with a bunch of 2TB disks. I've now got a mixture of 16TB down to 4TB, all in the original pool.
- Snow_Falls 3y ago50% storage efficiency is a tough pill to swallow, but drives are pretty big and the ability to expand as you go means it can be cheaper in the long run to just buy the larger, new drives coming out than pay upfront for a bunch of drives in a raidz config.
- 3y ago
- sob727 3y agoCurious, what do you mean by "know what you go into" re glusterfs? I recently tried ceph in a homelab setup, gave up because of complexity, and settled on glusterfs. I'm not a pro though, so I'm not sure if there's any shortcomings that are clear to everybody but me, hence why your comment caught my attention.
- cholmon 3y agoGlusterFS support looks to be permanently ending later this year. https://access.redhat.com/support/policy/updates/rhs https://access.redhat.com/support/policy/updates/rhs Note that the Red Hat Gluster Storage product has a defined support lifecycle through to 31-Dec-24, after which the Red Hat Gluster Storage product will have reached its EOL. Specifically, RHGS 3.5 represents the final supported RHGS series of releases. For folks using GlusterFS currently, what's your plan after this year?
- loeg 3y agoWhy would you bother with a distributed filesystem when you don't have to?
- iwontberude 3y agoIt's cool to cluster everything for some people (myself included). I see it more like a design constraint than a pure benefit.
- imiric 3y agoFor the same reason you would use one in enterprise deployments: if setup properly, it's easier to scale. You don't need to invest in a huge storage server upfront, but could build it out as needed with cheap nodes. Assuming it works painlessly as a single node filesystem, of which I'm not yet convinced if the existing solutions do.
- loeg 3y ago> if setup properly, it's easier to scale For home use/needs, I think vertical scaling is much easier.
- imiric 3y agoNot really. Most consumer motherboards have a limited number of SATA ports, and server hardware is more expensive, noisy and requires a lot of space. Consumers usually go with branded NAS appliances, which are also expensive and limited at scaling. Setting up a cluster of small heterogeneous nodes is cheaper, more flexible, and can easily be scaled as needed, _assuming_ that the distributed storage software is easy to work with and trouble-free. This last part is what makes it difficult to setup and maintain, but if the software is stable, I would prefer this approach for home use.
- m463 3y agolol, wrong place to ask questions of such practicality. that said, I played with virtualization and I didn't need to. but then I retired a machine or two and it has been very helpful. And I used to just use physical disks and partitions. But with the VMs I started using volume manager. It became easier to grow and shrink storage. and... well, now a lot of this is second nature. I can spin up a new "machine" for a project and it doesn't affect anything else. I have better backups. I can move a virtual machine. yeah, there are extra layers of abstraction but hey.
- bluedino 3y agoRelated question, how does someone get into working with Ceph? Other than working somewhere that already uses it.
- SteveNuts 3y agoYou could start by installing Proxmox on old machines you have, it uses Ceph for its distributed storage, if you choose to use it.
- candiddevmike 3y agoLook into the Rook project
- hathawsh 3y agoThe recommended way to set up Ceph is cephadm, a single-file Python script that is a multi-tool for both creating and administering clusters. https://docs.ceph.com/en/latest/cephadm/ https://docs.ceph.com/en/latest/cephadm/ To learn about Ceph, I recommend you create at least 3 KVM virtual machines (using virt-manager) on a development box, network them together, and use cephadm to set up a cluster between the VMs. The RAM and storage requirements aren't huge (Ceph can run on Raspberry Pis, after all) and I find it a lot easier to figure things out when I have a desktop window for every node. I recently set up Ceph twice. Now that Ceph (specifically RBD) is providing the storage for virtual machines, I can live-migrate VMs between hosts and reboot hosts (with zero guest downtime) anytime I need. I'm impressed with how well it works.
- ianlevesque 3y agoI played around with it and it has a very cool web UI, object storage & file storage, but it was very hard to get decent performance and it was possible to get the metadata daemons stuck pretty easily with a small cluster. Ultimately when the fun wore off I just put zfs on a single box instead.
- reactordev 3y agoThere's a blog post they did where they setup Ceph on some rPI 4's. I'd say that's not significant hardware at all. [1] [1] https://ceph.io/en/news/blog/2022/install-ceph-in-a-raspberrypi-4-cluster/ https://ceph.io/en/news/blog/2022/install-ceph-in-a-raspberr...
- m463 3y agoI think "significant" turns out to mean the number of nodes required.
- m463 3y agoI think you need 3 or was it 5 machines? proxmox will use it - just click to install
- victorhooi 3y agoI have some experience with Ceph, both for work, and with homelab-y stuff. First, bear in mind that Ceph is a distributed storage system - so the idea is that you will have multiple nodes. For learning, you can definitely virtualise it all on a single box - but you'll have a better time with discrete physical machines. Also, Ceph does prefer physical access to disks (similar to ZFS). And you do need decent networking connectivity - I think that's the main thing people think of, when they think of high hardware requirements for Ceph. Ideally 10Gbe at the minimum - although more if you want higher performance - there can be a lot of network traffic, particularly with things like backfill. (25Gbps if you can find that gear cheap for homelab - 50Gbps is a technological dead-end. 100Gbps works well). But honestly, for a homelab, a cheap mini PC or NUC with 10Gbe will work fine, and you should get acceptable performance, and it'll be good for learning. You can install Ceph directly on bare-metal, or if you want to do the homelab k8s route, you can use Rook (https://rook.io/ https://rook.io/). Hope this helps, and good luck! Let me know if you have any other questions.
- eurekin 3y agoNUC with 10gbit eth - can you recommend any?
- justinclift 3y agoIf you want something cheap, you could go with Lenovo M720q's: https://www.servethehome.com/lenovo-thinkcentre-m720q-tinyminimicro-feature/ https://www.servethehome.com/lenovo-thinkcentre-m720q-tinymi... They have a PCIe slot and can take 8th/9th gen intel cpus (6 core, etc). That PCIe slot should let you throw in a decent network card (eg 10GbE, 25GbE, etc).
- chomp 3y agoI’ve ran Ceph in my home lab since Jewel (~8 years ago). Currently up to 70TB storage on a single node. Have been pretty successful vertically scaling, but will have to add a 2nd node here in a bit. Ceph isn’t the fastest, but it’s incredibly resilient and scalable. Haven’t needed any crazy hardware requirements, just ram and an i7.
- willglynn 3y agoThe hardware minimums are real, and the complexity floor is significant. Do not deploy Ceph unless you mean it. I started considering alternatives when my NAS crossed 100 TB of HDDs, and when a scary scrub prompted me to replace all the HDDs, I finally pulled the trigger. (ZFS resilvered everything fine, but replacing every disk sequentially gave me a lot of time to think.) Today I have far more HDD capacity and a few hundred terabytes of NVMe, and despite its challenges, I wouldn't dare run anything like it without Ceph.
- samcat116 3y agoCan I ask what you use all that storage for on your NAS?
- mcronce 3y agoI run Ceph in my lab. It's pretty heavy on CPU, but it works well as long as you're willing to spring for fast networking (at least 10Gb, ideally 40+) and at least a few nodes with 6+ disks each if you're using spinners. You can probably get away with far fewer disks per node if you're going all-SSD.
- aaronax 3y agoI just set up a three-node Proxmox+Ceph cluster a few weeks ago. Three Optiplex desktops 7040, 3060, and 7060 and 4x SSDs of 1TB and 2TB mix (was 5 until I noticed one of my scavenged SSDs was failed). Single 1gbps network on each so I am seeing 30-120MB/s disk performance depending on things. I think in a few months I will upgrade to 10gbps for about $400. I'm about 1/2 through the process of moving my 15 virtual machines over. It is a little slow but tolerable. Not having to decide on RAIDs or a NAS ahead of time is amazing. I can throw disks and nodes at it whenever.
- sgarland 3y agoYes. I first tried it with Rook, and that was a disaster, so I shifted to Longhorn. That has had its own share of problems, and is quite slow. Finally, I let Proxmox manage Ceph for me, and it’s been a dream. So far I haven’t migrated my K8s workloads to it, but I’ve used it for RDBMS storage (DBs in VMs), and it works flawlessly. I don’t have an incredibly great setup, either: 3x Dell R620s (Ivy Bridge-era Xeons), and 1GBe. Proxmox’s corosync has a dedicated switch, but that’s about it. The disks are nice to be fair - Samsung PM863 3.84 TB NVMe. They are absolutely bottlenecked by the LAN at the moment. I plan on upgrading to 10GBe as soon as I can convince myself to pay for an L3 10G switch.
- sixdonuts 3y agoJust get a 25G switch and MM fiber. 25G switches are cheaper, use less power and can work with 10 and 25G SFPs.
- sgarland 3y agoThe main blocker (other than needing to buy new NICs, since everything I have already came with quad 1/1/10/10) is I'm heavily invested into the Ubiquiti ecosystem, and since they killed off the USW-Leaf (and the even more brief UDC-Leaf), they don't have anything that fits the bill. I'm not entirely opposed to getting a Mikrotik or something and it just being the oddball out, but it's nice to have everything centrally managed. EDIT: They do have the PRO-Aggregation, but there are only 4x 25G ports. Technically it _would_ meet my needs for Ceph, and Ceph only.
- mmerlin 3y agoProxmox makes Ceph easy, even with just one single server if you are homelabbing... I had 4 NUCs running Proxmox+Ceph for a few years, and apart from slightly annoying slowness syncing after spinning the machines up from cold start, it all ran very smoothly.
- louwrentius 3y agoIf you want decent performance, you need a lot of OSDs especially if you use HDD. But a lot of consumer SDDs will suffer terrible performance degradation with writes depending on the circumstances and workloads.
- mikecoles 3y agoWorks great, depending on what you want to do. Running on SBCs or computers with cheap sata cards will greatly reduce the performance. It's been running well for years after I found out the issues regarding SMR drives and the SATA card bottlenecks. 45Drives has a homelab setup if you're looking for a canned solution.
- antongribok 3y agoI run Ceph on some Raspberry Pi 4s. It's super reliable, and with cephadm it's very easy[1] to install and maintain. My household is already 100% on Linux, so having a native network filesystem that I can just mount from any laptop is very handy. Works great over Tailscale too, so I don't even have to be at home. [1] I run a large install of Ceph at work, so "easy" might be a bit relative.
- dcplaya 3y agoWhat are your speeds? Do you rub ceph FS too? I'm trying to do similar.
- antongribok 3y agoIt's been a while since I've done some benchmarks, but it can definitely do 40MB/s sustained writes, which is very good given the single 1GbE links on each node, and 5TB SMR drives. Latency is hilariously terrible though. It's funny to open a text file over the network in vi, paste a long blob of text and watch it sync that line by line over the network. If by "rub" you mean scrub, then yes, although I increased the scrub intervals. There's no need to scrub everything every week.
- dcplaya 3y agoRub, scrub... Was really supposed to be run but it got the answer I was looking for! Right now, I'm doing about half of that but I am rcloning data between 2 ceph clusters. (Proxmox and a arm rook ceph cluster).