20 ms·
We have presence both at AWS and in a colo facility. I wouldn't say it's a slam dunk for either one. The Colo monthly bill is much lower than AWS, but when yo
by linsomniac 5y ago
We have presence both at AWS and in a colo facility.
I wouldn't say it's a slam dunk for either one. The Colo monthly bill is much lower than AWS, but when you add in our initial and yearly hardware expenses they are similar. They run different workloads, not smaller or larger, just different.
I'd generally say that maintenance at the Colo has been low, but we've had some instances where managing our own hardware has delayed rollouts (getting a vendor to replace some bad RAM on a brand new box), and where we put huge amounts of time into investigating hardware problems.
In particular we have a couple of machines that have been in service a year or two, that started experiencing data corruption and performance issues. We have really good hardware support, and they tried, but honestly I think the issue is that the SSDs they were sourcing were weren't great. I've spent probably 100+ hours on that issue alone.
There's also the cost of maintaining your own setup. The number of hours we've poured into Kafka, ElasticSearch, and Kubernetes that probably could have been reduced by just clicking a button at AWS is pretty high.
Also, it's very nice to just have resources. We are spending a lot of money on primary and backup copies of some data at S3, plus egress. It would be cheaper to serve it from the Colo, but then we need ~50TB of storage. Provisioning that amount of storage at the colo wouldn't be cheap. Disks are cheap, storage is not.
- edude03 5y ago> Disks are cheap, storage is not. I really like how you worded this, because I'm running into the same issue with my current fleet of servers. They all run kubernetes, have 100gb interconnects, are stuffed with NVMe drives and spinning rust (28tb per node, 2x2Tb nvme, and 4x6tb spinning drives) and, I've had the hardest time actually turning all those disks into a reliable, performant and easy to manage storage system.
- PaywallBuster 5y agoWas doing that just recently, dedicated server stuffed with spinning rust. ZFS to the rescue > zpool create tank raidz2 sda sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo > zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT tank 127T 1.30M 127T - - 0% 0% 1.00x ONLINE - > df -h | grep tank tank 102T 256K 102T 1% /tank PS: not an expert, just my 2 cents. do your homework
- edude03 5y agoOh yes, I’ma big zfs fan as well. To be clear though I need to be able to survive a node failure in addition to a disk failure
- washadjeffmad 5y agoAny specific challenges? I'm running something similar with a little more storage density.
- linsomniac 5y agoHave you tried Ceph? Seems purpose built for just that sort of setup (metadata on NVMe, data on spinning). I've been watching it for years, but only recently set up a test, and it worked well even through a power failure at home. My previous company had been using it after I left, and they seemed happy with it, but I haven't asked too much about it. I'd like to start doing some testing with it at work.
- edude03 5y agoI am currently running ceph actually! Out of the box, only having NVME in my storage pool, with 100GB interconnects, I'm getting about 50Mbit/s in Fio write tests. I'm not a storage expert, or a ceph expert clearly, but so far of the options I've tried (Portworx, Rancher) it's the worst performing, and I don't know why.