4 ms·

Have you looked at Ceph ad a storage backend? It allows per-file erasure coding configuration and has a lot of cool features. It can also manage raw block devic

Have you looked at Ceph ad a storage backend? It allows per-file erasure coding configuration and has a lot of cool features. It can also manage raw block devices from what I'm told. It's really awesome you've gotten a chance to build up this cluster. I wish more startups realized that, despite the initial price tag, some of this is way better than what you'll get from AWS. Your NVMe is likely 10x to 100x faster than premium ssd storage at a fraction of the price. If you ever use GPUs you'll be saving loads of money. It's hard to convince people though that bare metal isn't scary.

FilingTrader 6y ago
I haven't tried Ceph. It didn't seem as appealing, judging from comments I found (mostly on Hacker News): http://www.regressionist.com/2020/06/20/reviews-of-distributed-filesystems/ http://www.regressionist.com/2020/06/20/reviews-of-distribut... But it's also hard to tell exactly what people are reviewing. Evidently CephFS is less mature than Ceph object store. But, yeah. This just all seems so much cheaper than AWS. There are other advantages, too. Like, a lot of stuff is better with the low latency that comes with having the cluster on-location. Python plots are more pleasant to interact with (ssh -X). NoMachine remote desktop is faster, etc. Also, it's a bit like an all-you-can-eat buffet. You pay once, and then you can just relax and enjoy. I have enough storage now that I don't need to stress about costs before deciding to cache intermediate data. And I don't need to worry about whether I really need those older results that I still have saved on disk that I'm paying each minute to store on AWS. Those kind of thoughts are just gone. I do have one GPU now, a 1080 Ti ($450 on eBay). I am ready to house a total of 6, but I'm waiting to buy more until I really need them.

4 ms·

Have you looked at Ceph ad a storage backend? It allows per-file erasure coding configuration and has a lot of cool features. It can also manage raw block devic

by gravypod 6y ago

FilingTrader 6y ago
I haven't tried Ceph. It didn't seem as appealing, judging from comments I found (mostly on Hacker News): http://www.regressionist.com/2020/06/20/reviews-of-distributed-filesystems/ http://www.regressionist.com/2020/06/20/reviews-of-distribut... But it's also hard to tell exactly what people are reviewing. Evidently CephFS is less mature than Ceph object store. But, yeah. This just all seems so much cheaper than AWS. There are other advantages, too. Like, a lot of stuff is better with the low latency that comes with having the cluster on-location. Python plots are more pleasant to interact with (ssh -X). NoMachine remote desktop is faster, etc. Also, it's a bit like an all-you-can-eat buffet. You pay once, and then you can just relax and enjoy. I have enough storage now that I don't need to stress about costs before deciding to cache intermediate data. And I don't need to worry about whether I really need those older results that I still have saved on disk that I'm paying each minute to store on AWS. Those kind of thoughts are just gone. I do have one GPU now, a 1080 Ti ($450 on eBay). I am ready to house a total of 6, but I'm waiting to buy more until I really need them.