4 ms·

SpiderOak (the backup and sync company) is on the verge of launching a S3-like service, tuned for for archival class data. It's in beta now. It's also entirel

SpiderOak (the backup and sync company) is on the verge of launching a S3-like service, tuned for for archival class data. It's in beta now. It's also entirely open source, so you could run it on your own hardware if you wanted. It uses parity instead of replication for storing data (but with arbitrary effectively replication levels.) So with R=3, you need a minimum of about 10 nodes to be efficient. The beta uses AMQP internally, and the upcoming release version is Python/zeromq/gevent based. https://spideroak.com/diy/ https://spideroak.com/diy/

e1ven 15y ago
This seems to be a great offering The pricing is not entirely out of line compared with what I can get in-house, and it's a fantastic solution for storing horribly bulk,low-reliability data. $10 / 100GB comes out to .01/GB/Month. That means that you pay the total cost of an in-house solution, every month you use their service.. That's 12x the price of doing it yourself, but you don't need to buy replacement drives, pay for electricity, or send someone to go replace drives. It's not that much cheaper than Amazon in the base pricing, but it looks like you can relax the number of copies for them to keep from 3(default) to 1, and cut the price down to 1/3? Is this your experience? Thanks for the link.
- rarrrrrr 15y ago
  FYI, I'm one of the founders of SpiderOak, it case that was not clear (it's in my profile.) For the storage cost calculations: electricity, cooling, and bandwidth end up costing more than the drives, even once you include all additional hardware to keep them running. Note of caution: I would recommend that shops build bulk storage hardware themselves only if someone on their team has an intensive storage background. There are a number of gotchas that are distinctly non obvious; they are capital intensive lessons to learn, and endanger data integrity. You saw what Backblaze went through to design their storage pods: all of the self hosted offsite backup companies do likewise. It's a core competency. Typically it's much better to outsource or buy the more expensive business storage products with hardware RAID, enterprise class drives, cache batteries, and all that. SpiderOak's DIY is designed for high reliability, but not high performance. In price comparisons to S3, there's no option for reduced-reliability storage, but also no charge for bandwidth in our out (up to a reasonable point.) Having said all that, if you are interested in building 100+ TB of storage hardware in house, feel free to send me a mail; I may be able to save you some difficulty. If you'd rather us host it, we do discounts for startups. :)
  - e1ven 15y ago
    Oh, Man, that'll teach me to look more carefully ;) Oops. Thanks for the awesome service. I've used SO for backups on and off, and I appreciate the tech work you do. I particularly like that you do client-side encryption. I just wish your UI was a bit better ;) I must have been mixing up your service with diomede; IIRC, their techcrunch article was about letting you choose the levels of redundancy you want to keep. I'll grant you that the bandwidth is certainly a price-factor, but your page talks a LOT about how you're higher latency than S3, and not everyone needs that speed for that price, etc, etc, and then your price comes out to be about 80% of theirs. Honestly, after reading your product positioning, I was envisioning it coming out to like 10%. You're absolutely right that building a huge in-house storage network is a huge ordeal.. If you're worried about making it fast and scalable. What I want to do is to treat it as the equivalent of a huge tape drive- Just throw stuff onto it, and hope for the best. I've been toying around with the idea of dealing with it entirely in application logic. Rather than RAIDing the disks, mount each one separately, and keep a hashtable of my files in memory with their full path, including redundant copies. Then, if a file fails, I have application logic try the backup copy. Anyway, you know a LOT more about this than I do, so I'll shut up before I embarrass myself further. My point is just that there's a lot of people in the market who have huge datasets currently, for various reasons. Lots of science departments, analytics groups, etc. I'd love to be able to keep an nearline copy of work. I don't mind if it takes a good 15-20 minutes to be able to access the first byte, I just want to know that I can if I need to, faster than waiting for Iron Mountain to ship me something. Side note- I considered trying to justify it under BackBlaze's "$5, unlimited space" offer, but I didn't think they'd go for it. ;)

e1ven 15y ago

This seems to be a great offering The pricing is not entirely out of line compared with what I can get in-house, and it's a fantastic solution for storing horribly bulk,low-reliability data. $10 / 100GB comes out to .01/GB/Month. That means that you pay the total cost of an in-house solution, every month you use their service.. That's 12x the price of doing it yourself, but you don't need to buy replacement drives, pay for electricity, or send someone to go replace drives. It's not that much cheaper than Amazon in the base pricing, but it looks like you can relax the number of copies for them to keep from 3(default) to 1, and cut the price down to 1/3? Is this your experience? Thanks for the link.

rarrrrrr 15y ago
FYI, I'm one of the founders of SpiderOak, it case that was not clear (it's in my profile.) For the storage cost calculations: electricity, cooling, and bandwidth end up costing more than the drives, even once you include all additional hardware to keep them running. Note of caution: I would recommend that shops build bulk storage hardware themselves only if someone on their team has an intensive storage background. There are a number of gotchas that are distinctly non obvious; they are capital intensive lessons to learn, and endanger data integrity. You saw what Backblaze went through to design their storage pods: all of the self hosted offsite backup companies do likewise. It's a core competency. Typically it's much better to outsource or buy the more expensive business storage products with hardware RAID, enterprise class drives, cache batteries, and all that. SpiderOak's DIY is designed for high reliability, but not high performance. In price comparisons to S3, there's no option for reduced-reliability storage, but also no charge for bandwidth in our out (up to a reasonable point.) Having said all that, if you are interested in building 100+ TB of storage hardware in house, feel free to send me a mail; I may be able to save you some difficulty. If you'd rather us host it, we do discounts for startups. :)
- e1ven 15y ago
  Oh, Man, that'll teach me to look more carefully ;) Oops. Thanks for the awesome service. I've used SO for backups on and off, and I appreciate the tech work you do. I particularly like that you do client-side encryption. I just wish your UI was a bit better ;) I must have been mixing up your service with diomede; IIRC, their techcrunch article was about letting you choose the levels of redundancy you want to keep. I'll grant you that the bandwidth is certainly a price-factor, but your page talks a LOT about how you're higher latency than S3, and not everyone needs that speed for that price, etc, etc, and then your price comes out to be about 80% of theirs. Honestly, after reading your product positioning, I was envisioning it coming out to like 10%. You're absolutely right that building a huge in-house storage network is a huge ordeal.. If you're worried about making it fast and scalable. What I want to do is to treat it as the equivalent of a huge tape drive- Just throw stuff onto it, and hope for the best. I've been toying around with the idea of dealing with it entirely in application logic. Rather than RAIDing the disks, mount each one separately, and keep a hashtable of my files in memory with their full path, including redundant copies. Then, if a file fails, I have application logic try the backup copy. Anyway, you know a LOT more about this than I do, so I'll shut up before I embarrass myself further. My point is just that there's a lot of people in the market who have huge datasets currently, for various reasons. Lots of science departments, analytics groups, etc. I'd love to be able to keep an nearline copy of work. I don't mind if it takes a good 15-20 minutes to be able to access the first byte, I just want to know that I can if I need to, faster than waiting for Iron Mountain to ship me something. Side note- I considered trying to justify it under BackBlaze's "$5, unlimited space" offer, but I didn't think they'd go for it. ;)