4 ms·

As a provider of cloud storage (who actually owns the infrastructure) this is both exciting and frustrating... Exciting because we love the idea of deploying a

by rsync 11y ago

As a provider of cloud storage (who actually owns the infrastructure) this is both exciting and frustrating... Exciting because we love the idea of deploying a 450TB zpool in a 4U chassis ... Frustrating because we will have to keep reworking our price per GB. With that in mind, our new attic/borg[1] support has dropped down to 3 cents per GB, per month to kind-of-sort-of match S3 pricing[2]. [1] http://www.stavros.io/posts/holy-grail-backups/ http://www.stavros.io/posts/holy-grail-backups/ [2] http://www.rsync.net/products/attic.html http://www.rsync.net/products/attic.html

pil4rin 11y ago
(who actually owns the infrastructure) Hipster </sarcasm>
- vetrom 11y ago
  If your business is enhanced storage, your margin in storage infra is directly relevant to your COGS. I got to learn this the hard way, having once worked in the field. (SpiderOak)
- toomuchtodo 11y ago
  By Hipster you must mean "intelligent business owner not throwing money away". Storage is cheap when you own it vs renting it: https://i.imgur.com/WZriUP5.jpg https://i.imgur.com/WZriUP5.jpg
drzaiusapelord 11y ago
Whats your current storage in a 4U chassis?
- rsync 11y ago
  45 drives @ 4TB, but we create 15 drive raidz3 vdevs, so we lose 3 drives per vdev ... so (45-9)*4 = 144TB.
  - drzaiusapelord 11y ago
    Any reason why you're not using 6TB yet? Reliability?
    - rsync 11y ago
      We like to match drive types in chassis and sometimes cabinets - that way we don't have 8 different kids of spares to juggle. Right now we just happen to have a lot of 4TB arrays in use. But yes, we will deploy arrays with 6TB very soon.
  - davidandgoliath 11y ago
    Curious — what sort of iops do you get out of a setup like that?
X-Istence 11y ago
SuperMicro has 72 drives in a single system, with 10TB drives you are hitting 720TB in a single 4U package. Dell also has their DSS7000, which is 90 drives in a 4U system, so 900 TB in a single 4U package... 9 Petabyte of storage in a single 42U rack...
- rsync 11y ago
  Once you go past the 45 drive JBOD (which costs $2000 give or take) the price per slot starts to go up quickly ... the last quote we had for the supermicro 90 drive JBOD was $9000. So if you are optimizing for rackspace, it might make sense, but if you're keeping costs down, the 45 drive chassis is the best bet.
  - toomuchtodo 11y ago
    Have you considered using the Backblaze pods?
    - rsync 11y ago
      Yes, from the very beginning, but they are based on SATA expanders and have relatively very poor performance. They are really built specifically for that workload and not for a lot of other storage workloads. That was depressing circa the first BB pods where there really wasn't another option (other than using a Sun Thumper/x4500, which we also considered) ... but now the chassis options from supermicro are so good - ranging from 45-90 drives - that it's not a problem at all.
  - X-Istence 11y ago
    The Dell DSS7000 chassis without the computer nodes on the back (which are a requirement (45 drives per compute node)) is cheaper than the 90 disk JBOD from SuperMicro. I can't disclose pricing, but it may be worth reaching out to Dell for what they can offer there. You might be able to get better density for similar pricing. Just a suggestion :-)
JungleGymSam 11y ago
> If you're not sure what this means, our product is Not For You. Hey, thanks! I stopped reading there. :)
- rsync 11y ago
  We both win!
  - lfowles 11y ago
    That's a refreshing attitude. I wish AWS had something like that warning... "If you only want a VPS to idle in IRC with... stop reading, this is not for you."
  - PhantomGremlin 11y ago
    I agree with this. However, it would be nice to know a little about the company. I perused your website, but didn't see any pages about the company history and current employees. Did I miss that? I did finally see the "CEO page", but it doesn't seem to be linked from the main page. It also appears to be something a technical person shows their CEO, rather than something that is about the CEO of rsync.net. I also didn't even see a physical address displayed anywhere. Maybe it's just me, but I wouldn't feel comfortable dealing with such a black box business situation. Perhaps you need to add a second disclaimer? If you want to know who you're doing business with, our product is Not for You.
    - rsync 11y ago
      We just redesigned our website and I guess our old "about" page didn't make it in the redeploy. FWIW, I'm John Kozubik, owner and founder, and we started offering this service in 2001 as an add-on to JohnCompanies colocation (the first VPS provider)[1][2]. We spun rsync.net out as a standalone entity in 2006. Address is 524 San Anselmo Ave., Suite 107, San Anselmo, CA 94960. Although data is housed in either San Diego, Fremont, Denver, Zurich or Hong Kong datacenters. [1] Yes, really. [2] We called them "server instances" - someone else coined "VPS" and it stuck.
monochromatic 11y ago
> Frustrating because we will have to keep reworking our price per GB. You don't have to pass every savings on to the consumer, you know.
zx2c4 11y ago
Does anybody take the encryption support in attic/borg seriously? They seem to be implementing their own AE construction. Maybe they did it right, but it makes be a bit uneasy. While they're at it wrapping OpenSSL, I don't know why they wouldn't use one of OpenSSL's AEAD EVPs...
JoshTriplett 11y ago
Which of Attic or Borg would you recommend for new usage? Deduplicated always-incremental backups have appealed to me ever since I started following the development of obnam, and I'd love to switch away from duplicity and stop requiring periodic full backups.
- rsync 11y ago
  Yes, that's the reason everyone is eschewing duplicity (which we've always loved and supported) for attic and borg. borg appears to be much healthier as a project, has a lot of steam behind it, and has some additional flexibility in block sizes, etc., so we think that's the train to hitch your wagon to ...

by rsync 11y ago

pil4rin 11y ago

(who actually owns the infrastructure) Hipster </sarcasm>

vetrom 11y ago
If your business is enhanced storage, your margin in storage infra is directly relevant to your COGS. I got to learn this the hard way, having once worked in the field. (SpiderOak)
toomuchtodo 11y ago
By Hipster you must mean "intelligent business owner not throwing money away". Storage is cheap when you own it vs renting it: https://i.imgur.com/WZriUP5.jpg https://i.imgur.com/WZriUP5.jpg

drzaiusapelord 11y ago

Whats your current storage in a 4U chassis?

rsync 11y ago
45 drives @ 4TB, but we create 15 drive raidz3 vdevs, so we lose 3 drives per vdev ... so (45-9)*4 = 144TB.
- drzaiusapelord 11y ago
  Any reason why you're not using 6TB yet? Reliability?
  - rsync 11y ago
    We like to match drive types in chassis and sometimes cabinets - that way we don't have 8 different kids of spares to juggle. Right now we just happen to have a lot of 4TB arrays in use. But yes, we will deploy arrays with 6TB very soon.
- davidandgoliath 11y ago
  Curious — what sort of iops do you get out of a setup like that?

X-Istence 11y ago

SuperMicro has 72 drives in a single system, with 10TB drives you are hitting 720TB in a single 4U package. Dell also has their DSS7000, which is 90 drives in a 4U system, so 900 TB in a single 4U package... 9 Petabyte of storage in a single 42U rack...

rsync 11y ago
Once you go past the 45 drive JBOD (which costs $2000 give or take) the price per slot starts to go up quickly ... the last quote we had for the supermicro 90 drive JBOD was $9000. So if you are optimizing for rackspace, it might make sense, but if you're keeping costs down, the 45 drive chassis is the best bet.
- toomuchtodo 11y ago
  Have you considered using the Backblaze pods?
  - rsync 11y ago
    Yes, from the very beginning, but they are based on SATA expanders and have relatively very poor performance. They are really built specifically for that workload and not for a lot of other storage workloads. That was depressing circa the first BB pods where there really wasn't another option (other than using a Sun Thumper/x4500, which we also considered) ... but now the chassis options from supermicro are so good - ranging from 45-90 drives - that it's not a problem at all.
- X-Istence 11y ago
  The Dell DSS7000 chassis without the computer nodes on the back (which are a requirement (45 drives per compute node)) is cheaper than the 90 disk JBOD from SuperMicro. I can't disclose pricing, but it may be worth reaching out to Dell for what they can offer there. You might be able to get better density for similar pricing. Just a suggestion :-)

JungleGymSam 11y ago

> If you're not sure what this means, our product is Not For You. Hey, thanks! I stopped reading there. :)

rsync 11y ago
We both win!
- lfowles 11y ago
  That's a refreshing attitude. I wish AWS had something like that warning... "If you only want a VPS to idle in IRC with... stop reading, this is not for you."
- PhantomGremlin 11y ago
  I agree with this. However, it would be nice to know a little about the company. I perused your website, but didn't see any pages about the company history and current employees. Did I miss that? I did finally see the "CEO page", but it doesn't seem to be linked from the main page. It also appears to be something a technical person shows their CEO, rather than something that is about the CEO of rsync.net. I also didn't even see a physical address displayed anywhere. Maybe it's just me, but I wouldn't feel comfortable dealing with such a black box business situation. Perhaps you need to add a second disclaimer? If you want to know who you're doing business with, our product is Not for You.
  - rsync 11y ago
    We just redesigned our website and I guess our old "about" page didn't make it in the redeploy. FWIW, I'm John Kozubik, owner and founder, and we started offering this service in 2001 as an add-on to JohnCompanies colocation (the first VPS provider)[1][2]. We spun rsync.net out as a standalone entity in 2006. Address is 524 San Anselmo Ave., Suite 107, San Anselmo, CA 94960. Although data is housed in either San Diego, Fremont, Denver, Zurich or Hong Kong datacenters. [1] Yes, really. [2] We called them "server instances" - someone else coined "VPS" and it stuck.

monochromatic 11y ago

> Frustrating because we will have to keep reworking our price per GB. You don't have to pass every savings on to the consumer, you know.

zx2c4 11y ago

Does anybody take the encryption support in attic/borg seriously? They seem to be implementing their own AE construction. Maybe they did it right, but it makes be a bit uneasy. While they're at it wrapping OpenSSL, I don't know why they wouldn't use one of OpenSSL's AEAD EVPs...

JoshTriplett 11y ago

Which of Attic or Borg would you recommend for new usage? Deduplicated always-incremental backups have appealed to me ever since I started following the development of obnam, and I'd love to switch away from duplicity and stop requiring periodic full backups.

rsync 11y ago
Yes, that's the reason everyone is eschewing duplicity (which we've always loved and supported) for attic and borg. borg appears to be much healthier as a project, has a lot of steam behind it, and has some additional flexibility in block sizes, etc., so we think that's the train to hitch your wagon to ...