9 ms·

Backblaze Storage Pod 4.5 – Tweaking a Proven Design

73 points by slyall 12y ago

fraXis 12y ago
Duplicate: https://news.ycombinator.com/item?id=9154273 https://news.ycombinator.com/item?id=9154273
- atYevP 12y ago
  Weird how timing is everything, right?
- slyall 12y ago
  I submitted and got that one. Problem is it had already vanished from the "new" page so wasn't going to make the front, ever. Resubmitted with "Backblaze" in the title to get better traction. I got lucky this time, sometimes I submit something and get nowhere while another submit is the ones that takes off. Maybe a tweak to the new page so that if somebody resubmits a URL that has fallen off the new page it gets pushed back on there. The current system is a bit of a lottery, just one chance for a URL to make it and then it's gone.
k1t 12y ago
Secure Connection Failed An error occurred during a connection to www.backblaze.com. Cannot communicate securely with peer: no common encryption algorithm(s). (Error code: ssl_error_no_cypher_overlap) Oh dear, please upgrade your SSL certs.
- duskwuff 12y ago
  This has nothing to do with the SSL certificate. The site is configured to use RC4, which used to be the recommended cipher to avoid certain attacks (particularly BEAST). However, RC4 has other weaknesses, and this is no longer recommended. Current versions of Firefox will actually refuse to negotiate an RC4 connection.
  - sp332 12y ago
    Doesn't that mean that the certificate is using the outdated RC4 cipher, and should be updated?
    - skuhn 12y ago
      No, the SSL cert does not use RC4. RC4 is a block cipher which is used to encrypt data transferred between your browser and the server, akin to AES. Backblaze does need to step it up in terms of their SSL configuration [1], particularly if this is indicative of their configuration for actual file transfer (no idea though). If I had to guess, they're using their OS's OpenSSL and thus trapped on 0.9.8 or 1.0.0 (both of which do not support TLS v1.2) and maybe haven't spent much time tuning the config lately. A lot of places are afraid of diverging from their OS vendor's version of OpenSSL, but I personally think it is a mistake to get stuck in time with such a crucial component. Likewise if your service is built around nginx -- are you going to stick with whatever RHEL 5 shipped for 8 years? [1] https://www.ssllabs.com/ssltest/analyze.html?d=www.backblaze.com https://www.ssllabs.com/ssltest/analyze.html?d=www.backblaze...
      - WatchDog 12y ago
        RC4 is a stream cipher, not a block cipher.
- miduil 12y ago
  @brianwski (or any other backblaze volunteer) We still can't visit your website. https://news.ycombinator.com/item?id=8999036 https://news.ycombinator.com/item?id=8999036 It's been about a month ago we mentioned this issue.
ricardobeat 12y ago
I really enjoy these posts, and would be a happy Backblaze customer if they didn't have this (unpublicized) policy of deleting external hard-drive backups after 30 days: Backblaze works best if you leave the external hard drive attached to your computer all the time. [...] If the drive is detached for more than 30 days, Backblaze interprets this as data that has been permanently deleted and securely deletes the copy from the Backblaze datacenter. This probably reduces the total amount of storage used, making the $5/month economically viable. I'd happily pay more if it weren't for this though, currently paying ~$12/month for Bitcasa.
- atYevP 12y ago
  Yev from Backblaze here -> we do have that on our website (https://www.backblaze.com/cloud-backup.html https://www.backblaze.com/cloud-backup.html and https://www.backblaze.com/remote-backup-everything.html https://www.backblaze.com/remote-backup-everything.html) and in some help articles but I hear you. It's a topic that comes up, but when we've done the calculations in the past, keeping a copy forever as an archive raises our costs to the point where we'd have to raise our prices and we're not too keen on that! We have notification emails so that we let you know ahead of time before data is removed though, so it's rarely become an issue in terms of "accidental deletions".
  - carterf 12y ago
    Damn, I didn't know that either. Been a customer for years and wish I had seen that earlier. I've also had many lapses in time. As long as I'm paying why would you ever delete my data?
    - atYevP 12y ago
      You should be getting alerts whenever you're nearing a time lapse (up to two weeks ahead of one) so we do try to warn you when your nearing the 30 day timer (if not please contact support - seriously). The reason we have a 30 day timer is that we use the reclaimed space, if we were to infinitely house a copy of every file that was uploaded to us, it would dramatically raise our costs, and at $5/month for an unlimited product our margins are very thin as it is. Most companies that do keep data "forever" augment that cost by having either tiered pricing, or "per GB" pricing so that they can continue covering the cost of the space for as long as that file exists. Since we're unlimited and inexpensive for the service, we don't have that elasticity. That's not to say we haven't thought about enacting an additional fee for that type of service, but we haven't quite figured out a good way of doing it. But we're constantly thinking of ways to better the product, and know this is one that people do feel strongly about!
omarforgotpwd 12y ago
Why design your own storage pods when services like Amazon S3 offer similar prices, even with bandwidth?
- deleted 12y ago
  [deleted]
- skuhn 12y ago
  It's not trivial, but it is fairly easy to beat S3 on storage costs alone, let alone the other factors. I personally wouldn't use Backblaze's storage pod, so my numbers are based on equivalent components (and don't reflect what's possible if you negotiate prices and optimize for your particular needs -- Backblaze's costs are surely lower than this): 11 Supermicro 36 drive 4U server : $4000 each 1 48-port top-of-rack switch : $5000 each 374 4TB drive : $400 each 22 500GB drive : $200 each 2 CDU : $3000 each Rack + integration : $10000 each Total rack cost is $220,000 for 1.4PB or 700GB with basic 2-copy redundancy. Power draw is around 8kw, so 1 year of power is around $1100. Figure on a drive failure rate of 5%, memory 2%, PSU 3% and you might spend another $18,000 to stock spares. So besides your core datacenter costs, this solution will cost $239,100 to operate for a year. These costs only improve when you amortize the hardware over a 3 year period, which is standard. Now how much does S3 cost to store 700GB for a year? $248,530.94. Finally, consider bandwidth costs. I'll crib from a recent comment of mine [1]: 1 gbit/s in a datacenter: $30,000 / year 500 mbit/s in S3: $142,540 / year And that in a nutshell is why AWS doesn't make financial sense at any significant scale. S3 makes sense if you aren't really going to use all of that space: you can scale your spending up as your needs scale up. That isn't the kind of problem Backblaze or other storage businesses have, there isn't much chance of that storage going unused for any extended period of time. And if they architect their datacenter and service correctly, the down period between rolling in hardware and getting it utilized sufficiently to begin seeing a return on the investment should be minimal. [1] https://news.ycombinator.com/item?id=9139244 https://news.ycombinator.com/item?id=9139244
  - moe 12y ago
    Your cost estimates are outlandish, making Amazon S3 look like a bargain. In reality 1.4PB raw will set you back well below $50k/pa, not $239k/pa. If you're lazy you could even rent 1.4PB raw from e.g. Hetzner and you'd still be paying under $90k/pa (that would be 23 of their SX290 servers).
Scarbutt 12y ago
So they don't offer a linux client to keep people away from running it on servers? ;)
grandinj 12y ago
Aaaargh. Yet another idiotic website that disables zoom gestures, making it unreadable on my iphone.

73 points by slyall 12y ago

fraXis 12y ago

Duplicate: https://news.ycombinator.com/item?id=9154273 https://news.ycombinator.com/item?id=9154273

atYevP 12y ago
Weird how timing is everything, right?
slyall 12y ago
I submitted and got that one. Problem is it had already vanished from the "new" page so wasn't going to make the front, ever. Resubmitted with "Backblaze" in the title to get better traction. I got lucky this time, sometimes I submit something and get nowhere while another submit is the ones that takes off. Maybe a tweak to the new page so that if somebody resubmits a URL that has fallen off the new page it gets pushed back on there. The current system is a bit of a lottery, just one chance for a URL to make it and then it's gone.

k1t 12y ago

Secure Connection Failed An error occurred during a connection to www.backblaze.com. Cannot communicate securely with peer: no common encryption algorithm(s). (Error code: ssl_error_no_cypher_overlap) Oh dear, please upgrade your SSL certs.

duskwuff 12y ago
This has nothing to do with the SSL certificate. The site is configured to use RC4, which used to be the recommended cipher to avoid certain attacks (particularly BEAST). However, RC4 has other weaknesses, and this is no longer recommended. Current versions of Firefox will actually refuse to negotiate an RC4 connection.
- sp332 12y ago
  Doesn't that mean that the certificate is using the outdated RC4 cipher, and should be updated?
  - skuhn 12y ago
    No, the SSL cert does not use RC4. RC4 is a block cipher which is used to encrypt data transferred between your browser and the server, akin to AES. Backblaze does need to step it up in terms of their SSL configuration [1], particularly if this is indicative of their configuration for actual file transfer (no idea though). If I had to guess, they're using their OS's OpenSSL and thus trapped on 0.9.8 or 1.0.0 (both of which do not support TLS v1.2) and maybe haven't spent much time tuning the config lately. A lot of places are afraid of diverging from their OS vendor's version of OpenSSL, but I personally think it is a mistake to get stuck in time with such a crucial component. Likewise if your service is built around nginx -- are you going to stick with whatever RHEL 5 shipped for 8 years? [1] https://www.ssllabs.com/ssltest/analyze.html?d=www.backblaze.com https://www.ssllabs.com/ssltest/analyze.html?d=www.backblaze...
    - WatchDog 12y ago
      RC4 is a stream cipher, not a block cipher.
miduil 12y ago
@brianwski (or any other backblaze volunteer) We still can't visit your website. https://news.ycombinator.com/item?id=8999036 https://news.ycombinator.com/item?id=8999036 It's been about a month ago we mentioned this issue.

ricardobeat 12y ago

I really enjoy these posts, and would be a happy Backblaze customer if they didn't have this (unpublicized) policy of deleting external hard-drive backups after 30 days: Backblaze works best if you leave the external hard drive attached to your computer all the time. [...] If the drive is detached for more than 30 days, Backblaze interprets this as data that has been permanently deleted and securely deletes the copy from the Backblaze datacenter. This probably reduces the total amount of storage used, making the $5/month economically viable. I'd happily pay more if it weren't for this though, currently paying ~$12/month for Bitcasa.

atYevP 12y ago
Yev from Backblaze here -> we do have that on our website (https://www.backblaze.com/cloud-backup.html https://www.backblaze.com/cloud-backup.html and https://www.backblaze.com/remote-backup-everything.html https://www.backblaze.com/remote-backup-everything.html) and in some help articles but I hear you. It's a topic that comes up, but when we've done the calculations in the past, keeping a copy forever as an archive raises our costs to the point where we'd have to raise our prices and we're not too keen on that! We have notification emails so that we let you know ahead of time before data is removed though, so it's rarely become an issue in terms of "accidental deletions".
- carterf 12y ago
  Damn, I didn't know that either. Been a customer for years and wish I had seen that earlier. I've also had many lapses in time. As long as I'm paying why would you ever delete my data?
  - atYevP 12y ago
    You should be getting alerts whenever you're nearing a time lapse (up to two weeks ahead of one) so we do try to warn you when your nearing the 30 day timer (if not please contact support - seriously). The reason we have a 30 day timer is that we use the reclaimed space, if we were to infinitely house a copy of every file that was uploaded to us, it would dramatically raise our costs, and at $5/month for an unlimited product our margins are very thin as it is. Most companies that do keep data "forever" augment that cost by having either tiered pricing, or "per GB" pricing so that they can continue covering the cost of the space for as long as that file exists. Since we're unlimited and inexpensive for the service, we don't have that elasticity. That's not to say we haven't thought about enacting an additional fee for that type of service, but we haven't quite figured out a good way of doing it. But we're constantly thinking of ways to better the product, and know this is one that people do feel strongly about!

omarforgotpwd 12y ago

Why design your own storage pods when services like Amazon S3 offer similar prices, even with bandwidth?

deleted 12y ago
[deleted]
skuhn 12y ago
It's not trivial, but it is fairly easy to beat S3 on storage costs alone, let alone the other factors. I personally wouldn't use Backblaze's storage pod, so my numbers are based on equivalent components (and don't reflect what's possible if you negotiate prices and optimize for your particular needs -- Backblaze's costs are surely lower than this): 11 Supermicro 36 drive 4U server : $4000 each 1 48-port top-of-rack switch : $5000 each 374 4TB drive : $400 each 22 500GB drive : $200 each 2 CDU : $3000 each Rack + integration : $10000 each Total rack cost is $220,000 for 1.4PB or 700GB with basic 2-copy redundancy. Power draw is around 8kw, so 1 year of power is around $1100. Figure on a drive failure rate of 5%, memory 2%, PSU 3% and you might spend another $18,000 to stock spares. So besides your core datacenter costs, this solution will cost $239,100 to operate for a year. These costs only improve when you amortize the hardware over a 3 year period, which is standard. Now how much does S3 cost to store 700GB for a year? $248,530.94. Finally, consider bandwidth costs. I'll crib from a recent comment of mine [1]: 1 gbit/s in a datacenter: $30,000 / year 500 mbit/s in S3: $142,540 / year And that in a nutshell is why AWS doesn't make financial sense at any significant scale. S3 makes sense if you aren't really going to use all of that space: you can scale your spending up as your needs scale up. That isn't the kind of problem Backblaze or other storage businesses have, there isn't much chance of that storage going unused for any extended period of time. And if they architect their datacenter and service correctly, the down period between rolling in hardware and getting it utilized sufficiently to begin seeing a return on the investment should be minimal. [1] https://news.ycombinator.com/item?id=9139244 https://news.ycombinator.com/item?id=9139244
- moe 12y ago
  Your cost estimates are outlandish, making Amazon S3 look like a bargain. In reality 1.4PB raw will set you back well below $50k/pa, not $239k/pa. If you're lazy you could even rent 1.4PB raw from e.g. Hetzner and you'd still be paying under $90k/pa (that would be 23 of their SX290 servers).

Scarbutt 12y ago

So they don't offer a linux client to keep people away from running it on servers? ;)

grandinj 12y ago

Aaaargh. Yet another idiotic website that disables zoom gestures, making it unreadable on my iphone.