4 ms·

I have always loved these posts, they paint a picture of smart people with good processes in place. It's confidence-building. Unfortunately we were evaluating B

by thom 6y ago

I have always loved these posts, they paint a picture of smart people with good processes in place. It's confidence-building. Unfortunately we were evaluating Backblaze around the time they went down for a weekend and didn't even update their status page, which was a bit of a blow to that confidence.

andruby 6y ago
I hadn't heard about this downtime. When did that happen?
- thom 6y ago
  This was the HN thread: https://news.ycombinator.com/item?id=25147951 https://news.ycombinator.com/item?id=25147951 And I must apologise, I was wrong when I said they didn't update their status page during the outage. They don't have a status page.
csnover 6y ago
Their marketing blog does a great job of painting them as smart people with good processes. Sadly, I learned the hard way (by trialling their software and immediately discovering a handful of dumb bugs that should’ve been caught by QA, plus serious security problems[0], and some OSS licence violations[1]) that it seems to not actually be the case. This situation where they continue to pump out blog posts about hard drive stats, yet don’t even have a status page for reporting outages, is another example of their marketing-driven approach to development. I have mentioned this on HN a couple times now[2][3], including again just yesterday. I really dislike doing this because I feel like I am piling on—but as much as I hate it, I feel even more strongly that people deserve to be informed about these serious ongoing failures at Backblaze so that they can make more informed choices about what storage/backup provider to use. I also genuinely hope that I can incentivise them to actually start following software development best practices, since they do provide a valuable service and I’d like them to succeed. If they keep doing what they’re doing now, I absolutely expect to see a massive data breach and/or data loss event at some point, since they are clearly unwilling or unable to properly handle security or user privacy today—and I’ve gotten the impression over time that some prominent people at the company think these criticisms are invalid and they need not make any substantive changes to their processes. [0] https://twitter.com/zetafleet/status/1304664097989054464 https://twitter.com/zetafleet/status/1304664097989054464 [1] https://twitter.com/bagder/status/1215311286814281728 https://twitter.com/bagder/status/1215311286814281728 [2] https://news.ycombinator.com/item?id=25899802 https://news.ycombinator.com/item?id=25899802 [3] https://news.ycombinator.com/item?id=24839757 https://news.ycombinator.com/item?id=24839757
- atYevP 6y ago
  Yev from Backblaze here -> rest assured that we do read what you're writing on these posts and they've spurred some internal process discussions. I believe the bugs you mentioned were cleared/fixed with version 7.0.0.439 which was released in Q1 of 2020. We did leave HackerOne and switched over to BugCrowd to handle our bug program. It's private at the moment, but easy enough to get invited (by emailing bounty@backblaze.com). While we spin that program up (it's a new vendor for us) we may stay private, but hopefully that's not a permanent state. Edit -> I just noticed the Daniel Stenberg libcurl citation. Oof, yea that certainly a whiff on our end. Luckily though we were able to make up for it (he has a write-up here: https://daniel.haxx.se/blog/2020/01/14/backblazed/ https://daniel.haxx.se/blog/2020/01/14/backblazed/).
  - csnover 6y ago
    > rest assured that we do read what you're writing on these posts and they've spurred some internal process discussions. OK, that’s good to hear. Nobody from the company has reached out to me so this is the first time I’ve been made aware. The only public replies I’ve seen up until now seemed to focus exclusively on just the public bug bounty part, which is really the least important part of this whole thing. > I believe the bugs you mentioned were cleared/fixed with version 7.0.0.439 which was released in Q1 of 2020. It’s really critical to be transparent about this stuff and tell your users. You published release announcements for subsequent versions and there were no mentions of security issues being fixed. When you don’t do this, it looks like you’re intentionally trying to hide vulnerabilities from the public. This is not how any company should act, especially not one that promotes how radically transparent it is[0]. > I just noticed the Daniel Stenberg libcurl citation. Oof, yea that certainly a whiff on our end. Luckily though we were able to make up for it […] I reported license violations in a ticket and nobody replied. You did fix the libcurl violation, which is great, but it took a letter from the author, which is less great. You are still violating the OpenSSL license. It honestly baffles me that nobody at Backblaze thought to check the licenses of the other OSS libraries that you’re distributing after receiving a notice that one of them was being violated. It’s not like there’s a huge compliance burden or complicated dependency tree to evaluate—as far as I can tell, it’s zlib (which requires no acknowledgement), libcurl (which does), and OpenSSL (which also does, for the version you are using[1]). This would’ve taken like 30 seconds. [0] https://www.backblaze.com/blog/transparency-in-business/ https://www.backblaze.com/blog/transparency-in-business/ [1] https://www.openssl.org/source/license-openssl-ssleay.txt https://www.openssl.org/source/license-openssl-ssleay.txt
atYevP 6y ago
Yev from Backblaze here -> sorry about that Thom - it's one of the things we're definitely working on hammering down. Right now we're growing quite a bit on our engineer and infrastructure teams and one of the projects we'd like to see is more automated status updates. We typically will throw updates onto Twitter or our blog if it's a large outage - or affecting many different people, but totally recognize that process can use some sprucing.

by thom 6y ago

andruby 6y ago

I hadn't heard about this downtime. When did that happen?

thom 6y ago
This was the HN thread: https://news.ycombinator.com/item?id=25147951 https://news.ycombinator.com/item?id=25147951 And I must apologise, I was wrong when I said they didn't update their status page during the outage. They don't have a status page.

csnover 6y ago

Their marketing blog does a great job of painting them as smart people with good processes. Sadly, I learned the hard way (by trialling their software and immediately discovering a handful of dumb bugs that should’ve been caught by QA, plus serious security problems[0], and some OSS licence violations[1]) that it seems to not actually be the case. This situation where they continue to pump out blog posts about hard drive stats, yet don’t even have a status page for reporting outages, is another example of their marketing-driven approach to development. I have mentioned this on HN a couple times now[2][3], including again just yesterday. I really dislike doing this because I feel like I am piling on—but as much as I hate it, I feel even more strongly that people deserve to be informed about these serious ongoing failures at Backblaze so that they can make more informed choices about what storage/backup provider to use. I also genuinely hope that I can incentivise them to actually start following software development best practices, since they do provide a valuable service and I’d like them to succeed. If they keep doing what they’re doing now, I absolutely expect to see a massive data breach and/or data loss event at some point, since they are clearly unwilling or unable to properly handle security or user privacy today—and I’ve gotten the impression over time that some prominent people at the company think these criticisms are invalid and they need not make any substantive changes to their processes. [0] https://twitter.com/zetafleet/status/1304664097989054464 https://twitter.com/zetafleet/status/1304664097989054464 [1] https://twitter.com/bagder/status/1215311286814281728 https://twitter.com/bagder/status/1215311286814281728 [2] https://news.ycombinator.com/item?id=25899802 https://news.ycombinator.com/item?id=25899802 [3] https://news.ycombinator.com/item?id=24839757 https://news.ycombinator.com/item?id=24839757

atYevP 6y ago
Yev from Backblaze here -> rest assured that we do read what you're writing on these posts and they've spurred some internal process discussions. I believe the bugs you mentioned were cleared/fixed with version 7.0.0.439 which was released in Q1 of 2020. We did leave HackerOne and switched over to BugCrowd to handle our bug program. It's private at the moment, but easy enough to get invited (by emailing bounty@backblaze.com). While we spin that program up (it's a new vendor for us) we may stay private, but hopefully that's not a permanent state. Edit -> I just noticed the Daniel Stenberg libcurl citation. Oof, yea that certainly a whiff on our end. Luckily though we were able to make up for it (he has a write-up here: https://daniel.haxx.se/blog/2020/01/14/backblazed/ https://daniel.haxx.se/blog/2020/01/14/backblazed/).
- csnover 6y ago
  > rest assured that we do read what you're writing on these posts and they've spurred some internal process discussions. OK, that’s good to hear. Nobody from the company has reached out to me so this is the first time I’ve been made aware. The only public replies I’ve seen up until now seemed to focus exclusively on just the public bug bounty part, which is really the least important part of this whole thing. > I believe the bugs you mentioned were cleared/fixed with version 7.0.0.439 which was released in Q1 of 2020. It’s really critical to be transparent about this stuff and tell your users. You published release announcements for subsequent versions and there were no mentions of security issues being fixed. When you don’t do this, it looks like you’re intentionally trying to hide vulnerabilities from the public. This is not how any company should act, especially not one that promotes how radically transparent it is[0]. > I just noticed the Daniel Stenberg libcurl citation. Oof, yea that certainly a whiff on our end. Luckily though we were able to make up for it […] I reported license violations in a ticket and nobody replied. You did fix the libcurl violation, which is great, but it took a letter from the author, which is less great. You are still violating the OpenSSL license. It honestly baffles me that nobody at Backblaze thought to check the licenses of the other OSS libraries that you’re distributing after receiving a notice that one of them was being violated. It’s not like there’s a huge compliance burden or complicated dependency tree to evaluate—as far as I can tell, it’s zlib (which requires no acknowledgement), libcurl (which does), and OpenSSL (which also does, for the version you are using[1]). This would’ve taken like 30 seconds. [0] https://www.backblaze.com/blog/transparency-in-business/ https://www.backblaze.com/blog/transparency-in-business/ [1] https://www.openssl.org/source/license-openssl-ssleay.txt https://www.openssl.org/source/license-openssl-ssleay.txt

atYevP 6y ago

Yev from Backblaze here -> sorry about that Thom - it's one of the things we're definitely working on hammering down. Right now we're growing quite a bit on our engineer and infrastructure teams and one of the projects we'd like to see is more automated status updates. We typically will throw updates onto Twitter or our blog if it's a large outage - or affecting many different people, but totally recognize that process can use some sprucing.