4 ms·
Their marketing blog does a great job of painting them as smart people with good processes. Sadly, I learned the hard way (by trialling their software and immed
by csnover 6y ago
Their marketing blog does a great job of painting them as smart people with good processes. Sadly, I learned the hard way (by trialling their software and immediately discovering a handful of dumb bugs that should’ve been caught by QA, plus serious security problems[0], and some OSS licence violations[1]) that it seems to not actually be the case. This situation where they continue to pump out blog posts about hard drive stats, yet don’t even have a status page for reporting outages, is another example of their marketing-driven approach to development.
I have mentioned this on HN a couple times now[2][3], including again just yesterday. I really dislike doing this because I feel like I am piling on—but as much as I hate it, I feel even more strongly that people deserve to be informed about these serious ongoing failures at Backblaze so that they can make more informed choices about what storage/backup provider to use. I also genuinely hope that I can incentivise them to actually start following software development best practices, since they do provide a valuable service and I’d like them to succeed. If they keep doing what they’re doing now, I absolutely expect to see a massive data breach and/or data loss event at some point, since they are clearly unwilling or unable to properly handle security or user privacy today—and I’ve gotten the impression over time that some prominent people at the company think these criticisms are invalid and they need not make any substantive changes to their processes.
[0] https://twitter.com/zetafleet/status/1304664097989054464 https://twitter.com/zetafleet/status/1304664097989054464
[1] https://twitter.com/bagder/status/1215311286814281728 https://twitter.com/bagder/status/1215311286814281728
[2] https://news.ycombinator.com/item?id=25899802 https://news.ycombinator.com/item?id=25899802
[3] https://news.ycombinator.com/item?id=24839757 https://news.ycombinator.com/item?id=24839757
- atYevP 6y agoYev from Backblaze here -> rest assured that we do read what you're writing on these posts and they've spurred some internal process discussions. I believe the bugs you mentioned were cleared/fixed with version 7.0.0.439 which was released in Q1 of 2020. We did leave HackerOne and switched over to BugCrowd to handle our bug program. It's private at the moment, but easy enough to get invited (by emailing bounty@backblaze.com). While we spin that program up (it's a new vendor for us) we may stay private, but hopefully that's not a permanent state. Edit -> I just noticed the Daniel Stenberg libcurl citation. Oof, yea that certainly a whiff on our end. Luckily though we were able to make up for it (he has a write-up here: https://daniel.haxx.se/blog/2020/01/14/backblazed/ https://daniel.haxx.se/blog/2020/01/14/backblazed/).
- csnover 6y ago> rest assured that we do read what you're writing on these posts and they've spurred some internal process discussions. OK, that’s good to hear. Nobody from the company has reached out to me so this is the first time I’ve been made aware. The only public replies I’ve seen up until now seemed to focus exclusively on just the public bug bounty part, which is really the least important part of this whole thing. > I believe the bugs you mentioned were cleared/fixed with version 7.0.0.439 which was released in Q1 of 2020. It’s really critical to be transparent about this stuff and tell your users. You published release announcements for subsequent versions and there were no mentions of security issues being fixed. When you don’t do this, it looks like you’re intentionally trying to hide vulnerabilities from the public. This is not how any company should act, especially not one that promotes how radically transparent it is[0]. > I just noticed the Daniel Stenberg libcurl citation. Oof, yea that certainly a whiff on our end. Luckily though we were able to make up for it […] I reported license violations in a ticket and nobody replied. You did fix the libcurl violation, which is great, but it took a letter from the author, which is less great. You are still violating the OpenSSL license. It honestly baffles me that nobody at Backblaze thought to check the licenses of the other OSS libraries that you’re distributing after receiving a notice that one of them was being violated. It’s not like there’s a huge compliance burden or complicated dependency tree to evaluate—as far as I can tell, it’s zlib (which requires no acknowledgement), libcurl (which does), and OpenSSL (which also does, for the version you are using[1]). This would’ve taken like 30 seconds. [0] https://www.backblaze.com/blog/transparency-in-business/ https://www.backblaze.com/blog/transparency-in-business/ [1] https://www.openssl.org/source/license-openssl-ssleay.txt https://www.openssl.org/source/license-openssl-ssleay.txt
- brianwski 6y agoDisclaimer: I work at Backblaze. > as much as I hate it I'm following Backblaze around and posting incorrect information about them I get the impression Backblaze did something to upset you. Can you let me know what it is so I can try to fix it? If there wasn't a pandemic on I would invite you to come to our office and I could buy you lunch and I could try to make up for whatever we did to upset you.
- breakingcups 6y agoHi Brian, I'm not the person you responded to and I'm a happy user of both Backblaze and B2, but I wanted you to know that this response by you reads as quite disingenuous. You seem to want to shift the reason for his disgruntlement with Backblaze from all the reasons he already mentioned to some other, imaginary slight that you indicate you'll do your very best to fix. How about just reading his very real gripes and responding to those? Let's take his twitter thread for some highlights, these seem like very real reasons to get upset, maybe you "can try to fix" those? * Backblaze changed their client to add an allowlist some time after my report, while also intentionally breaking their TLS code so it would accept INVALID TLS certificates. Thereafter, the local code execution vuln became a full blown RCE vuln. * When I submitted my report 11 months ago, they told me they already knew about the problem, downplayed its severity, dodged follow-up questions, didn’t seem to understand how CVE IDs work and refused to issue one after being asked four times. It was not confidence-inspiring. The CVE ID for the vulnerability I gave to them is CVE-2019-19904. They should’ve announced it, but they never did. Actually, they never seem to voluntarily disclose any security bugs… there are a lot of verified, closed, undisclosed bugs on their HackerOne account. * This is all in stark contrast to their security page (https://backblaze.com/security.html https://backblaze.com/security.html) which makes many claims about best practices, and their blog & social media which present a sense of radical openness. I used to like their blog, but it all feels so gross and dishonest to me now. * Backblaze mislead users about PEK. The decryption key is sent to their server, and so is your password. The only way to restore data is to decrypt it on their servers first. It is not a zero-knowledge system. PEK data is not ‘inaccessible’ to them. They don’t care. At face value (and I haven't done any digging of my own) these all seem like valid reasons to distrust Backblaze. Not necessarily because they happened, but because of the way Backblaze has addressed them (read: apparently not)
- barrkel 6y agoThis seems on the money: https://news.ycombinator.com/item?id=25913516 https://news.ycombinator.com/item?id=25913516
- PhantomGremlin 6y agoI also genuinely hope that I can incentivise them to actually start following software development best practices, since they do provide a valuable service and I’d like them to succeed. I read thru your HN posts and twitter and I completely understand where you're coming from. Very good points. But here's my response to you: You're paying them $6 per month. That's not enough money for them to do better than what they're doing now. They literally can't afford to do better, given their paltry income stream. (Just my guess, I have no insider info about them). If they keep doing what they’re doing now, I absolutely expect to see a massive data breach and/or data loss event at some point I guess that's a chance they're prepared to take, though they probably haven't thought about it in such stark terms. I'm sure their development practices will improve, maybe in time to prevent a massive breach or loss.
- brianwski 6y agoDisclaimer: I work at Backblaze so I'm biased and you should keep me honest. > they probably haven't thought about it in such stark terms Backblaze takes security incredibly seriously, and I assure you we have thought about it in such stark terms. Expounding on that: Backblaze never raised any significant VC funding, we survive entirely on sales of our products, and most of that is keeping our customer's data utterly private, confidential, and safe. As this is the business we are in, our reputation is INCREDIBLY important to us. If we have a major breach we'll most likely lose our customer's trust, lose all our customers, and go out of business. Since this is all I've done for the last 14 years, and my income and life savings is all wrapped up in Backblaze, I take this issue as seriously as a heart attack. So do my business partners (Backblaze was founded by 5 equal partners.) We're also up to around 200 employees who all would suffer greatly if we lose the trust of our customers. Internally, Backblaze has a "Security Council" of software engineers and technical operations people with something like a combined 150 years of security experience and obsession. One of these council members got his CS degree from MIT and is both one of the smartest people I've ever worked with (we've worked together at 3 separate companies so far over 25 years) and also deeply paranoid and stressed out all the time. Another of the security council members has a PhD in computer science. And so on... They watch over all the design proposals, the APIs, the technical infrastructure, everything at Backblaze. They propose and implement new procedures, new programs like our BugCrowd program where we have external white hat hackers constantly trying to break in. We are also going through an internal security audit right now paying consultants to get yet another perspective. In addition to the Security Council, all software engineers and all technical operations people are expected to worry about security all of the time. It is quite possibly our most talked about, most worried about, most important thing we do. > I'm sure their development practices will improve We always, ALWAYS strive to do better and do more.