11 ms·
Yev from Backblaze here -> we’ve looked into and verified the issue and have pushed out a fix. We will continue to investigate and will provide updates as we ha
by atYevP 6y ago
Yev from Backblaze here -> we’ve looked into and verified the issue and have pushed out a fix. We will continue to investigate and will provide updates as we have them.
- KingOfCoders 6y agoCan you fix all your security processes too? And make tech drive tech decisions not let marketing define security? Thanks.
- tobr 6y agoThanks for participating in this thread. As a longtime paying customer, I consider this a monumental security breach and I will be leaving the service. It’s clear that Backblaze have prioritized growth hacking or whatever over the security and privacy of me as a paying customer, and that your security processes are woefully inadequate.
- skinkestek 6y agoWell, for many of us this is where it starts: I might very well start using backblaze next year or maybe even next month[1], but that is depending on the outcome of this event. For a comparison: one good friend once called med to apologize that he had been laughing behind my back with some friends. Guess who I definitely trust today? The one admitted his mistake. He always was a nice bloke and I guess he will never ever do anything like that ever again. [1]: I won't start using it this week or the next however.
- tobr 6y agoWhat kind of scenario do you have in mind? I think it’s possible to turn an incident like this around, PR-wise, but I can’t see how they will explain how they can sell something as secure and trusted, when their security process was unable to discover that they had deployed spyware in production. If it was there for a few hours before it was discovered and removed, well maybe.
- ehnto 6y agoJust hope they announce the changes to security and implementation processes rather than just if they fix this issue or not. This really shouldn't have occurred in the first place, so you want to know they've fixed the root cause, bad process, not just the symptom.
- atYevP 6y agoYev here - Tobr, thank you for being a customer. Wanted to let you know that we've updated our blog post after finishing our root cause analysis. You can read about what happened here: https://www.backblaze.com/blog/privacy-update-third-party-tracking/ https://www.backblaze.com/blog/privacy-update-third-party-tr....
- matkoniecz 6y agoWhat kind of failures lead to such critical data leak getting released? Who thought that Facebook tracking should be forced on paying customers?
- iamacyborg 6y agoA marketer who wanted to build lookalike audiences from existing customers, I’d guess.
- matkoniecz 6y agoIf marketers may randomly deploy code without review then something is deeply broken. If this passed review - it is likely even worse.
- iamacyborg 6y agoI agree. Unfortunately marketers deploying custom code via Google Tag Manager is the norm.
- varispeed 6y agoIt's troubling that companies don't see this as unethical at best.
- ohlookabird 6y agoThanks. However, it is just not excusable and a breach of trust. The Backblaze Twitter communication making pretty clear that they don't see a problem with tracking paying customers. We are moving somewhere else.
- nerdponx 6y agoSame, I'm out. Who are you moving to?
- quchen 6y agoI’ve looked at Backblaze and Wasabi for my personal off-site backups, and sided with Wasabi. Can’t complain so far.
- thejohnconway 6y agoDoes Wasabi do something similar? Have you checked? There’s probably a very limited window to do that, I’d imagine everyone in this space is checking their trackers now.
- jakemauer 6y agoI just signed up for a free trial with Wasabi and they include trackers from Google analytics and LogRocket. I'm not sure if they send filename data across the wire.
- ddalex 6y agoIs it smart to tell people that you are leaving before you actually backed-up and moved you data in the new place!?
- tobr 6y agoI kinda lost whatever trust I had in Backblaze here, but I don’t think they’re in the habit of deleting paying customers’ data in vengeance.
- walrus01 6y agoFixing the problem is only half of it, you need to make a commitment to a comprehensive and transparent review of the engineering practices that allowed this to go to production. And fully disclose how long it has been going on.
- atYevP 6y agoYev here -> we've since finished our root cause analysis and have updated our blog with additional information that you can find here: https://www.backblaze.com/blog/privacy-update-third-party-tracking/ https://www.backblaze.com/blog/privacy-update-third-party-tr...
- dgellow 6y agoHi Yev. It's great to know that you fixed it. Could you share since when you had this issue? I'm a long-time paying customer and feel somehow betrayed here.
- Symbiote 6y agoAccording to Backblaze's own policies, they will be emailing you about this data breach "without undue delay" — at least if you've logged in while the breach was present: > In the unlikely event of a data breach, as defined in the GDPR, Backblaze will without undue delay send its affected customers a notification email, and provide at its discretion, updates through other communications channels. This notification will describe the nature of the data breach, including where possible, the categories and approximate number of data subjects concerned, the categories and approximate number of personal data records concerned, the contact point where more information can be obtained, the likely consequences of the personal data breach, and the measures taken or proposed to be taken by Backblaze to address the data breach, including, where appropriate, measures to mitigate its possible adverse effects. https://www.backblaze.com/company/dpa.html https://www.backblaze.com/company/dpa.html
- atYevP 6y agoYev here -> thank you for being a customer! We just finished root cause analysis and have updated our blog post with additional information -> https://www.backblaze.com/blog/privacy-update-third-party-tracking/ https://www.backblaze.com/blog/privacy-update-third-party-tr...
- magicalhippo 6y agoAnd by pushed out a fix you mean remove the Facebook pixel entirely, and all other 3rd party scripts from the dashboard pages? Because that's the only valid fix here.
- knolan 6y agoI would think the type of customer that used Backblaze is orthogonal to a typical Facebook user and is actively hostile to Facebook’s practises. By even considering using their spyware in any shape or form is an egregious breach of trust and the response from your company needs to be significantly better to restore confidence. Also is this “bug” in breach of GDPR?
- mnw21cam 6y agoThe exact phrasing you have used here is repeated multiple times in the Twitter thread, and I can only conclude that "pushed out a fix" is what your marketing department has decided to call what you have done. What you have right here and right now is a public relations disaster. Trust in your brand has been damaged. It cannot be repaired by you providing minimal information. Your standardised message is akin to "Don't worry your little heads over the details - trust us, everything is fine now", and to be honest I find it a bit insulting. As far as we know, "pushed out a fix" could mean that you have hidden the tracking, so it is harder to find. Your short message is making the public relations disaster worse, not better. These are the steps that you need to take: 1. Provide an explanation of why tracking was being performed in the first place, including an analysis of how much of that was a mistake. 2. Make an apology for breaching your customers' trust. This is a really important step, and it should be repeated in each of your press releases. 3. Provide details on the steps you have taken to fix the problem, and what that means for tracking data. 4. Make a promise that strictly limits the level of tracking that you will be allowing yourself to make in the future. Ideally we would all want that to be zero, and if you intend to do business with certain jurisdictions then you are limited to what is legal, but you must in any case be clear about what tracking you will ever do. Honesty and transparency are the keys at this point to restoring your brand. I do not think the community will accept anything less.
- elefantastisch 6y agoThis. Yev, if you are not in the upper management chain of Backblaze, please show mnw21cam's message to someone who is. The problem is not that there was a little bug which caused the Facebook tracker to get a few little pieces of information it shouldn't have. The problem is that Backblaze failed to understand how to distinguish appropriate and inappropriate uses of third-party trackers for signed-in users on a security-critical application. The Facebook pixel should never have been there at all. It shouldn't even have been considered. It should've been an absolute no-brainer that Facebook has no business being on secure pages on a critical infrastructure service for paying customers. The fact that the pixel even showed up at all on a logged in page represents a breach of trust for customers and casts doubt on Backblaze's competence in handling security issues. This warrants a serious reply from the CEO, not a copy-pasted meaningless reassurance.
- Ensorceled 6y agoHey Yev, As another data point, Backblaze has pretty much until this weekend to provide an update that includes "we have removed Facebook and are getting a 3rd party review for other security holes in our product". After this weekend, I won't care because I'll be on a different product. This is embarrassingly bad, as in, I'm now embarrassed for recommending using Backblaze at my company. Crap, I just realized I got Backblaze installed at two previous companies. Thanks!
- atYevP 6y agoYev here -> thanks for being a customer! We've finished our root cause analysis and updated our blog post with additional information -> https://www.backblaze.com/blog/privacy-update-third-party-tracking/ https://www.backblaze.com/blog/privacy-update-third-party-tr....
- pdimitar 6y agoI've been considering using Backblaze for both personal and company needs -- and we're talking 50+ TB here -- but this incident made me reconsider. I'd still use Backblaze but that's VERY dependent on how do you handle this. Just saying "we fixed it" doesn't answer the much more fundamental question of "what is the FB tracking pixel doing in a privacy-critical page in the first place?". Please, do a thorough post-mortem analysis and publish it. Looking at the comments here, this could mean you get or lose the business of many.
- atYevP 6y agoYev here -> thank you for being a customer. Wanted to let you know that we've finished our root cause analysis and have updated our blog post with additional information -> https://www.backblaze.com/blog/privacy-update-third-party-tracking/ https://www.backblaze.com/blog/privacy-update-third-party-tr....
- bogomipz 6y agoWill you be pushing out a fix to address the complete lack of any process that should have flagged and prevented this from happening in the first place? The quick fix I am pushing out for my clients who use Backblaze is moving them to another backup provider.
- atYevP 6y agoYev here with a brief update on the fix that was pushed out - we removed the offending code from the logged in web pages. We will continue to investigate and provide updates as we have them.
- atYevP 6y agoYev here -> we've published a blog post that has our findings thus far, we're still investigating and will be updating that post: https://www.backblaze.com/blog/privacy-update-third-party-tracking/ https://www.backblaze.com/blog/privacy-update-third-party-tr....
- petee 6y agoAnd you were planning on emailing the rest of your customers? Finding out because of a Tuesday blog post on HN is a bad look for your company...