4 ms·

I wish that more companies would provide information on their disk fleets like this—especially on SSDs, which would be much more useful information for everyday

I wish that more companies would provide information on their disk fleets like this—especially on SSDs, which would be much more useful information for everyday consumers. I also wish that Backblaze would focus on their product security[0] as much as they focus on promoting this quarterly hard drive report. [0] https://twitter.com/zetafleet/status/1304664097989054464 https://twitter.com/zetafleet/status/1304664097989054464

bdd 6y ago
Publishing SSD reliability of datacenter setups would yield no useful information for everyday consumers. See my comment at https://news.ycombinator.com/item?id=24840914 https://news.ycombinator.com/item?id=24840914. Neither load nor the operating environment of such reports will be anything remotely similar to everyday consumers' use and environments. In 2015 Facebook published* its study of flash memory failures at scale. By all means it's a terrific paper (of course I'm biased) but I think it was irrelevant not only for consumers but pretty much 99% of the flash users in the industry. [*] http://users.ece.cmu.edu/~omutlu/pub/flash-memory-failures-in-the-field-at-facebook_sigmetrics15.pdf http://users.ece.cmu.edu/~omutlu/pub/flash-memory-failures-i...
- csnover 6y ago
  Sorry, I was not clear. When I said company disk fleets, I was thinking of places like corporate head offices which have thousands of workstations, not datacentres.
  - bdd 6y ago
    Ah. That'd actually be useful I think.
  - greedo 6y ago
    Lol, that's assuming your workstations are deployed with SSDs. My company thinks they're a waste of money for the plebes.
- ksec 6y ago
  Well I am under the impression most DC would be operating in ideal condition compare to consumer usage. So it does provide some useful numbers in the grand overview.
outworlder 6y ago
That tweet is a big deal if true. Were there any follow-ups from Backblaze?
- csnover 6y ago
  > That tweet is a big deal if true. One thing you can independently verify is that the backdoor string is still in their client. The installer starts transmitting data during the installation process, so I would not recommend installing it outside of a VM—just look at the files directly. The macOS installer has it in `Backblaze Installer.app/Contents/Resources/instfiles.zip/bztransmit`. The Windows installer is a self-extracting ZIP file, so just use unzip and look in `bztransmit.exe` and `bztransmit64.exe`. $ strings bztransmit |grep BACKDOOR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file exists: ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file could not be read: ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file existed but less than 10 chars or could not be read: ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file did not contain bz_cvt: ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file contained bz_cvt but wrong num digits: ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file did not contain bz_upload_url: ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file contained bz_upload_url but did not start with http: DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file exists and is valid and bz_cvt= DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml SUCCESSFULLY_swapped_in new_bz_cvt= What does the corresponding code do? I genuinely don’t know. My goal was to find backup software, not to do a security analysis. An easy-to-exploit root code execution vulnerability was enough for me to uninstall the software, submit a report as a professional courtesy, and go do something else. Clearly it’s dumb to put the word BACKDOOR in your code if your goal is to plant a secret backdoor, but it’s also pretty dumb to use world-writable directories, disable host certificate verification, use magic hard-coded strings to “sign” updates, and implement data encryption in a way which requires the ‘private’ password to be sent to the server, so who knows. As I said in the tweet, even if it turns out to be innocuous, the optics are terrible and show a serious lack of good judgement on their part, especially given how much they claim to be security experts who care about their reputation[0]. > Were there any follow-ups from Backblaze? No. The only “follow-up”, as it were, was to cancel their HackerOne public bug bounty programme. (Though this was a month ago, their web site still tells people to “visit our public bug bounty program managed through Hacker One”[2].) They have not communicated with me at all except for one tweet about that change[1]. I have seen no public statement from them acknowledging that this happened, or that they made mistakes, or that they have steps they plan to take to improve their internal software development practices. [0] “We stand by our reputation as trustworthy, careful programmers who have worked in the security field for over a decade. […] we have LOTS of interest in keeping our reputations rock solid and utterly clean.” https://help.backblaze.com/hc/en-us/articles/217664798-Security-Question-Round-up- https://help.backblaze.com/hc/en-us/articles/217664798-Secur... [1] https://twitter.com/backblaze/status/1308157606368882688 https://twitter.com/backblaze/status/1308157606368882688 [2] https://www.backblaze.com/security.html https://www.backblaze.com/security.html
atYevP 6y ago
Yev here -> Hey there! We've just signed with a new program provider and hope to be rolling out the private program in early November with a public rollout to follow. For now though we ask folks to reach out to bounty@backblaze.com directly with any bugs they find so we can track them internally!

bdd 6y ago

Publishing SSD reliability of datacenter setups would yield no useful information for everyday consumers. See my comment at https://news.ycombinator.com/item?id=24840914 https://news.ycombinator.com/item?id=24840914. Neither load nor the operating environment of such reports will be anything remotely similar to everyday consumers' use and environments. In 2015 Facebook published* its study of flash memory failures at scale. By all means it's a terrific paper (of course I'm biased) but I think it was irrelevant not only for consumers but pretty much 99% of the flash users in the industry. [*] http://users.ece.cmu.edu/~omutlu/pub/flash-memory-failures-in-the-field-at-facebook_sigmetrics15.pdf http://users.ece.cmu.edu/~omutlu/pub/flash-memory-failures-i...

csnover 6y ago
Sorry, I was not clear. When I said company disk fleets, I was thinking of places like corporate head offices which have thousands of workstations, not datacentres.
- bdd 6y ago
  Ah. That'd actually be useful I think.
- greedo 6y ago
  Lol, that's assuming your workstations are deployed with SSDs. My company thinks they're a waste of money for the plebes.
ksec 6y ago
Well I am under the impression most DC would be operating in ideal condition compare to consumer usage. So it does provide some useful numbers in the grand overview.

outworlder 6y ago

That tweet is a big deal if true. Were there any follow-ups from Backblaze?

csnover 6y ago
> That tweet is a big deal if true. One thing you can independently verify is that the backdoor string is still in their client. The installer starts transmitting data during the installation process, so I would not recommend installing it outside of a VM—just look at the files directly. The macOS installer has it in `Backblaze Installer.app/Contents/Resources/instfiles.zip/bztransmit`. The Windows installer is a self-extracting ZIP file, so just use unzip and look in `bztransmit.exe` and `bztransmit64.exe`. $ strings bztransmit |grep BACKDOOR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file exists: ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file could not be read: ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file existed but less than 10 chars or could not be read: ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file did not contain bz_cvt: ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file contained bz_cvt but wrong num digits: ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file did not contain bz_upload_url: ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file contained bz_upload_url but did not start with http: DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file exists and is valid and bz_cvt= DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml SUCCESSFULLY_swapped_in new_bz_cvt= What does the corresponding code do? I genuinely don’t know. My goal was to find backup software, not to do a security analysis. An easy-to-exploit root code execution vulnerability was enough for me to uninstall the software, submit a report as a professional courtesy, and go do something else. Clearly it’s dumb to put the word BACKDOOR in your code if your goal is to plant a secret backdoor, but it’s also pretty dumb to use world-writable directories, disable host certificate verification, use magic hard-coded strings to “sign” updates, and implement data encryption in a way which requires the ‘private’ password to be sent to the server, so who knows. As I said in the tweet, even if it turns out to be innocuous, the optics are terrible and show a serious lack of good judgement on their part, especially given how much they claim to be security experts who care about their reputation[0]. > Were there any follow-ups from Backblaze? No. The only “follow-up”, as it were, was to cancel their HackerOne public bug bounty programme. (Though this was a month ago, their web site still tells people to “visit our public bug bounty program managed through Hacker One”[2].) They have not communicated with me at all except for one tweet about that change[1]. I have seen no public statement from them acknowledging that this happened, or that they made mistakes, or that they have steps they plan to take to improve their internal software development practices. [0] “We stand by our reputation as trustworthy, careful programmers who have worked in the security field for over a decade. […] we have LOTS of interest in keeping our reputations rock solid and utterly clean.” https://help.backblaze.com/hc/en-us/articles/217664798-Security-Question-Round-up- https://help.backblaze.com/hc/en-us/articles/217664798-Secur... [1] https://twitter.com/backblaze/status/1308157606368882688 https://twitter.com/backblaze/status/1308157606368882688 [2] https://www.backblaze.com/security.html https://www.backblaze.com/security.html

atYevP 6y ago

Yev here -> Hey there! We've just signed with a new program provider and hope to be rolling out the private program in early November with a public rollout to follow. For now though we ask folks to reach out to bounty@backblaze.com directly with any bugs they find so we can track them internally!