6 ms·

Kudos to BB for their swift actions. Under no situation should any software randomly delete folders and files it doesn't own, period, specially something so wi

Kudos to BB for their swift actions. Under no situation should any software randomly delete folders and files it doesn't own, period, specially something so widely used by professionals. This is not just a little bug on a rarely used OS platform not limited to people using backblaze on the Mac, it should have been a show-stopper, and should have never left the test environment (do they even have one?). It reflects on the abysmal quality of Adobe's software and their QA. There's no reason Adobe should even touch the root directory "/" on the Mac, leave alone delete anything from there. Why does Adobe CC need root permissions to install on the Mac in the first place? I hope the people responsible for this disaster, their managers and the QA team were fired. In my business, this sort of mess can sink our company.

voltagex_ 11y ago
Adobe is the company who once thought modifying the boot sector of a hard drive was OK [1]. I'm just glad they didn't migrate to messing around with EFI variables for their copy protection. 1: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/441941/comments/164 https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/441941/...
- gnoway 11y ago
  FWIW FlexNet is not an Adobe product, it's by Flexera[0] (Acresso at the time, prior owner Macrovision). It is/was used by a lot of products for license enforcement. I wasn't personally aware of the boot sector crap until now though. Doesn't really excuse Adobe. [0] https://en.wikipedia.org/wiki/FlexNet_Publisher https://en.wikipedia.org/wiki/FlexNet_Publisher
  - bonestamp2 11y ago
    I've implemented FlexNet licensing in some of our software and my first thought when I read about this adobe bug was that it was an error in the FlexNet licensing implementation since it likes to just own a folder and wipe its contents before refreshing licensing info from the server (which would make sense after an install/update). The only thing I could find from Adobe was the basic release notes, which really don't say much. Has anyone seen more details on how this happened? If it is licensing, I doubt they'd admit it.
    - vsl 11y ago
      Messing around in /? Pretty certain it is licensing, putting information in obscure places most users won't look for them.
StavrosK 11y ago
It sounds like it touched the homedir, not the root dir.
viperscape 11y ago
I saw something interesting last week where an upgraded osx to el cap screwed up the adobe fodder permissions lib. Adobe's suggested fix was to give full access to everyone. I don't suggest upgrading to el capitan, clean install only. I also suggest people use standard accounts, and authenticate updates and installs with a separate admin account. I'm eager to see what the real reasoning behind the bug was, interesting programmer error using osx API I imagine.
michaelt 11y ago
It reflects on the abysmal quality of Adobe's software and their QA. [...] I hope the people responsible for this disaster, their managers and the QA team were fired. Eh, I can kind of see how it might happen myself. Remember that time Valve had a bug in Steam for Linux that would wipe the user's hard drive? [1] It's easy to imagine how this sort of thing would happen - manual test guys don't have hidden folders in their root directories because their boxes are locked down by corporate IT and they can't create any folders in the root directory. Developers don't have them because they understand the unix way and know they shouldn't be putting files there. CI servers don't have such folders because, well, why would they? None of the test environments include the specific condition needed to trigger this bug - a hidden folder in the root directory - so the bug never appears in testing. Is it bad and serious? Absolutely. But I don't think the fact testing missed this is a sign of testing incompetence - you could have cutting edge testing following every industry best practice and still miss this bug. [1] https://github.com/ValveSoftware/steam-for-linux/issues/3671 https://github.com/ValveSoftware/steam-for-linux/issues/3671
- danso 11y ago
  The Steam bug was the first thing that popped into my mind...But does application testing have to be so dependent on testers having all possible permutations of a system configuration? I mean, I can't offer a mathematical proof for this, but it seems to me possible to write code that pre-empts the situation where an arbitrary directory is blown away, willy-nilly. In the same way that while it's hard to effectively blacklist all the dangers, it's still possible to effectively implement security by whitelisting allowed behaviors.
  - cowsandmilk 11y ago
    For the Steam bug, it could have been fixed by running the shell script with options that prevent use of undefined variables. It really isn't hard to have a company policy about nounset, errexit, and, in the extreme, noclobber.
    - stephengillie 11y ago
danieldk 11y ago
There's no reason Adobe should even touch the root directory "/" on the Mac, There is no reason why an application should touch anything outside its own sandbox (unless requested by a user via a open/save dialog). Incidents like this show why developers should distribute their Mac applications sandboxed by default.
dgacmu 11y ago
> I hope the people responsible for this disaster, their managers and the QA team were fired. In my business, this sort of mess can sink our company. I hope something very different: I hope they're required to write a thorough post-mortem analysis of what went wrong, and what in their development & testing process led to the failure. And then required to implement improvements to both that would prevent this class of bugs from happening again. Firing people for a mistake is rarely a good idea. It makes them not want to admit mistakes. Firing them for making the same mistake several times and not learning for it -- fire away. c.f., http://itrevolution.com/uncovering-the-devops-improvement-principles-behind-google-randy-shoup-interview/ http://itrevolution.com/uncovering-the-devops-improvement-pr... and the "blameless post-mortem"
- atYevP 11y ago
  Yev from Backblaze here -> That would be cool, though there's no real "requiring body" to require post-mortems. The company has to want them. We do them ALL the time internally at Backblaze, they help us piece together what the hell went wrong and gives us next steps to work on. We find it really useful, especially with software/hardware issues (less so UI glitches and stuff).
- x0x0 11y ago
  When you find yourself writing code that deletes directories, you need to be extraordinarily careful. It should be subject not only to extensive testing, but a design review that questions if there isn't a better way to achieve your goal because of the many ways code that runs rm -rf can go wrong. If code like this escapes code review and testing and wacks actual customer data, firing is an appropriate response.
  - imron 11y ago
    In the code I've written that deletes directories, I never call the system level delete functions directly, and instead have abstracted them away to a function with added checks to the beginning that fail/assert with big warning messages if parameters ever try to delete root or other system directories. It's never been triggered except by my unit tests, but I'm sure glad it's there.
deleted 11y ago
[deleted]
vlod 11y ago
Is there any kind of docker-like container solution for macosx? In a docker-like world, this wouldn't happen would it? An installed application (in a container) wouldn't have access to delete in a willy–nilly manner.
arjie 11y ago
> I hope the people responsible for this disaster, their managers and the QA team were fired. This is a actually a rather typical response, made with full awareness of the consequences of an action by people not under the same constraints as the participants in an incident. Diane Vaughan (an expert on risk, causes of incidents, and how organizations come to become error-prone) terms these actions 'ritualistic': > Causes must be identified, and in order to move forward, the organization must appear to have resolved the problem. The direct benefit of identifying the cause of an accident as Operator Error is that the individual operator is the target of change. The responsible party can be transferred, demoted, retrained, or fired, all of which obscure flaws in the organization that may have contributed to the mistakes made by the individual who is making judgments about risk. The dark side of this ritualistic practice is that organizational and institutional sources of risk and error are not systematically subject either to investigation or other technologies of control. To make it even better, you're now throwing away all the people most experienced in the conditions that lead to a specific incident. Unless very specific conditions are true, this is a sub-optimal recommendation. The evidence seems to be against building a culture of fear.
No1 11y ago
> I hope the people responsible for this disaster, their managers and the QA team were fired. Yes, call out the lynch mob. Coincidentally, this bug was released a day before a holiday weekend. It's practically guaranteed that people were crunched on their deadlines and faced with either a) working all weekend b) getting reprimanded for failing to meet their deadlines or c) calling it good, releasing with inadequate review, and enjoying the long weekend (or so they thought). Firing minions and writing technical post-mortems won't help with creating a sane work environment.

voltagex_ 11y ago

Adobe is the company who once thought modifying the boot sector of a hard drive was OK [1]. I'm just glad they didn't migrate to messing around with EFI variables for their copy protection. 1: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/441941/comments/164 https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/441941/...

gnoway 11y ago
FWIW FlexNet is not an Adobe product, it's by Flexera[0] (Acresso at the time, prior owner Macrovision). It is/was used by a lot of products for license enforcement. I wasn't personally aware of the boot sector crap until now though. Doesn't really excuse Adobe. [0] https://en.wikipedia.org/wiki/FlexNet_Publisher https://en.wikipedia.org/wiki/FlexNet_Publisher
- bonestamp2 11y ago
  I've implemented FlexNet licensing in some of our software and my first thought when I read about this adobe bug was that it was an error in the FlexNet licensing implementation since it likes to just own a folder and wipe its contents before refreshing licensing info from the server (which would make sense after an install/update). The only thing I could find from Adobe was the basic release notes, which really don't say much. Has anyone seen more details on how this happened? If it is licensing, I doubt they'd admit it.
  - vsl 11y ago
    Messing around in /? Pretty certain it is licensing, putting information in obscure places most users won't look for them.

StavrosK 11y ago

It sounds like it touched the homedir, not the root dir.

viperscape 11y ago

I saw something interesting last week where an upgraded osx to el cap screwed up the adobe fodder permissions lib. Adobe's suggested fix was to give full access to everyone. I don't suggest upgrading to el capitan, clean install only. I also suggest people use standard accounts, and authenticate updates and installs with a separate admin account. I'm eager to see what the real reasoning behind the bug was, interesting programmer error using osx API I imagine.

michaelt 11y ago

It reflects on the abysmal quality of Adobe's software and their QA. [...] I hope the people responsible for this disaster, their managers and the QA team were fired. Eh, I can kind of see how it might happen myself. Remember that time Valve had a bug in Steam for Linux that would wipe the user's hard drive? [1] It's easy to imagine how this sort of thing would happen - manual test guys don't have hidden folders in their root directories because their boxes are locked down by corporate IT and they can't create any folders in the root directory. Developers don't have them because they understand the unix way and know they shouldn't be putting files there. CI servers don't have such folders because, well, why would they? None of the test environments include the specific condition needed to trigger this bug - a hidden folder in the root directory - so the bug never appears in testing. Is it bad and serious? Absolutely. But I don't think the fact testing missed this is a sign of testing incompetence - you could have cutting edge testing following every industry best practice and still miss this bug. [1] https://github.com/ValveSoftware/steam-for-linux/issues/3671 https://github.com/ValveSoftware/steam-for-linux/issues/3671

danso 11y ago
The Steam bug was the first thing that popped into my mind...But does application testing have to be so dependent on testers having all possible permutations of a system configuration? I mean, I can't offer a mathematical proof for this, but it seems to me possible to write code that pre-empts the situation where an arbitrary directory is blown away, willy-nilly. In the same way that while it's hard to effectively blacklist all the dangers, it's still possible to effectively implement security by whitelisting allowed behaviors.
- cowsandmilk 11y ago
  For the Steam bug, it could have been fixed by running the shell script with options that prevent use of undefined variables. It really isn't hard to have a company policy about nounset, errexit, and, in the extreme, noclobber.
  - stephengillie 11y ago

danieldk 11y ago

There's no reason Adobe should even touch the root directory "/" on the Mac, There is no reason why an application should touch anything outside its own sandbox (unless requested by a user via a open/save dialog). Incidents like this show why developers should distribute their Mac applications sandboxed by default.

dgacmu 11y ago

> I hope the people responsible for this disaster, their managers and the QA team were fired. In my business, this sort of mess can sink our company. I hope something very different: I hope they're required to write a thorough post-mortem analysis of what went wrong, and what in their development & testing process led to the failure. And then required to implement improvements to both that would prevent this class of bugs from happening again. Firing people for a mistake is rarely a good idea. It makes them not want to admit mistakes. Firing them for making the same mistake several times and not learning for it -- fire away. c.f., http://itrevolution.com/uncovering-the-devops-improvement-principles-behind-google-randy-shoup-interview/ http://itrevolution.com/uncovering-the-devops-improvement-pr... and the "blameless post-mortem"

atYevP 11y ago
Yev from Backblaze here -> That would be cool, though there's no real "requiring body" to require post-mortems. The company has to want them. We do them ALL the time internally at Backblaze, they help us piece together what the hell went wrong and gives us next steps to work on. We find it really useful, especially with software/hardware issues (less so UI glitches and stuff).
x0x0 11y ago
When you find yourself writing code that deletes directories, you need to be extraordinarily careful. It should be subject not only to extensive testing, but a design review that questions if there isn't a better way to achieve your goal because of the many ways code that runs rm -rf can go wrong. If code like this escapes code review and testing and wacks actual customer data, firing is an appropriate response.
- imron 11y ago
  In the code I've written that deletes directories, I never call the system level delete functions directly, and instead have abstracted them away to a function with added checks to the beginning that fail/assert with big warning messages if parameters ever try to delete root or other system directories. It's never been triggered except by my unit tests, but I'm sure glad it's there.

deleted 11y ago

[deleted]

vlod 11y ago

Is there any kind of docker-like container solution for macosx? In a docker-like world, this wouldn't happen would it? An installed application (in a container) wouldn't have access to delete in a willy–nilly manner.

arjie 11y ago

> I hope the people responsible for this disaster, their managers and the QA team were fired. This is a actually a rather typical response, made with full awareness of the consequences of an action by people not under the same constraints as the participants in an incident. Diane Vaughan (an expert on risk, causes of incidents, and how organizations come to become error-prone) terms these actions 'ritualistic': > Causes must be identified, and in order to move forward, the organization must appear to have resolved the problem. The direct benefit of identifying the cause of an accident as Operator Error is that the individual operator is the target of change. The responsible party can be transferred, demoted, retrained, or fired, all of which obscure flaws in the organization that may have contributed to the mistakes made by the individual who is making judgments about risk. The dark side of this ritualistic practice is that organizational and institutional sources of risk and error are not systematically subject either to investigation or other technologies of control. To make it even better, you're now throwing away all the people most experienced in the conditions that lead to a specific incident. Unless very specific conditions are true, this is a sub-optimal recommendation. The evidence seems to be against building a culture of fear.

No1 11y ago

> I hope the people responsible for this disaster, their managers and the QA team were fired. Yes, call out the lynch mob. Coincidentally, this bug was released a day before a holiday weekend. It's practically guaranteed that people were crunched on their deadlines and faced with either a) working all weekend b) getting reprimanded for failing to meet their deadlines or c) calling it good, releasing with inadequate review, and enjoying the long weekend (or so they thought). Firing minions and writing technical post-mortems won't help with creating a sane work environment.