8 ms·

I can only imagine the amount of bug reports, real and false, that a company of Apple's size must receive on a daily basis. Is there any company at that scale t

by jm20 8y ago

I can only imagine the amount of bug reports, real and false, that a company of Apple's size must receive on a daily basis. Is there any company at that scale that can reliably filter through all of them to find actual, critical bugs quickly? It simply isn't as easy as saying 'flag all reports with 'security vulnerability' in the submission for priority.' That could still be thousands of reports in the 'priority' queue, most of which some person would need to manually investigate one by one.

jonathanberger 8y ago
It's not hard to reach a living person who works at Apple. The next step should have been reproducing it for that person. It's not clear that the finders in this story did that - it seems they might have been trying to find out about bounty payments first. If any tech support or Apple Genius were to have seen this bug reproduced, it should have immediately been easy to flag to the right person.
- dewey 8y ago
  What if you don’t live near an Apple Store like a lot of people do? Saying that it’s not hard for a person to get in contact with someone working at Apple seems like a pretty biased view.
- gpvos 8y ago
  "Looking for a bounty?" Where did you get that from?
  - justtopost 8y ago
    Bug Bounty I would assume, for finding the error.
    - gpvos 8y ago
      Of course. However: I didn't re-read the article, but I don't remember that it suggested anything of the sort, so GGP just projected that onto them.
kerng 8y ago
There are a couple odd gaps that seem to be been in place at Apple. First, Apple did not even acknowledge the report to the initial finder. This is flawed, like Google, Microsoft and other big players acknowledge receipt. Secondly, the person tried to go beyond after not hearing anything, by calling, faxing and other means to no avail. Third, Apple needs to staff their responders better it seems. Most issues can be filtered out quickly to be left with the few interesting ones. The repro here doesnt need any technical knowledge! Many companies outsource the initial triage process even to have it scale when needed. So, one take away for Apple is to improve their response process and transparency. Transparency they have always lacked when it comes to security.
- duxup 8y ago
  >First, Apple did not even acknowledge the report to the initial finder. That is strange. I worked with some folks who interacted with researchers and their reports. Half the battle with them was getting back to the researcher and getting their cooperation about keeping quiet + assuring them they're working on it, and working with them if additional data is needed. And at the same time fending off internal folks who have poor instincts and want to push back against, blame, or even punish the researcher (this was surprisingly common at companies who even should know better). It's not the hardest part but developing that trust can be a big difference between a possible PR nightmare or not, and the initial contacts are a big deal. Also researcher's who you get along with sometimes come back with better data, additional bugs,etc.
- olliej 8y ago
  How long is reasonable? Consider the number of incoming bugs, etc
  - kerng 8y ago
    With a standard case management system this can (and is at other companies) automated - so maybe an hour max for an acknowledgement that a case got created? It sounds like Apple might be doing do this all via email, no case management software. That would be pretty bad.
why_only_15 8y ago
There's a guy on my team (of 12 people) in Apple that literally does nothing but screen bugs. I think he screened 100 bugs last week and there are still hundreds more unscreened. It's really hard to keep track of.
- saagarjha 8y ago
  There's literally a team of people that send bugs to the appropriate group, where there are other people to do secondary screening…
- pbhjpbhj 8y ago
  Other people are saying "Apple must get thousands of reports of dubious quality". If that's true I'd expect O(100) reports screened per day? I guess automated screening roots out lots of them?
  - why_only_15 8y ago
    This is for one feature that's not even that big - this is not just randomly screening bugs, this is screening all the bugs that get assigned to us.
rgovostes 8y ago
Apple has a dedicated team that triages incoming security vulnerability reports. If you search for how to report a security issue to Apple, you would find their e-mail address. The weird thing to me is that the NYT article says that she tried "faxing Apple’s security team." Having some familiarity with the team and the process of reporting security vulnerabilities to them, I do not recall them ever claiming to have a fax machine. The idea of Apple asking you to fax in a bug report is ludicrous.
- olliej 8y ago
  For it to reach that screening step it has to be tagged as security bug. Do you really think a consumer would think to do that? Also you have to consider the number of bugs that are incorrectly flagged as being security.
  - saagarjha 8y ago
    I'm sure reports to product-security@apple.com get immediately flagged as "security".
    - bostik 8y ago
      Knowing how much outright spam a security email address gets... (luckily spam filtering is good enough to not surface them but a human still has to periodically go through the spams just to ensure there were no false positives) Even then, the modern day incentives around vulnerability disclosure are not helping. Because security bugs are awarded bounties based on their severity, every single reporter has a financial incentive to hype and inflate their findings. "URGENT" this, "CRITICAL" that, "ACCOUNT TAKEOVER" due to already compromised computer/device, you name it. Teams without sufficient resources will spend a lot of time dealing with the maladjusted severities. And yes, I believe the "mal-" prefix is warranted. If your report does go through with inflated severity, you stand to make more money. I am starting to think that a reasonably run bounty programme should state up front that inflated severities in bug reports will reduce their payouts.
    - olliej 8y ago
sgentle 8y ago
If you are able to perform the following steps for any of Amazon, Google, Facebook, Netflix, Microsoft or Twitter, I will literally eat a hat (you may choose what kind): 1. Discover an easily exploitable vulnerability that allows access to a chosen user's private data 2. Email their security address about it 3. Tweet at them about it 4. Fax them about it 5. A week later the vulnerability is still exploitable You do not have to play fair. You're allowed to impersonate a suburban mom or a grandpa who's not good with technology. You're allowed to ramble or use vague and non-technical terms as long as a reasonably qualified person could determine what the vulnerability is. Specifically, it is not required that you include relevant product versions, steps to reproduce, a full reproduction video, or a one-sentence impact summary like "a caller can eavesdrop on the recipient of a Group Facetime call without their knowledge or consent". There's no apologising this away. The vulnerability was already a monumental fuckup, but this detail propels it into the realm of cultural dysfunction. It should not be possible to fail this badly. If you put listening devices in people's pockets, you need to hold yourself to a higher standard than "I dunno, bug reporting is hard".
- doctorpangloss 8y ago
  Well, pointing out that tech company engineers aren’t really all the great, indeed even below average, threatens like, one of HN’s biggest orthodoxies: that the people who read and write in this forum are brilliant.
- sec-throwaway 8y ago
  I work for a large software company. I am not on our security team, but have worked with our security team to investigate and resolve reported issues. I agree that it should not be possible to fail this badly. Even nonsensical reports to our security address will, with an SLA measured in hours, be read by a qualified human who will then reply to at least say "we're looking into it". A credible report will be immediately escalated to someone with relevant domain expertise for investigation. The security engineer will attempt to reproduce. A confirmed report will be escalated to an executive, who will determine urgency. For issues like this, where sensitive data is exposed, people will be woken up and several things will happen in parallel: the scope of the issue will be assessed, the root cause will be found, potential workarounds will be identified, a fix will be implemented, the potential existence of related issues will be investigated, and the reporter will be contacted to assess disclosure risk. Even in the worst case, where a complicated vulnerability exists in multiple versions of multiple products, requiring multiple patches and backports and requiring coordinated disclosure with partners, I'd expect a fix to be in customers' hands within 14 days.
philip1209 8y ago
What would happen if you submitted the bug via certified mail? How would that get triaged?
cm2187 8y ago
Also when a user reports a major flaw in your product and your response is that you will not lift a finger until the user fills the correct form, you have reached the state of a useless bureaucracy, completely unconcerned about the quality of your product. I remember being given that advice by an employee on the vendor’s own forum where I reported a problem. They are still waiting for their bug report and I have found an alternative product.
- benologist 8y ago
  I sent Stripe a bug report once and they replied I had to sign in on their website to email them.
- DeathRay2K 8y ago
  I found a bug on the PayPal site (I can't associate a new email address). I tried the contact forms in their app and on the website, both resulted in error messages. I tried emailing them, but got an automatic message back saying that they don't monitor emails, instead directing me to the support form on their website, which you have to log in to access. And as I mentioned, that form just ends in an error. So it seems like a way more common problem than it should be.
huffmsa 8y ago
If this were their only QA/QC issue, then fine, it might be excusable. But it's this, broken MBP keyboards, broken MBP hinges, bent iPads, the MBP core-i9 thermal issue and probably a few I'm forgetting. Nintendo doesn't have these kinds of failures, they're pretty analogous to Apple. Hardware and software. What's the difference? Nintendo cares and Apple doesn't. (Not to say Nintendo is issue free by any means). Maybe they should put some of that $250b in cash to work upgrading their processes.
- javagram 8y ago
  Nintendo has different priorities. They use plastic screens instead of glass for instance. Their focus is on building cheap but durable goods, e.g. plastic scratches more easily but is less likely to shatter catastrophically. They have definitely had their own industrial design failures - the Wii would sometimes overheat in sleep mode, the 3DS would scratch its own screen when placed under pressure in a pocket, the Switch supposedly can have its screen scratched by a bent dock. The initial run of switch controllers had an issue where the left side controller could disconnect very easily. Of course when your product is $149 or $299 replacing it or living with a scratch doesn’t hurt as much...
  - huffmsa 8y ago
    Like I said, Nintendo is not perfect. But they at least try. Apple frankly doesn't appear to give a damn about shipping broken devices / software right now. And they have the resources to easily give a damn
    - javagram 8y ago
      Nintendo gets criticized pretty heavily for their decisions to do stuff like use plastic. Their devices are frequently described as “toys” on gaming forums that I visit due to this. Apple has an entirely different brand they need to maintain which is why they use metal, make their devices thin, have edge to edge screens, etc. Personally I have no problems with “toy” style design but I think a lot of apple’s premium is based on their sense of style and the feeling people have that iPhones, iPads, Macs are a “premium” product due to the industrial design. As an old school Mac user from back when Macs were beige boxes I personally don’t care about that stuff.
deleted 8y ago
[deleted]
jssmith 8y ago
One startup that’s working on this is unitQ: https://www.unitq.com https://www.unitq.com
bleriot 8y ago
Yes, a company at that scale could do it, but they’d have to actually spend some of their $200 billion instead of hoarding it Scrooge McDuck style.
- Bud 8y ago
  This is amusing to type, I guess, but not really true. Apple spent a huge amount last year: on stock buybacks (sigh), on building an enormous TV/film production studio from scratch, on a huge effort to develop self-driving cars, etc., etc. If Apple were doing anything remotely similar to "hoarding", they'd have more like $700 billion in cash right now.
tialaramex 8y ago
Raymond Chen (at Microsoft), for example has written repeatedly about incidents he spent lots of time investigating where your first instinct is "That's not a bug" and the eventual outcome was "Yup, that's not a bug" but it was a security ticket and so Raymond doggedly chased down every aspect to make sure it isn't a problem. He mostly invokes the Hitchiker's Guide quote "It rather involved being on the other side of this airtight hatchway" to suggest that often the problem in these bugs is that they have a step where you've legitimately got privileges, and then they use those privileges to... do something you legitimately need privileges for. That's not a bug, if you try it without the privileges it doesn't work, but Raymond has to walk through all the steps seeing whether anything surprising is going on. Now, whether engineers at a corp actually get given the space to do this stuff is an executive policy decision. Maybe at Apple not enough of them do, I can't say. But if it doesn't get done you are sooner or later going to miss cases where it _sounds_ like it's not a bug but actually there's a serious bug if you looked closely.

by jm20 8y ago

jonathanberger 8y ago

It's not hard to reach a living person who works at Apple. The next step should have been reproducing it for that person. It's not clear that the finders in this story did that - it seems they might have been trying to find out about bounty payments first. If any tech support or Apple Genius were to have seen this bug reproduced, it should have immediately been easy to flag to the right person.

dewey 8y ago
What if you don’t live near an Apple Store like a lot of people do? Saying that it’s not hard for a person to get in contact with someone working at Apple seems like a pretty biased view.
gpvos 8y ago
"Looking for a bounty?" Where did you get that from?
- justtopost 8y ago
  Bug Bounty I would assume, for finding the error.
  - gpvos 8y ago
    Of course. However: I didn't re-read the article, but I don't remember that it suggested anything of the sort, so GGP just projected that onto them.

kerng 8y ago

There are a couple odd gaps that seem to be been in place at Apple. First, Apple did not even acknowledge the report to the initial finder. This is flawed, like Google, Microsoft and other big players acknowledge receipt. Secondly, the person tried to go beyond after not hearing anything, by calling, faxing and other means to no avail. Third, Apple needs to staff their responders better it seems. Most issues can be filtered out quickly to be left with the few interesting ones. The repro here doesnt need any technical knowledge! Many companies outsource the initial triage process even to have it scale when needed. So, one take away for Apple is to improve their response process and transparency. Transparency they have always lacked when it comes to security.

duxup 8y ago
>First, Apple did not even acknowledge the report to the initial finder. That is strange. I worked with some folks who interacted with researchers and their reports. Half the battle with them was getting back to the researcher and getting their cooperation about keeping quiet + assuring them they're working on it, and working with them if additional data is needed. And at the same time fending off internal folks who have poor instincts and want to push back against, blame, or even punish the researcher (this was surprisingly common at companies who even should know better). It's not the hardest part but developing that trust can be a big difference between a possible PR nightmare or not, and the initial contacts are a big deal. Also researcher's who you get along with sometimes come back with better data, additional bugs,etc.
olliej 8y ago
How long is reasonable? Consider the number of incoming bugs, etc
- kerng 8y ago
  With a standard case management system this can (and is at other companies) automated - so maybe an hour max for an acknowledgement that a case got created? It sounds like Apple might be doing do this all via email, no case management software. That would be pretty bad.

why_only_15 8y ago

There's a guy on my team (of 12 people) in Apple that literally does nothing but screen bugs. I think he screened 100 bugs last week and there are still hundreds more unscreened. It's really hard to keep track of.

saagarjha 8y ago
There's literally a team of people that send bugs to the appropriate group, where there are other people to do secondary screening…
pbhjpbhj 8y ago
Other people are saying "Apple must get thousands of reports of dubious quality". If that's true I'd expect O(100) reports screened per day? I guess automated screening roots out lots of them?
- why_only_15 8y ago
  This is for one feature that's not even that big - this is not just randomly screening bugs, this is screening all the bugs that get assigned to us.

rgovostes 8y ago

Apple has a dedicated team that triages incoming security vulnerability reports. If you search for how to report a security issue to Apple, you would find their e-mail address. The weird thing to me is that the NYT article says that she tried "faxing Apple’s security team." Having some familiarity with the team and the process of reporting security vulnerabilities to them, I do not recall them ever claiming to have a fax machine. The idea of Apple asking you to fax in a bug report is ludicrous.

olliej 8y ago
For it to reach that screening step it has to be tagged as security bug. Do you really think a consumer would think to do that? Also you have to consider the number of bugs that are incorrectly flagged as being security.
- saagarjha 8y ago
  I'm sure reports to product-security@apple.com get immediately flagged as "security".
  - bostik 8y ago
    Knowing how much outright spam a security email address gets... (luckily spam filtering is good enough to not surface them but a human still has to periodically go through the spams just to ensure there were no false positives) Even then, the modern day incentives around vulnerability disclosure are not helping. Because security bugs are awarded bounties based on their severity, every single reporter has a financial incentive to hype and inflate their findings. "URGENT" this, "CRITICAL" that, "ACCOUNT TAKEOVER" due to already compromised computer/device, you name it. Teams without sufficient resources will spend a lot of time dealing with the maladjusted severities. And yes, I believe the "mal-" prefix is warranted. If your report does go through with inflated severity, you stand to make more money. I am starting to think that a reasonably run bounty programme should state up front that inflated severities in bug reports will reduce their payouts.
  - olliej 8y ago

sgentle 8y ago

If you are able to perform the following steps for any of Amazon, Google, Facebook, Netflix, Microsoft or Twitter, I will literally eat a hat (you may choose what kind): 1. Discover an easily exploitable vulnerability that allows access to a chosen user's private data 2. Email their security address about it 3. Tweet at them about it 4. Fax them about it 5. A week later the vulnerability is still exploitable You do not have to play fair. You're allowed to impersonate a suburban mom or a grandpa who's not good with technology. You're allowed to ramble or use vague and non-technical terms as long as a reasonably qualified person could determine what the vulnerability is. Specifically, it is not required that you include relevant product versions, steps to reproduce, a full reproduction video, or a one-sentence impact summary like "a caller can eavesdrop on the recipient of a Group Facetime call without their knowledge or consent". There's no apologising this away. The vulnerability was already a monumental fuckup, but this detail propels it into the realm of cultural dysfunction. It should not be possible to fail this badly. If you put listening devices in people's pockets, you need to hold yourself to a higher standard than "I dunno, bug reporting is hard".

doctorpangloss 8y ago
Well, pointing out that tech company engineers aren’t really all the great, indeed even below average, threatens like, one of HN’s biggest orthodoxies: that the people who read and write in this forum are brilliant.
sec-throwaway 8y ago
I work for a large software company. I am not on our security team, but have worked with our security team to investigate and resolve reported issues. I agree that it should not be possible to fail this badly. Even nonsensical reports to our security address will, with an SLA measured in hours, be read by a qualified human who will then reply to at least say "we're looking into it". A credible report will be immediately escalated to someone with relevant domain expertise for investigation. The security engineer will attempt to reproduce. A confirmed report will be escalated to an executive, who will determine urgency. For issues like this, where sensitive data is exposed, people will be woken up and several things will happen in parallel: the scope of the issue will be assessed, the root cause will be found, potential workarounds will be identified, a fix will be implemented, the potential existence of related issues will be investigated, and the reporter will be contacted to assess disclosure risk. Even in the worst case, where a complicated vulnerability exists in multiple versions of multiple products, requiring multiple patches and backports and requiring coordinated disclosure with partners, I'd expect a fix to be in customers' hands within 14 days.

philip1209 8y ago

What would happen if you submitted the bug via certified mail? How would that get triaged?

cm2187 8y ago

Also when a user reports a major flaw in your product and your response is that you will not lift a finger until the user fills the correct form, you have reached the state of a useless bureaucracy, completely unconcerned about the quality of your product. I remember being given that advice by an employee on the vendor’s own forum where I reported a problem. They are still waiting for their bug report and I have found an alternative product.

benologist 8y ago
I sent Stripe a bug report once and they replied I had to sign in on their website to email them.
DeathRay2K 8y ago
I found a bug on the PayPal site (I can't associate a new email address). I tried the contact forms in their app and on the website, both resulted in error messages. I tried emailing them, but got an automatic message back saying that they don't monitor emails, instead directing me to the support form on their website, which you have to log in to access. And as I mentioned, that form just ends in an error. So it seems like a way more common problem than it should be.

huffmsa 8y ago

If this were their only QA/QC issue, then fine, it might be excusable. But it's this, broken MBP keyboards, broken MBP hinges, bent iPads, the MBP core-i9 thermal issue and probably a few I'm forgetting. Nintendo doesn't have these kinds of failures, they're pretty analogous to Apple. Hardware and software. What's the difference? Nintendo cares and Apple doesn't. (Not to say Nintendo is issue free by any means). Maybe they should put some of that $250b in cash to work upgrading their processes.

javagram 8y ago
Nintendo has different priorities. They use plastic screens instead of glass for instance. Their focus is on building cheap but durable goods, e.g. plastic scratches more easily but is less likely to shatter catastrophically. They have definitely had their own industrial design failures - the Wii would sometimes overheat in sleep mode, the 3DS would scratch its own screen when placed under pressure in a pocket, the Switch supposedly can have its screen scratched by a bent dock. The initial run of switch controllers had an issue where the left side controller could disconnect very easily. Of course when your product is $149 or $299 replacing it or living with a scratch doesn’t hurt as much...
- huffmsa 8y ago
  Like I said, Nintendo is not perfect. But they at least try. Apple frankly doesn't appear to give a damn about shipping broken devices / software right now. And they have the resources to easily give a damn
  - javagram 8y ago
    Nintendo gets criticized pretty heavily for their decisions to do stuff like use plastic. Their devices are frequently described as “toys” on gaming forums that I visit due to this. Apple has an entirely different brand they need to maintain which is why they use metal, make their devices thin, have edge to edge screens, etc. Personally I have no problems with “toy” style design but I think a lot of apple’s premium is based on their sense of style and the feeling people have that iPhones, iPads, Macs are a “premium” product due to the industrial design. As an old school Mac user from back when Macs were beige boxes I personally don’t care about that stuff.

deleted 8y ago

[deleted]

jssmith 8y ago

One startup that’s working on this is unitQ: https://www.unitq.com https://www.unitq.com

bleriot 8y ago

Yes, a company at that scale could do it, but they’d have to actually spend some of their $200 billion instead of hoarding it Scrooge McDuck style.

Bud 8y ago
This is amusing to type, I guess, but not really true. Apple spent a huge amount last year: on stock buybacks (sigh), on building an enormous TV/film production studio from scratch, on a huge effort to develop self-driving cars, etc., etc. If Apple were doing anything remotely similar to "hoarding", they'd have more like $700 billion in cash right now.

tialaramex 8y ago

Raymond Chen (at Microsoft), for example has written repeatedly about incidents he spent lots of time investigating where your first instinct is "That's not a bug" and the eventual outcome was "Yup, that's not a bug" but it was a security ticket and so Raymond doggedly chased down every aspect to make sure it isn't a problem. He mostly invokes the Hitchiker's Guide quote "It rather involved being on the other side of this airtight hatchway" to suggest that often the problem in these bugs is that they have a step where you've legitimately got privileges, and then they use those privileges to... do something you legitimately need privileges for. That's not a bug, if you try it without the privileges it doesn't work, but Raymond has to walk through all the steps seeing whether anything surprising is going on. Now, whether engineers at a corp actually get given the space to do this stuff is an executive policy decision. Maybe at Apple not enough of them do, I can't say. But if it doesn't get done you are sooner or later going to miss cases where it _sounds_ like it's not a bug but actually there's a serious bug if you looked closely.