6 ms·

Apple gave Uber access to a secret feature that can record iPhone screens (2017)

91 points by minikites 5y ago

lallysingh 5y ago
Are they saying that they needed the API to have the iPhone app drive the watch app? By rendering to the screen, screenshotting it, and sending that to the watch? Sounds like bullshit. What does the watch have to do with the phone?
- willio58 5y ago
  Not long ago the Apple Watch was very limited in what it could render on-device. So a bunch of apps had to do it this way.
  - basseq 5y ago
    But only Uber (allegedly) had access to the private entitlement. So how did other apps do it, and why couldn't Uber? That said, this is certainly a "special relationship" with big partners like Uber to get their apps working at Apple Watch launch in 2015[1]. That the access "wasn't removed already if it was no longer required", well... I think we can all understand legacy code. [1] "It is likely that the Cupertino tech giant had to give this access after it gave app developers a four-month window to develop apps for its Watch before the unveiling of the product."
    - willio58 5y ago
      I’m not sure why this access required a “special relationship”. I remember years ago reading public Apple dev docs that recommended rendering certain things on iPhone and sending them over to the watch as a rasterized image.
- wlesieutre 5y ago
  It says specifically for maps rendering. Keep in mind the original Apple Watch had a 520 mhz single core processor and 256 MB of RAM shared with the OS and anything else running in the background. And pre-SwiftUI, 3rd party developers were working with WatchKit which was a lot more limited than what Apple was using internally. Could that render a single map frame? Probably. But if this was an animated map showing the car location with zoom/pan to follow along, I can definitely see it making sense to offload to the phone. What surprises me is that they needed a private entitlement for this, since you should be able to render a view out to an image without any entitlements [1]. Maybe that solution doesn't work offscreen? Or if the map was animated, maybe this can't play smoothly enough? [1] https://stackoverflow.com/questions/30696307/how-to-convert-a-uiview-to-an-image https://stackoverflow.com/questions/30696307/how-to-convert-...
  - wlesieutre 5y ago
    More info on WatchKit, per [1] > Can I use custom views in the Apple Watch? Can I customize the interface elements beyond their public API? > No. You can’t use custom views. WatchKit only supports certain native interface elements. None of the interface elements can be subclassed or customized beyond their public API. The available interface elements are WKInterfaceLabel, WKInterfaceButton, WKInterfaceImage, WKInterfaceGroup, WKInterfaceSeparator, WKInterfaceTable, WKInterfaceSwitch, WKInterfaceMap, WKInterfaceSlider and WKInterfaceTimer. In the original watchOS release, you can't even draw shapes, only display images. So to make a custom map you'd have to render the image yourself for display as a WKInterfaceImage. With watchOS 2 you were able to draw to a WKInterfaceImage with Core Graphics [2], but Uber had written their app before this. [1] https://www.raywenderlich.com/1950-watchkit-faq https://www.raywenderlich.com/1950-watchkit-faq [2] https://github.com/shu223/watchOS-2-Sampler#draw-paths-updated-by-hoppenichu https://github.com/shu223/watchOS-2-Sampler#draw-paths-updat...
  - lallysingh 5y ago
    Yeah that's the part that seems off. It implies that the image for the watch was on the iphone screen first, which doesn't seem quite "professional quality". Did they render something high quality (to look good on three options) with the GPU and then subsample the image for the watch?
- dmajor2 5y ago
  If I remember correctly, the "secret API call" was to the framebuffer. They were rendering the maps using the iPhones cpu/gpu, then pushing the rasters to the watch. The watch was not yet able to render the maps in realtime. Uber had to be able to create screens even in the background, so the API could have taken a snapshot of the active part of the screen buffer.
bellyfullofbac 5y ago
Don't worry people, Apple cares about your privacy! /s
ImTheMaddest 5y ago
Seems mostly reasonable to me. “It was used for an old version of the Apple Watch app, specifically to run the heavy lifting of rendering maps on your phone & then send the rendering to the Watch app. This dependency was removed with previous improvements to Apple’s OS & our app. Therefore, we’re removing this API from our iOS codebase."
- JasonFruit 5y ago
  The explanation is good, but that's a heck of a permission to allow. I hope Apple wisely took special precautions to make sure Uber wasn't using other functions that permission allows.
- wavefunction 5y ago
  I guess the argument then is "technical limitations excuse privacy lapses."
- danpalmer 5y ago
  Further to this, Apple auditing that Uber is not abusing this permission is pretty easy: they could reasonably have Apple engineers embedded in the team at Uber – something that would make sense for the scale, Uber is a US company, based in CA, so dealing with it in court would be possible and a fairly well-defined process, Apple could even have engineers trace the application properly during App Store review to check what it was doing at runtime, or put very stringent requirements on _how_ the entitlement was used at the code level to make it easier for them to enforce usage. None of that scales to the millions of developers they have on their platform, but for such a key application, for essentially marketing purposes for the Apple Watch product launch, sure.
  - hdjjhhvvhga 5y ago
    > Further to this, Apple auditing that Uber is not abusing this permission is pretty easy Are we talking about the same company using Greyball to fool local authorities not just abroad, but also in the USA? Taking into account total lack of moral integrity in their top management, I would not believe in anything they promise.
dang 5y ago
Discussed at the time: Apple iPhone Security Issues - https://news.ycombinator.com/item?id=15418558 https://news.ycombinator.com/item?id=15418558 - Oct 2017 (1 comment) Apple gave Uber's app 'unprecedented' access to a secret backdoor - https://news.ycombinator.com/item?id=15417137 https://news.ycombinator.com/item?id=15417137 - Oct 2017 (3 comments) Uber app can secretly record your iPhone screen thanks to special ‘entitlement’ - https://news.ycombinator.com/item?id=15414884 https://news.ycombinator.com/item?id=15414884 - Oct 2017 (39 comments) Apple gave Uber app 'unprecedented' access to backdoor that can record screens - https://news.ycombinator.com/item?id=15413036 https://news.ycombinator.com/item?id=15413036 - Oct 2017 (4 comments) Uber’s iOS App Had Secret Permissions That Allowed It to Copy Your Phone Screen - https://news.ycombinator.com/item?id=15411533 https://news.ycombinator.com/item?id=15411533 - Oct 2017 (21 comments)
lerp9896 5y ago
although I get that they gave access to recording the screen, literally every app can integrate https://www.smartlook.com/ https://www.smartlook.com/ and record user sessions while using the app .
parkefoyde 5y ago
There are many mobile analytics applications such as smartlook.com, hotjar.com that provide screen recording feature.
- lerp9896 5y ago
  exactly what i was thinking. Why is everyone acting like its a priveleged access. Any app dev can integrate smartlook and watch how you use the app

91 points by minikites 5y ago

Are they saying that they needed the API to have the iPhone app drive the watch app? By rendering to the screen, screenshotting it, and sending that to the watch? Sounds like bullshit. What does the watch have to do with the phone?

willio58 5y ago
Not long ago the Apple Watch was very limited in what it could render on-device. So a bunch of apps had to do it this way.
- basseq 5y ago
  But only Uber (allegedly) had access to the private entitlement. So how did other apps do it, and why couldn't Uber? That said, this is certainly a "special relationship" with big partners like Uber to get their apps working at Apple Watch launch in 2015[1]. That the access "wasn't removed already if it was no longer required", well... I think we can all understand legacy code. [1] "It is likely that the Cupertino tech giant had to give this access after it gave app developers a four-month window to develop apps for its Watch before the unveiling of the product."
  - willio58 5y ago
    I’m not sure why this access required a “special relationship”. I remember years ago reading public Apple dev docs that recommended rendering certain things on iPhone and sending them over to the watch as a rasterized image.
wlesieutre 5y ago
It says specifically for maps rendering. Keep in mind the original Apple Watch had a 520 mhz single core processor and 256 MB of RAM shared with the OS and anything else running in the background. And pre-SwiftUI, 3rd party developers were working with WatchKit which was a lot more limited than what Apple was using internally. Could that render a single map frame? Probably. But if this was an animated map showing the car location with zoom/pan to follow along, I can definitely see it making sense to offload to the phone. What surprises me is that they needed a private entitlement for this, since you should be able to render a view out to an image without any entitlements [1]. Maybe that solution doesn't work offscreen? Or if the map was animated, maybe this can't play smoothly enough? [1] https://stackoverflow.com/questions/30696307/how-to-convert-a-uiview-to-an-image https://stackoverflow.com/questions/30696307/how-to-convert-...
- wlesieutre 5y ago
  More info on WatchKit, per [1] > Can I use custom views in the Apple Watch? Can I customize the interface elements beyond their public API? > No. You can’t use custom views. WatchKit only supports certain native interface elements. None of the interface elements can be subclassed or customized beyond their public API. The available interface elements are WKInterfaceLabel, WKInterfaceButton, WKInterfaceImage, WKInterfaceGroup, WKInterfaceSeparator, WKInterfaceTable, WKInterfaceSwitch, WKInterfaceMap, WKInterfaceSlider and WKInterfaceTimer. In the original watchOS release, you can't even draw shapes, only display images. So to make a custom map you'd have to render the image yourself for display as a WKInterfaceImage. With watchOS 2 you were able to draw to a WKInterfaceImage with Core Graphics [2], but Uber had written their app before this. [1] https://www.raywenderlich.com/1950-watchkit-faq https://www.raywenderlich.com/1950-watchkit-faq [2] https://github.com/shu223/watchOS-2-Sampler#draw-paths-updated-by-hoppenichu https://github.com/shu223/watchOS-2-Sampler#draw-paths-updat...
- lallysingh 5y ago
  Yeah that's the part that seems off. It implies that the image for the watch was on the iphone screen first, which doesn't seem quite "professional quality". Did they render something high quality (to look good on three options) with the GPU and then subsample the image for the watch?
dmajor2 5y ago
If I remember correctly, the "secret API call" was to the framebuffer. They were rendering the maps using the iPhones cpu/gpu, then pushing the rasters to the watch. The watch was not yet able to render the maps in realtime. Uber had to be able to create screens even in the background, so the API could have taken a snapshot of the active part of the screen buffer.

bellyfullofbac 5y ago

Don't worry people, Apple cares about your privacy! /s

ImTheMaddest 5y ago

Seems mostly reasonable to me. “It was used for an old version of the Apple Watch app, specifically to run the heavy lifting of rendering maps on your phone & then send the rendering to the Watch app. This dependency was removed with previous improvements to Apple’s OS & our app. Therefore, we’re removing this API from our iOS codebase."

JasonFruit 5y ago
The explanation is good, but that's a heck of a permission to allow. I hope Apple wisely took special precautions to make sure Uber wasn't using other functions that permission allows.
wavefunction 5y ago
I guess the argument then is "technical limitations excuse privacy lapses."
danpalmer 5y ago
Further to this, Apple auditing that Uber is not abusing this permission is pretty easy: they could reasonably have Apple engineers embedded in the team at Uber – something that would make sense for the scale, Uber is a US company, based in CA, so dealing with it in court would be possible and a fairly well-defined process, Apple could even have engineers trace the application properly during App Store review to check what it was doing at runtime, or put very stringent requirements on _how_ the entitlement was used at the code level to make it easier for them to enforce usage. None of that scales to the millions of developers they have on their platform, but for such a key application, for essentially marketing purposes for the Apple Watch product launch, sure.
- hdjjhhvvhga 5y ago
  > Further to this, Apple auditing that Uber is not abusing this permission is pretty easy Are we talking about the same company using Greyball to fool local authorities not just abroad, but also in the USA? Taking into account total lack of moral integrity in their top management, I would not believe in anything they promise.

dang 5y ago

Discussed at the time: Apple iPhone Security Issues - https://news.ycombinator.com/item?id=15418558 https://news.ycombinator.com/item?id=15418558 - Oct 2017 (1 comment) Apple gave Uber's app 'unprecedented' access to a secret backdoor - https://news.ycombinator.com/item?id=15417137 https://news.ycombinator.com/item?id=15417137 - Oct 2017 (3 comments) Uber app can secretly record your iPhone screen thanks to special ‘entitlement’ - https://news.ycombinator.com/item?id=15414884 https://news.ycombinator.com/item?id=15414884 - Oct 2017 (39 comments) Apple gave Uber app 'unprecedented' access to backdoor that can record screens - https://news.ycombinator.com/item?id=15413036 https://news.ycombinator.com/item?id=15413036 - Oct 2017 (4 comments) Uber’s iOS App Had Secret Permissions That Allowed It to Copy Your Phone Screen - https://news.ycombinator.com/item?id=15411533 https://news.ycombinator.com/item?id=15411533 - Oct 2017 (21 comments)

lerp9896 5y ago

although I get that they gave access to recording the screen, literally every app can integrate https://www.smartlook.com/ https://www.smartlook.com/ and record user sessions while using the app .

parkefoyde 5y ago

There are many mobile analytics applications such as smartlook.com, hotjar.com that provide screen recording feature.

lerp9896 5y ago
exactly what i was thinking. Why is everyone acting like its a priveleged access. Any app dev can integrate smartlook and watch how you use the app