11 ms·

Python grapples with Apple App Store rejections

254 points by leephillips 2y ago

gjsman-1000 2y ago
Why can’t Apple just add “itms-services” as a forbidden URL scheme on a sandbox level? I don’t see why the App Sandbox can’t block (and isn’t already blocking) certain protocols. Heck, what if I have a malicious web frame inside my app that tries to invoke “itms-services”, similar to this Polyfill.io debacle?
- PaulHoule 2y ago
  Python just has to rot13 the scheme and hash the encrypted schemes.
  - gjsman-1000 2y ago
    That came up in the original post with whether an obfuscation could be an acceptable workaround; but it came up that Apple really doesn’t like obfuscation techniques. The workaround for now is a compiler flag that excludes problematic code from iOS builds. I’m still asking though why, if Apple doesn’t like apps which use this protocol, the sandbox is not intervening; as surely an approved app could have a malicious web view.
    - favorited 2y ago
      Sandboxed apps are allowed to use itms-services:// links, it's just not allowed in the App Store - iOS enterprise apps using in-house deployments can use it for installs and updates, and sandboxed Mac apps deployed outside of the App Store can use it as well. However, App Review Guidelines forbid App Store apps from installing other apps, so that scheme gets scanned during review.
      - gjsman-1000 2y ago
        That makes some sense, but then I have a new question: Why doesn’t Apple have different certificate schemes for in-house versus App Store (if they don’t already)? In which case, the iOS Sandbox should be smart enough, and probably does already delineate, allowed functionality based on whether an app comes via App Store or via a private deployment.
        dwaite 2y ago
        > Why doesn’t Apple have different certificate schemes for in-house versus App Store (if they don’t already)? Enterprise distribution allows you to deploy applications to corporate managed devices with no App Store review whatsoever. The restrictions are mostly in the business agreement in who you can provide enterprise distribution to (e.g. employees and contractors) and on what your apps can do. The justification is that the enterprise has a relationship with the employee/contractor, is ultimately on the hook for abuses/harms they do via MDM on employee devices. This is the situation that led to both Facebook and Google having their enterprise accounts banned temporarily a few years ago, as they were each using them as part of a market analysis program - they offered to install VPN software onto consumer devices that monitored web and third-party app usage. Such monitoring is not allowed even for employees per the enterprise developer account agreement.
      - bmarswalker 2y ago
        This can be handled by granting privileges to open that scheme to enterprise Apps and not granting to regular App Store apps. Relying on string scanning is simply not secure.
        AshamedCaptain 2y ago
        Welcome to Apple's much applauded security model. More seriously, I'm sure they also prevent the privilege to that URI scheme. This is likely part of some ill-thought defense-in-depth approach. Same way they search for the names of private symbols in the exec, even when the linker will outright refuse to give you those. I absolutely detest this pervasiveness of useless layers of security that add almost nothing. But since almost nothing is not nothing, no one can remove any of them. Like cockroach papers, I'm going to call them "cockroach security". Practically everything is infested with those these days.
    - dwaite 2y ago
      > The workaround for now is a compiler flag that excludes problematic code from iOS builds. That sounds like the correct fix, in that it is least likely to get your developer account banned
- praseodym 2y ago
  I’m not sure what the big deal with the url handler is, but I can’t imagine it causing remote code execution or other actual malicious behaviour. At this point Apple seems to be using simple substring matches, so if there is any exploit vector the malware authors can circumvent the check using "itms" + "-services" or something more sophisticated like ROT13.
  - gjsman-1000 2y ago
    Which is also just why… the App Store review process claiming this is a problem doesn’t seem to make any sense. Imagine your app embeds a WebView at myapp.com/terms. It’s your Terms of Service, you show it to everyone when they sign up. Everyone clicks OK. After it’s on the App Store, you modify your WebView to include `itms-services` for some reason. You’ve just completely bypassed App Store review and gotten that URL handler into your app. The sandbox should stop you - but clearly the review processes don’t consider this possibility and are enforcing it before you publish. Why? My point is that the scanning for this handler, if Apple doesn’t want to allow it, seems misplaced if they wanted the ban to actually be effective.
    - threeseed 2y ago
      The review process is about Apple giving you a clear direction about what is acceptable. There are many ways to get around their restrictions. But doing so will get you banned.
      - thfuran 2y ago
        I haven't heard anything to suggest that Apple intends the review process to give clear direction.
      - RobotToaster 2y ago
        They refused to tell him why it was rejected, hardly "clear direction"
        2y ago
- lilyball 2y ago
  The URL is likely used from within Apple frameworks for various purposes, and therefore it's possible for an app process to open the URL even without the app itself knowing about the URL.
heavyset_go 2y ago
It's not just Apple that pulls shenanigans like this. Try building a Python app with PyInstaller while you have Windows Defender live scanning on, which is the default setting. You won't even be able to compile a binary without Defender preventing you from doing so. Similarly, try running the binary produced by PyInstaller with Windows Defender on. Defender will say it's malicious and won't run it. It's a bit dystopian that both major OS platforms go out of their way to prevent you from distributing and running your Python apps.
- hulitu 2y ago
  MS Defender _is_ malware. Avoid at all costs. /s
- josephcsible 2y ago
  It's not just Python apps. It's anything by small-time developers without expensive certificates. I once used MSVC to compile a C program that was little more than a "Hello, World", and Defender called it the Win32/Wacatac Trojan.
  - heavyset_go 2y ago
    > It's not just Python apps. It's anything by small-time developers without expensive certificates. This is definitely the case and has been my experience, as well. We live in some dark times when it comes to building and sharing anything as small developers, especially if the things you're building are free. I stopped updating my open-source Mac apps because I can't justify the cost of jumping over artificial hurdles Apple puts in place that ensure users can't run the apps they want to use. I have other hobbies where spending money actually gives me tangible goods and benefits versus paying an arbitrary yearly tax for the privilege to build stuff that ultimately benefits Apple.
    - OsrsNeedsf2P 2y ago
      This was the case for our open source app as well. The only reason we're on Apple is one of our users likes the app so much, they handle the certificates. Which is deeply ironic, if you think about it.
TillE 2y ago
Obfuscation seems like a great way to get your developer account suspended. I suspect Apple is doing a lot more than just basic static analysis of the binary on disk. Glad they went with a config option instead.
- veeti 2y ago
  They are opaquely rejecting apps for just literally containing the string "itms-services" in the binary and you still give them credit for a more sophisticated analysis? Lol.
  - nozzlegear 2y ago
    We can assume that a simple string search is one of the basic checks they do before moving onto more advanced checks.
    - malfist 2y ago
      Why can we assume that?
      - zarathustreal 2y ago
        Because it’s less effort to implement?
      - oldpersonintx 2y ago
        [dead]
    - youcantcook 2y ago
      [dead]
    - coldtea 2y ago
      Every story I've seen over the years about MAS/iOS AS rejections point to their checks not being advanced at all.
  - 2y ago
lilyball 2y ago
Why does urllib have this URL scheme anyway? If Python libraries are hard-coding knowledge about Apple proprietary stuff, then it should be no surprise that Apple may take issue with that.
- deleted 2y ago
  [deleted]
- burnte 2y ago
  The article is pretty clear that it has this string because it's the Apple recommended method for launching a new app, and Python on MacOS does that. It's part of a standard library.
- 7A2A-4AC 2y ago
  I believe it was added as the `itms-services://?` URL format is non-standard and broke the existing logic in `urllib.parse`: https://github.com/python/cpython/pull/104312 https://github.com/python/cpython/pull/104312
  - dwaite 2y ago
    My understanding per RFC 3986 is this is a valid URL. The authority consists of a host with a blank name and blank path (path-abempty). The issue is a lot of libraries do not make a distinction between a field explicitly being declared blank and being left undeclared (e.g. nil vs ""). Looks like rather than change python's URL library to recognize this distinction, they created an exception list that just hard-codes output behavior. One can make a case that client software should be robust enough to handle both forms of URL (e.g. itms-service URLs are likely just URI, but the original developers thought all URI should have two forward slashes in them). I can say specifically for "scheme://" calls that such robustness on the recipient side is very rare, because if they understood URI well enough to be robust then they wouldn't have had two extra slashes that they will never use.
    - lilyball 2y ago
      A comment on the issue linked from the PR references https://url.spec.whatwg.org/#url-serializing https://url.spec.whatwg.org/#url-serializing as a rationale for why itms-services would normally not have the "//" and therefore needs the exception. However that standard says non-null, not non-empty¹, so it sounds like python's URL library is actually just wrong here in that it's treating an empty host the same as null. ¹Also I checked, that standard does actually define an empty host (https://url.spec.whatwg.org/#empty-host https://url.spec.whatwg.org/#empty-host) as a distinct concept from a null host.
deleted 2y ago
[deleted]
amelius 2y ago
Can we have Separation of Powers on our digital platforms? It is pretty shitty that the one who sells phones also determines what goes on them.
- denkmoon 2y ago
  I choose Apple to be the custodian of apps that go on my phone. There is a platform where the user is the one that determines what goes on their phone, it's called Android.
  - areoform 2y ago
    While I am glad that you have the choice to be a digital serf - paying your lord for using land (devices), I would much rather prefer the choice to not be one. That way, you can opt into digital serfdom and people like me can opt out. - Serf here isn’t pejorative, but descriptive. Apple is behaving like a feudal lord. Down to the claims of protecting the land and arbitrating disputes.
    - denkmoon 2y ago
      Why do you act like Android doesn't exist? You _can_ opt out of "digital serfdom".
      - mirsadm 2y ago
        How is Android any different?
        pinusc 2y ago
        In some cases one can install a de-googlified version of android, and install whatever app (.apk) they wish through alternative app stores or directly. For 99.9% of people, Android and iOS will be virtually identical in terms of freedom, as they will just install apps from the play store and use google-services on the manufacturer-provided (and -bloated) android install that comes with their phone. That being said, for those who do care, the ability to take control of your phone and run AOSP, an actually FOSS distribution, and only run FOSS apps, or install whatever app you want, is unparalleled on Android vs iOS.
edflsafoiewq 2y ago
I thought this was interesting > Alex Gaynor suggested that the project try a an approach that Keith-Magee had not put forward inspired by Gaynor's experience with the cryptography library. The project often receives complaints that the library refuses to parse a certificate that is technically invalid, but was in wide use. He said that the policy was to accept pull requests that work around those issues ""provided they are small, localized, and generally aren't too awful"". But, he added, these patches should only be accepted on the condition that someone complains to the third party (in this case Apple), and extracts some kind of commitment that they would do something about it. He suggested that the workaround be time-limited, to give users a decent experience ""while also not letting large firms simply externalize their bizarre issues onto OSS projects"". as a solution to the familiar problem of users wanting OSS to work around bugs in commercial software because OSS maintainers are easier to bully and they know bug reports to Megacorp go straight to a black hole.
- dnawlp 2y ago
  Seems like the quote is a non-statement given that Apple's bug report system is a black hole. Big talk as usual from Pythonistas without any connection to reality.
gorgoiler 2y ago
The offending string is only there because Python’s urllib has a hard-coded list of schemes which use a hostname component or “netloc”. It’s fine for that list to contain known schemes from RFCs. Anything else — including proprietary third party schemes — should just use a heuristic. The list is called uses_netloc and is used to help parse the user@host:port part of https, ftp, etc. domains. It’s this list of schemes that includes the forbidden string itms-services, used for Apple’s proprietary iTunes software. The only code that needs this is urlunsplit and urljoin. If your parsed URL has a netloc then the list isn’t even relevant — if you have a netloc then you are assumed to be in uses_netloc. This all seems like a much more sensible approach than trying to selectively include or exclude naughty strings from the source code, per some corporation’s passive aggressive demands.
deleted 2y ago
[deleted]
wodenokoto 2y ago
Why is the “ itms-services URL scheme” in base Python to begin with? Why does Python have code for interacting with iTunes out of the box?
- lifthrasiir 2y ago
  Apparently it was another hack to prevent incorrect parsing of such URLs on the server side, without the actual interaction happening [1]. [1] https://github.com/python/cpython/issues/104139 https://github.com/python/cpython/issues/104139
  - polski-g 2y ago
    This is from a fucking test case to detect hyphens in the URL scheme? Just change it to asdf-zxcv. Or remove test cases from end-user python installs. What are we doing here?

254 points by leephillips 2y ago

Why can’t Apple just add “itms-services” as a forbidden URL scheme on a sandbox level? I don’t see why the App Sandbox can’t block (and isn’t already blocking) certain protocols. Heck, what if I have a malicious web frame inside my app that tries to invoke “itms-services”, similar to this Polyfill.io debacle?

PaulHoule 2y ago
Python just has to rot13 the scheme and hash the encrypted schemes.
- gjsman-1000 2y ago
  That came up in the original post with whether an obfuscation could be an acceptable workaround; but it came up that Apple really doesn’t like obfuscation techniques. The workaround for now is a compiler flag that excludes problematic code from iOS builds. I’m still asking though why, if Apple doesn’t like apps which use this protocol, the sandbox is not intervening; as surely an approved app could have a malicious web view.
  - favorited 2y ago
    Sandboxed apps are allowed to use itms-services:// links, it's just not allowed in the App Store - iOS enterprise apps using in-house deployments can use it for installs and updates, and sandboxed Mac apps deployed outside of the App Store can use it as well. However, App Review Guidelines forbid App Store apps from installing other apps, so that scheme gets scanned during review.
    - gjsman-1000 2y ago
      That makes some sense, but then I have a new question: Why doesn’t Apple have different certificate schemes for in-house versus App Store (if they don’t already)? In which case, the iOS Sandbox should be smart enough, and probably does already delineate, allowed functionality based on whether an app comes via App Store or via a private deployment.
      - dwaite 2y ago
        > Why doesn’t Apple have different certificate schemes for in-house versus App Store (if they don’t already)? Enterprise distribution allows you to deploy applications to corporate managed devices with no App Store review whatsoever. The restrictions are mostly in the business agreement in who you can provide enterprise distribution to (e.g. employees and contractors) and on what your apps can do. The justification is that the enterprise has a relationship with the employee/contractor, is ultimately on the hook for abuses/harms they do via MDM on employee devices. This is the situation that led to both Facebook and Google having their enterprise accounts banned temporarily a few years ago, as they were each using them as part of a market analysis program - they offered to install VPN software onto consumer devices that monitored web and third-party app usage. Such monitoring is not allowed even for employees per the enterprise developer account agreement.
    - bmarswalker 2y ago
      This can be handled by granting privileges to open that scheme to enterprise Apps and not granting to regular App Store apps. Relying on string scanning is simply not secure.
      - AshamedCaptain 2y ago
        Welcome to Apple's much applauded security model. More seriously, I'm sure they also prevent the privilege to that URI scheme. This is likely part of some ill-thought defense-in-depth approach. Same way they search for the names of private symbols in the exec, even when the linker will outright refuse to give you those. I absolutely detest this pervasiveness of useless layers of security that add almost nothing. But since almost nothing is not nothing, no one can remove any of them. Like cockroach papers, I'm going to call them "cockroach security". Practically everything is infested with those these days.
  - dwaite 2y ago
    > The workaround for now is a compiler flag that excludes problematic code from iOS builds. That sounds like the correct fix, in that it is least likely to get your developer account banned
praseodym 2y ago
I’m not sure what the big deal with the url handler is, but I can’t imagine it causing remote code execution or other actual malicious behaviour. At this point Apple seems to be using simple substring matches, so if there is any exploit vector the malware authors can circumvent the check using "itms" + "-services" or something more sophisticated like ROT13.
- gjsman-1000 2y ago
  Which is also just why… the App Store review process claiming this is a problem doesn’t seem to make any sense. Imagine your app embeds a WebView at myapp.com/terms. It’s your Terms of Service, you show it to everyone when they sign up. Everyone clicks OK. After it’s on the App Store, you modify your WebView to include `itms-services` for some reason. You’ve just completely bypassed App Store review and gotten that URL handler into your app. The sandbox should stop you - but clearly the review processes don’t consider this possibility and are enforcing it before you publish. Why? My point is that the scanning for this handler, if Apple doesn’t want to allow it, seems misplaced if they wanted the ban to actually be effective.
  - threeseed 2y ago
    The review process is about Apple giving you a clear direction about what is acceptable. There are many ways to get around their restrictions. But doing so will get you banned.
    - thfuran 2y ago
      I haven't heard anything to suggest that Apple intends the review process to give clear direction.
    - RobotToaster 2y ago
      They refused to tell him why it was rejected, hardly "clear direction"
      - 2y ago
lilyball 2y ago
The URL is likely used from within Apple frameworks for various purposes, and therefore it's possible for an app process to open the URL even without the app itself knowing about the URL.

heavyset_go 2y ago

It's not just Apple that pulls shenanigans like this. Try building a Python app with PyInstaller while you have Windows Defender live scanning on, which is the default setting. You won't even be able to compile a binary without Defender preventing you from doing so. Similarly, try running the binary produced by PyInstaller with Windows Defender on. Defender will say it's malicious and won't run it. It's a bit dystopian that both major OS platforms go out of their way to prevent you from distributing and running your Python apps.

hulitu 2y ago
MS Defender _is_ malware. Avoid at all costs. /s
josephcsible 2y ago
It's not just Python apps. It's anything by small-time developers without expensive certificates. I once used MSVC to compile a C program that was little more than a "Hello, World", and Defender called it the Win32/Wacatac Trojan.
- heavyset_go 2y ago
  > It's not just Python apps. It's anything by small-time developers without expensive certificates. This is definitely the case and has been my experience, as well. We live in some dark times when it comes to building and sharing anything as small developers, especially if the things you're building are free. I stopped updating my open-source Mac apps because I can't justify the cost of jumping over artificial hurdles Apple puts in place that ensure users can't run the apps they want to use. I have other hobbies where spending money actually gives me tangible goods and benefits versus paying an arbitrary yearly tax for the privilege to build stuff that ultimately benefits Apple.
  - OsrsNeedsf2P 2y ago
    This was the case for our open source app as well. The only reason we're on Apple is one of our users likes the app so much, they handle the certificates. Which is deeply ironic, if you think about it.

TillE 2y ago

Obfuscation seems like a great way to get your developer account suspended. I suspect Apple is doing a lot more than just basic static analysis of the binary on disk. Glad they went with a config option instead.

veeti 2y ago
They are opaquely rejecting apps for just literally containing the string "itms-services" in the binary and you still give them credit for a more sophisticated analysis? Lol.
- nozzlegear 2y ago
  We can assume that a simple string search is one of the basic checks they do before moving onto more advanced checks.
  - malfist 2y ago
    Why can we assume that?
    - zarathustreal 2y ago
      Because it’s less effort to implement?
    - oldpersonintx 2y ago
      [dead]
  - youcantcook 2y ago
    [dead]
  - coldtea 2y ago
    Every story I've seen over the years about MAS/iOS AS rejections point to their checks not being advanced at all.
- 2y ago

lilyball 2y ago

Why does urllib have this URL scheme anyway? If Python libraries are hard-coding knowledge about Apple proprietary stuff, then it should be no surprise that Apple may take issue with that.

deleted 2y ago
[deleted]
burnte 2y ago
The article is pretty clear that it has this string because it's the Apple recommended method for launching a new app, and Python on MacOS does that. It's part of a standard library.
7A2A-4AC 2y ago
I believe it was added as the `itms-services://?` URL format is non-standard and broke the existing logic in `urllib.parse`: https://github.com/python/cpython/pull/104312 https://github.com/python/cpython/pull/104312
- dwaite 2y ago
  My understanding per RFC 3986 is this is a valid URL. The authority consists of a host with a blank name and blank path (path-abempty). The issue is a lot of libraries do not make a distinction between a field explicitly being declared blank and being left undeclared (e.g. nil vs ""). Looks like rather than change python's URL library to recognize this distinction, they created an exception list that just hard-codes output behavior. One can make a case that client software should be robust enough to handle both forms of URL (e.g. itms-service URLs are likely just URI, but the original developers thought all URI should have two forward slashes in them). I can say specifically for "scheme://" calls that such robustness on the recipient side is very rare, because if they understood URI well enough to be robust then they wouldn't have had two extra slashes that they will never use.
  - lilyball 2y ago
    A comment on the issue linked from the PR references https://url.spec.whatwg.org/#url-serializing https://url.spec.whatwg.org/#url-serializing as a rationale for why itms-services would normally not have the "//" and therefore needs the exception. However that standard says non-null, not non-empty¹, so it sounds like python's URL library is actually just wrong here in that it's treating an empty host the same as null. ¹Also I checked, that standard does actually define an empty host (https://url.spec.whatwg.org/#empty-host https://url.spec.whatwg.org/#empty-host) as a distinct concept from a null host.

deleted 2y ago

[deleted]

amelius 2y ago

Can we have Separation of Powers on our digital platforms? It is pretty shitty that the one who sells phones also determines what goes on them.

denkmoon 2y ago
I choose Apple to be the custodian of apps that go on my phone. There is a platform where the user is the one that determines what goes on their phone, it's called Android.
- areoform 2y ago
  While I am glad that you have the choice to be a digital serf - paying your lord for using land (devices), I would much rather prefer the choice to not be one. That way, you can opt into digital serfdom and people like me can opt out. - Serf here isn’t pejorative, but descriptive. Apple is behaving like a feudal lord. Down to the claims of protecting the land and arbitrating disputes.
  - denkmoon 2y ago
    Why do you act like Android doesn't exist? You _can_ opt out of "digital serfdom".
    - mirsadm 2y ago
      How is Android any different?
      - pinusc 2y ago
        In some cases one can install a de-googlified version of android, and install whatever app (.apk) they wish through alternative app stores or directly. For 99.9% of people, Android and iOS will be virtually identical in terms of freedom, as they will just install apps from the play store and use google-services on the manufacturer-provided (and -bloated) android install that comes with their phone. That being said, for those who do care, the ability to take control of your phone and run AOSP, an actually FOSS distribution, and only run FOSS apps, or install whatever app you want, is unparalleled on Android vs iOS.

edflsafoiewq 2y ago

I thought this was interesting > Alex Gaynor suggested that the project try a an approach that Keith-Magee had not put forward inspired by Gaynor's experience with the cryptography library. The project often receives complaints that the library refuses to parse a certificate that is technically invalid, but was in wide use. He said that the policy was to accept pull requests that work around those issues ""provided they are small, localized, and generally aren't too awful"". But, he added, these patches should only be accepted on the condition that someone complains to the third party (in this case Apple), and extracts some kind of commitment that they would do something about it. He suggested that the workaround be time-limited, to give users a decent experience ""while also not letting large firms simply externalize their bizarre issues onto OSS projects"". as a solution to the familiar problem of users wanting OSS to work around bugs in commercial software because OSS maintainers are easier to bully and they know bug reports to Megacorp go straight to a black hole.

dnawlp 2y ago
Seems like the quote is a non-statement given that Apple's bug report system is a black hole. Big talk as usual from Pythonistas without any connection to reality.

gorgoiler 2y ago

The offending string is only there because Python’s urllib has a hard-coded list of schemes which use a hostname component or “netloc”. It’s fine for that list to contain known schemes from RFCs. Anything else — including proprietary third party schemes — should just use a heuristic. The list is called uses_netloc and is used to help parse the user@host:port part of https, ftp, etc. domains. It’s this list of schemes that includes the forbidden string itms-services, used for Apple’s proprietary iTunes software. The only code that needs this is urlunsplit and urljoin. If your parsed URL has a netloc then the list isn’t even relevant — if you have a netloc then you are assumed to be in uses_netloc. This all seems like a much more sensible approach than trying to selectively include or exclude naughty strings from the source code, per some corporation’s passive aggressive demands.

deleted 2y ago

[deleted]

wodenokoto 2y ago

Why is the “ itms-services URL scheme” in base Python to begin with? Why does Python have code for interacting with iTunes out of the box?

lifthrasiir 2y ago
Apparently it was another hack to prevent incorrect parsing of such URLs on the server side, without the actual interaction happening [1]. [1] https://github.com/python/cpython/issues/104139 https://github.com/python/cpython/issues/104139
- polski-g 2y ago
  This is from a fucking test case to detect hyphens in the URL scheme? Just change it to asdf-zxcv. Or remove test cases from end-user python installs. What are we doing here?