36 ms·

It seems like there is a lot of confusion here as to whether this is real or not. I've been able to confirm the behavior in the post by: - Using a new, random

It seems like there is a lot of confusion here as to whether this is real or not. I've been able to confirm the behavior in the post by: - Using a new, random executable. Even echo $rand_int will work. Edit: What I mean here is generate your rand int beforehand and statically include it in your script. - Using a fresh filename too. Just throw a rand int at the end there. e.g. /tmp/test4329.sh I MITMd myself while recording the network traffic and, sure enough, there is a request to ocsp.apple.com with a hash in the URL path and a bunch of binary data in the response body. Unsure what it is yet but the URL suggests it is generating a cert for the binary and checking it. See: https://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol https://en.wikipedia.org/wiki/Online_Certificate_Status_Prot... Here's the URL I saw: http://ocsp.apple.com/ocsp-devid01/ME4wTKADAgEAMEUwQzBBMAkGBSsOAwIaBQAEFDOB0e%2FbaLCFIU0u76%2BMSmlkPCpsBBRXF%2B2iz9x8mKEQ4Py%2Bhy0s8uMXVAIIfYbtkeEKZsI%3D http://ocsp.apple.com/ocsp-devid01/ME4wTKADAgEAMEUwQzBBMAkGB... Edit2: Anyone know what this hash format is? It's not quite base64, nor is it multiple base64 strings separated with '+'s but it seems similar... Edit3: Here is the exact filename and file I used: https://gist.github.com/UsmannK/abb4b239c98ee45bdfcc5b284bf0029e https://gist.github.com/UsmannK/abb4b239c98ee45bdfcc5b284bf0... Edit4 (final one probably...): On subsequent attempts I'm only seeing a request to https://api.apple-cloudkit.com https://api.apple-cloudkit.com and not the OCSP one anymore. Curiously, there's no headers at all. It is just checking for connectivity.

jwatte 6y ago
Let's assume that sending network packets to verify the trustworthiness of commands is a good idea. (It may not be, but that's a different discussion.) If you have a modern OS with sufficient virtualization and containerization and indirection, you could optimistically let the commands run, and not commit the side effects of the command until you get back a result. Create little write logged mini branches of your file system, and only actually pause when someone else wants to inspect your side effects. By then, an asynchronous check should have gotten back to you.
kccqzy 6y ago
OCSP is Online Certificate Status Protocol, generally used for checking the revocation status of certificates. You used to be able to turn it off in keychain access, but that ability went away in recent macOS releases.
- VonGuard 6y ago
  Ah, Apple. When you can no longer innovate, just start removing features and call it simplicity...
  - D-Coder 6y ago
    Feature-removal has been the most aggravating part of my Mac life for the past several years. Admittedly I tend to use unusual features, but it's just another PITA when they go away.
  - throwaway851 6y ago
    Another way to look at it is that Apple is making it harder to run the system in an insecure fashion. You may not agree with that decision, but I certainly appreciate how Apple is looking out for the safety and security of the user. Tangent: as much as some developers hate that the only way to distribute apps for the iPhone is through the App Store, as a user I consider that walled garden of apps to be a real security benefit. When John Gruber says “If you must use Zoom or simply want to use it, I highly recommend using it on your iPad and iPhone only. The iOS version is sandboxed and reviewed by the App Store.” There’s a reason why he can say things like that and it’s because Apple draws a hard line in the sand that not everyone will be happy with.
    - 43920 6y ago
      Wouldn't a sandboxed Zoom downloaded directly from them be equally secure?
      - Retric 6y ago
        Apple’s rejected a huge number of App updates for security reasons. It’s not a huge benefit, but it does exist.
        cliffsteele 6y ago
        And also allowed a jailbreak app in the iOS App Store. Yes, it only happened once (that I know of), but it still shows you can't really be oblivious to their practices.
deleted 6y ago
[deleted]
markandrewj 6y ago
The isn't specific to the article, but another place that can be interesting to look at system activity on Mac OS is the console. https://support.apple.com/en-ca/guide/console/cnslbf30b61a/mac https://support.apple.com/en-ca/guide/console/cnslbf30b61a/m...
usmannk 6y ago
I can't edit anymore but it seems like the OCSP link could potentially be a red herring just checking the cert for the next request to https://api.apple-cloudkit.com/ https://api.apple-cloudkit.com/. It's worth looking further!
saagarjha 6y ago
I believe it's just Base64 encoded DER information, based on the code that seems to be similar: https://github.com/apple-open-source-mirror/Security/blob/70c059a4fd48e34d6a3a2578be3e86d781753b19/OSX/sec/Security/SecCertificate.c#L1218 https://github.com/apple-open-source-mirror/Security/blob/70...
- caf 6y ago
  Yes, that base64 decodes to: OCSP Request Data: Version: 1 (0x0) Requestor List: Certificate ID: Hash Algorithm: sha1 Issuer Name Hash: 3381D1EFDB68B085214D2EEFAF8C4A69643C2A6C Issuer Key Hash: 5717EDA2CFDC7C98A110E0FCBE872D2CF2E31754 Serial Number: 7D86ED91E10A66C2
torstenvl 6y ago
What happens if you edit /private/etc/hosts to point ocsp.apple.com to 0.0.0.0 and flush the DNS cache?
- saagarjha 6y ago
  I think it is fairly likely that your system would not work at all.
- Myrmornis 6y ago
  This seems like an interesting line of inquiry. AIUI doing what you said would permit the network request to proceed, and it would fail because nothing is listening on port 80 [1] We already know that the phone-home bails out when there's no network connection, so perhaps that code also bails out on connection failure? Alternatively, is there some way to make DNS lookup itself fail for ocsp.apple.com? Last resort, if we know how to fake the response, running a dummy server listening on localhost would be faster than allowing the request to go over the internet. [1] Empirically, `curl http://0.0.0.0` http://0.0.0.0` yields a connection failure. I think I know that 0.0.0.0 is used in a listening context to mean "listen on all interfaces" but tbh I don't really know what it means in a sending context. Maybe someone can educate me?
  - IncRnd 6y ago
    Sending to 0.0.0.0 will fail immediately. This differs from sending to 127.0.0.0/8 that may connect to a server on the local machine.
    - Myrmornis 6y ago
      > Sending to 0.0.0.0 will fail immediately. Right, and as far as we know that exception might be caught in the same way as "your computer doesn't have any network connection at all" is caught. Or would those be likely to generate the same exception? Either way, there's a chance that it would result in exec gracefully and quickly not doing the blocking phone-home isn't there?
  - usmannk 6y ago
varenc 6y ago
Here's some shell script to use a random file name and have friendlier output. RAND_FILE="/tmp/test-$RANDOM.sh"; time_helper() { /usr/bin/time $RAND_FILE 2>&1 | tail -1 | awk '{print $1}'; } # this just returns the real run time echo $'#!/bin/sh\necho Hello' $RANDOM > $RAND_FILE && chmod a+x $RAND_FILE; echo "Testing $RAND_FILE"; echo "execution time #1: $(time_helper) seconds"; echo "execution time #2: $(time_helper) seconds"; Introducing a network delay makes the effect much more obvious. Normally I see a delay of about 0.1 seconds, but after using the XCode network link conditioner (pf rules) to add 500ms latency to everything the delay shoots way up to ~2 seconds. example output: Testing /tmp/test-24411.sh execution time #1: 2.32 seconds execution time #2: 0.00 seconds with developer tools checked both executions report "0.0 seconds".
- krferriter 6y ago
  Huh this is crazy. 2 seconds is way slow and this shouldn't involve any network activity. Seems like a real problem.
  - Erlich_Bachman 6y ago
    He/she added an artificial network latency/delay into the config, just like they describe. That is the reason for the delay. It is made artificially long on purpose.
    - maremp 6y ago
      It’s not an unreasonable delay on a slow 3g hotspot. It’s problematic to have the performance tied to the network speed and suffer an overall slow performance because your network happens to be slow.
      - Erlich_Bachman 6y ago
        Have I written anything that is contradicts that? I simply pointed out that in the example the delay was artificial, and it was definitely due to network, not due to something other than network, as the comment suggested.
ignoranceprior 6y ago
Does this mean you can't run a custom shell script without an internet connection?
- usmannk 6y ago
  If the connection fails it goes ahead and grants permission.
moyix 6y ago
Were you able to MITM the api.apple-cloudkit.com connection? I tried with MITMProxy but ran into a client error, which made me think they were doing cert pinning. If you did get it to work could you paste the logs somewhere?
- usmannk 6y ago
  Yes but it looks like there is no actual session, at least for shell scripts that don't have an app bundle ID. There is just an HTTP CONNECT, TLS negotiation, then nothing.
rurban 6y ago
It's called lockdown for a reason. Apple was just the very first to implement centralized binary blacklisting, revocation. They call it notarization. Problem is, that they did it unannounced. There must be really some weird stuff going on in those managers heads. How can they possibly think to go away with that?
- m463 6y ago
  Once you start something, it's hard to stop it. Every software place I've worked gives a special urgency to security stuff. And even if features don't come out regularly, security updates do. This is more of that.
- dagmx 6y ago
  There were announcements about notarization around WWDC last year. They didn't seem to get a lot of media traction however, but there were specific pages detailing what's required from a developer and some basic details on how it would work From April 10, 2019: https://developer.apple.com/news/?id=04102019a https://developer.apple.com/news/?id=04102019a https://developer.apple.com/documentation/xcode/notarizing_macos_software_before_distribution https://developer.apple.com/documentation/xcode/notarizing_m...
  - rurban 6y ago
    For each and every shell or perl script that I create and use privately? No, certainly not.
- kevinh456 6y ago
  There was nothing "unannounced" about it. Notarization was introduced at WWDC 2018 and announced as required at WWDC 2019. Every macOS developer should have been aware of this requirement. It was a special project for my apps.
  - ghayes 6y ago
    I believe the concern here is that this is affecting not just macOS developers, but all developers who use macOS. That's an important distinction.
pinopinopino 6y ago
God, this shit makes me laugh. Why are they doing this. But from Edit2: Your hash is some sort of base64 let str = "ME4wTKADAgEAMEUwQzBBMAkGBSsOAwIaBQAEFDOB0e_baLCFIU0u76+MSmlkPCpsBBRXF+2iz9x8mKEQ4Py+hy0s8uMXVAIIfYbtkeEKZsI=" Then we see weird random gaps in the alphabet used, not so weird, because not every character will be used in every string: Prelude Data.List> map head $ group $ sort $ str "+0246789=ABCDEFGIKLMOPQRSTUVXYZ_abefghiklmpstuwxyz" If we fill these up then: Prelude Data.List> let xs = "+0123456789=ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz" Prelude Data.List> length xs 65 So base64 with some non standard symbols. I don't know what standard base64 is supposed to look to be honest, so perhaps it is standard base64. The = is definitely padding.
- saagarjha 6y ago
  It decodes cleanly as base64.
Darkstryder 6y ago
I'm surprised nobody mentioned that Windows Defender does something very similar (checking for never-seen-before binaries at runtime, uploading them to Microsoft servers, then running them there) : https://news.ycombinator.com/item?id=21180019 https://news.ycombinator.com/item?id=21180019

jwatte 6y ago

Let's assume that sending network packets to verify the trustworthiness of commands is a good idea. (It may not be, but that's a different discussion.) If you have a modern OS with sufficient virtualization and containerization and indirection, you could optimistically let the commands run, and not commit the side effects of the command until you get back a result. Create little write logged mini branches of your file system, and only actually pause when someone else wants to inspect your side effects. By then, an asynchronous check should have gotten back to you.

kccqzy 6y ago

OCSP is Online Certificate Status Protocol, generally used for checking the revocation status of certificates. You used to be able to turn it off in keychain access, but that ability went away in recent macOS releases.

VonGuard 6y ago
Ah, Apple. When you can no longer innovate, just start removing features and call it simplicity...
- D-Coder 6y ago
  Feature-removal has been the most aggravating part of my Mac life for the past several years. Admittedly I tend to use unusual features, but it's just another PITA when they go away.
- throwaway851 6y ago
  Another way to look at it is that Apple is making it harder to run the system in an insecure fashion. You may not agree with that decision, but I certainly appreciate how Apple is looking out for the safety and security of the user. Tangent: as much as some developers hate that the only way to distribute apps for the iPhone is through the App Store, as a user I consider that walled garden of apps to be a real security benefit. When John Gruber says “If you must use Zoom or simply want to use it, I highly recommend using it on your iPad and iPhone only. The iOS version is sandboxed and reviewed by the App Store.” There’s a reason why he can say things like that and it’s because Apple draws a hard line in the sand that not everyone will be happy with.
  - 43920 6y ago
    Wouldn't a sandboxed Zoom downloaded directly from them be equally secure?
    - Retric 6y ago
      Apple’s rejected a huge number of App updates for security reasons. It’s not a huge benefit, but it does exist.
      - cliffsteele 6y ago
        And also allowed a jailbreak app in the iOS App Store. Yes, it only happened once (that I know of), but it still shows you can't really be oblivious to their practices.

deleted 6y ago

[deleted]

markandrewj 6y ago

The isn't specific to the article, but another place that can be interesting to look at system activity on Mac OS is the console. https://support.apple.com/en-ca/guide/console/cnslbf30b61a/mac https://support.apple.com/en-ca/guide/console/cnslbf30b61a/m...

usmannk 6y ago

I can't edit anymore but it seems like the OCSP link could potentially be a red herring just checking the cert for the next request to https://api.apple-cloudkit.com/ https://api.apple-cloudkit.com/. It's worth looking further!

saagarjha 6y ago

I believe it's just Base64 encoded DER information, based on the code that seems to be similar: https://github.com/apple-open-source-mirror/Security/blob/70c059a4fd48e34d6a3a2578be3e86d781753b19/OSX/sec/Security/SecCertificate.c#L1218 https://github.com/apple-open-source-mirror/Security/blob/70...

caf 6y ago
Yes, that base64 decodes to: OCSP Request Data: Version: 1 (0x0) Requestor List: Certificate ID: Hash Algorithm: sha1 Issuer Name Hash: 3381D1EFDB68B085214D2EEFAF8C4A69643C2A6C Issuer Key Hash: 5717EDA2CFDC7C98A110E0FCBE872D2CF2E31754 Serial Number: 7D86ED91E10A66C2

torstenvl 6y ago

What happens if you edit /private/etc/hosts to point ocsp.apple.com to 0.0.0.0 and flush the DNS cache?

saagarjha 6y ago
I think it is fairly likely that your system would not work at all.
Myrmornis 6y ago
This seems like an interesting line of inquiry. AIUI doing what you said would permit the network request to proceed, and it would fail because nothing is listening on port 80 [1] We already know that the phone-home bails out when there's no network connection, so perhaps that code also bails out on connection failure? Alternatively, is there some way to make DNS lookup itself fail for ocsp.apple.com? Last resort, if we know how to fake the response, running a dummy server listening on localhost would be faster than allowing the request to go over the internet. [1] Empirically, `curl http://0.0.0.0` http://0.0.0.0` yields a connection failure. I think I know that 0.0.0.0 is used in a listening context to mean "listen on all interfaces" but tbh I don't really know what it means in a sending context. Maybe someone can educate me?
- IncRnd 6y ago
  Sending to 0.0.0.0 will fail immediately. This differs from sending to 127.0.0.0/8 that may connect to a server on the local machine.
  - Myrmornis 6y ago
    > Sending to 0.0.0.0 will fail immediately. Right, and as far as we know that exception might be caught in the same way as "your computer doesn't have any network connection at all" is caught. Or would those be likely to generate the same exception? Either way, there's a chance that it would result in exec gracefully and quickly not doing the blocking phone-home isn't there?
- usmannk 6y ago

varenc 6y ago

Here's some shell script to use a random file name and have friendlier output. RAND_FILE="/tmp/test-$RANDOM.sh"; time_helper() { /usr/bin/time $RAND_FILE 2>&1 | tail -1 | awk '{print $1}'; } # this just returns the real run time echo $'#!/bin/sh\necho Hello' $RANDOM > $RAND_FILE && chmod a+x $RAND_FILE; echo "Testing $RAND_FILE"; echo "execution time #1: $(time_helper) seconds"; echo "execution time #2: $(time_helper) seconds"; Introducing a network delay makes the effect much more obvious. Normally I see a delay of about 0.1 seconds, but after using the XCode network link conditioner (pf rules) to add 500ms latency to everything the delay shoots way up to ~2 seconds. example output: Testing /tmp/test-24411.sh execution time #1: 2.32 seconds execution time #2: 0.00 seconds with developer tools checked both executions report "0.0 seconds".

krferriter 6y ago
Huh this is crazy. 2 seconds is way slow and this shouldn't involve any network activity. Seems like a real problem.
- Erlich_Bachman 6y ago
  He/she added an artificial network latency/delay into the config, just like they describe. That is the reason for the delay. It is made artificially long on purpose.
  - maremp 6y ago
    It’s not an unreasonable delay on a slow 3g hotspot. It’s problematic to have the performance tied to the network speed and suffer an overall slow performance because your network happens to be slow.
    - Erlich_Bachman 6y ago
      Have I written anything that is contradicts that? I simply pointed out that in the example the delay was artificial, and it was definitely due to network, not due to something other than network, as the comment suggested.

ignoranceprior 6y ago

Does this mean you can't run a custom shell script without an internet connection?

usmannk 6y ago
If the connection fails it goes ahead and grants permission.

moyix 6y ago

Were you able to MITM the api.apple-cloudkit.com connection? I tried with MITMProxy but ran into a client error, which made me think they were doing cert pinning. If you did get it to work could you paste the logs somewhere?

usmannk 6y ago
Yes but it looks like there is no actual session, at least for shell scripts that don't have an app bundle ID. There is just an HTTP CONNECT, TLS negotiation, then nothing.

rurban 6y ago

It's called lockdown for a reason. Apple was just the very first to implement centralized binary blacklisting, revocation. They call it notarization. Problem is, that they did it unannounced. There must be really some weird stuff going on in those managers heads. How can they possibly think to go away with that?

m463 6y ago
Once you start something, it's hard to stop it. Every software place I've worked gives a special urgency to security stuff. And even if features don't come out regularly, security updates do. This is more of that.
dagmx 6y ago
There were announcements about notarization around WWDC last year. They didn't seem to get a lot of media traction however, but there were specific pages detailing what's required from a developer and some basic details on how it would work From April 10, 2019: https://developer.apple.com/news/?id=04102019a https://developer.apple.com/news/?id=04102019a https://developer.apple.com/documentation/xcode/notarizing_macos_software_before_distribution https://developer.apple.com/documentation/xcode/notarizing_m...
- rurban 6y ago
  For each and every shell or perl script that I create and use privately? No, certainly not.
kevinh456 6y ago
There was nothing "unannounced" about it. Notarization was introduced at WWDC 2018 and announced as required at WWDC 2019. Every macOS developer should have been aware of this requirement. It was a special project for my apps.
- ghayes 6y ago
  I believe the concern here is that this is affecting not just macOS developers, but all developers who use macOS. That's an important distinction.

pinopinopino 6y ago

God, this shit makes me laugh. Why are they doing this. But from Edit2: Your hash is some sort of base64 let str = "ME4wTKADAgEAMEUwQzBBMAkGBSsOAwIaBQAEFDOB0e_baLCFIU0u76+MSmlkPCpsBBRXF+2iz9x8mKEQ4Py+hy0s8uMXVAIIfYbtkeEKZsI=" Then we see weird random gaps in the alphabet used, not so weird, because not every character will be used in every string: Prelude Data.List> map head $ group $ sort $ str "+0246789=ABCDEFGIKLMOPQRSTUVXYZ_abefghiklmpstuwxyz" If we fill these up then: Prelude Data.List> let xs = "+0123456789=ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz" Prelude Data.List> length xs 65 So base64 with some non standard symbols. I don't know what standard base64 is supposed to look to be honest, so perhaps it is standard base64. The = is definitely padding.

saagarjha 6y ago
It decodes cleanly as base64.

Darkstryder 6y ago

I'm surprised nobody mentioned that Windows Defender does something very similar (checking for never-seen-before binaries at runtime, uploading them to Microsoft servers, then running them there) : https://news.ycombinator.com/item?id=21180019 https://news.ycombinator.com/item?id=21180019