6 ms·

Also, I'm not aware that the image to text scan (Vision framework) in iOS 15 and macOS (user-accessible in macOS 12, but requiring some third party frontend in

Also, I'm not aware that the image to text scan (Vision framework) in iOS 15 and macOS (user-accessible in macOS 12, but requiring some third party frontend in Big Sur) connects to a remote service. As for end-to-end encryption breaking image search, as I understood it, it was meant to locally match images to a set of encrypted hashes of known offending images before sending them to the iCloud and would warn the user of any conflicting images (also locally). (Personally, I think this may be just a step too far, but it's not implemented as represented in the article.) Finally, any biometric hashes are stored in a secure enclave and are not accessible by the "normal" machine.

CubsFan1060 5y ago
FWIW, I just checked, and live text can be used with no network connection.
- masswerk 5y ago
  Also checked on Big Sur using a tiny menu bar add-on as a frontend. No connection needed. It's using a local neural network as said in the Vision framework docs.
walterbell 5y ago
OCR is local, but the resulting text can be indexed by "Siri & Search" (e.g. "Learn from this app" and "Show content in search"), which default on. Is OCR text stored alongside photos on the file system? E.g. can text be backed up along with photos, or is photo text recorded in OS storage that is unaccessible to the user?
- masswerk 5y ago
  As I know it, as an enduser service, the input is a selected portion of the screen display (much like a screenshot), meaning, there shouldn't come any association with this. However, the framework is a general service available to about any app (probably requiring some sort of consent [I'm not an app developer]) and an app may use this for embedding or associating data. So availability to Siri indexing and/or the question of trusting an app regarding its data management shouldn't be different from what applies to any other stored content. Mind that there is no obligation to activate Siri.
  - walterbell 5y ago
    Text in photos saved to the Camera Roll can be searched by Spotlight. Even with Siri disabled, the option for "Siri & Search" defaults to on for all apps, including indexing of app data. This isn't specific to OCR, so is an existing issue with Apple's harvesting of data in iOS apps. But at least it can be turned off, one app at a time, with about 5 clicks per app = 100 clicks for 20 apps.
    - masswerk 5y ago
      If you use all these services, well… Personally, I'm running happily without iCloud, Siri, and with Spotlight reduced to a mere file search. Also, I dislike features like Camera Roll. The point being, you can still run macOS and probably wide portions of iOS (which I'm not using much) much like a legacy machine. However, if you're using cloud services (including third party services like Dropbox, etc, esp., if you're a non-US-resident), you're making yourself accessible. But this is by no way exclusive to Apple devices. (Arguably, more services run locally on Apple devices than on others – e.g., I don't think Google translate for images runs locally, requiring you to share both content and endpoint data – and there are more provisions for privacy in iCloud than in most of comparable third party services.)
      - walterbell 5y ago
        Agreed, I don't use iCloud or Siri on iOS/MacOS. On iOS, the default way to "save a photo" is to the Camera Roll, which means automatic OCR. On iOS, there's no way to disable Spotlight system wide, you only have the option to disable indexing of content on a per-app basis. That means hundreds of millions of iOS users are, by default, having all of their photos indexed locally with Siri + Spotlight extracting unknown "Learn From" analytics. This needs more coverage by tech publications, as it's far more invasive than client-side hash matching for illegal content.

CubsFan1060 5y ago

FWIW, I just checked, and live text can be used with no network connection.

masswerk 5y ago
Also checked on Big Sur using a tiny menu bar add-on as a frontend. No connection needed. It's using a local neural network as said in the Vision framework docs.

walterbell 5y ago

OCR is local, but the resulting text can be indexed by "Siri & Search" (e.g. "Learn from this app" and "Show content in search"), which default on. Is OCR text stored alongside photos on the file system? E.g. can text be backed up along with photos, or is photo text recorded in OS storage that is unaccessible to the user?

masswerk 5y ago
As I know it, as an enduser service, the input is a selected portion of the screen display (much like a screenshot), meaning, there shouldn't come any association with this. However, the framework is a general service available to about any app (probably requiring some sort of consent [I'm not an app developer]) and an app may use this for embedding or associating data. So availability to Siri indexing and/or the question of trusting an app regarding its data management shouldn't be different from what applies to any other stored content. Mind that there is no obligation to activate Siri.
- walterbell 5y ago
  Text in photos saved to the Camera Roll can be searched by Spotlight. Even with Siri disabled, the option for "Siri & Search" defaults to on for all apps, including indexing of app data. This isn't specific to OCR, so is an existing issue with Apple's harvesting of data in iOS apps. But at least it can be turned off, one app at a time, with about 5 clicks per app = 100 clicks for 20 apps.
  - masswerk 5y ago
    If you use all these services, well… Personally, I'm running happily without iCloud, Siri, and with Spotlight reduced to a mere file search. Also, I dislike features like Camera Roll. The point being, you can still run macOS and probably wide portions of iOS (which I'm not using much) much like a legacy machine. However, if you're using cloud services (including third party services like Dropbox, etc, esp., if you're a non-US-resident), you're making yourself accessible. But this is by no way exclusive to Apple devices. (Arguably, more services run locally on Apple devices than on others – e.g., I don't think Google translate for images runs locally, requiring you to share both content and endpoint data – and there are more provisions for privacy in iCloud than in most of comparable third party services.)
    - walterbell 5y ago
      Agreed, I don't use iCloud or Siri on iOS/MacOS. On iOS, the default way to "save a photo" is to the Camera Roll, which means automatic OCR. On iOS, there's no way to disable Spotlight system wide, you only have the option to disable indexing of content on a per-app basis. That means hundreds of millions of iOS users are, by default, having all of their photos indexed locally with Siri + Spotlight extracting unknown "Learn From" analytics. This needs more coverage by tech publications, as it's far more invasive than client-side hash matching for illegal content.