3 ms·

> When I entered a credit card number and a random username / password into a Windows Notepad window, Recall captured it, despite the fact that I had text such

by Terr_ 2y ago

> When I entered a credit card number and a random username / password into a Windows Notepad window, Recall captured it, despite the fact that I had text such as “Capital One Visa” right next to the numbers. That undesirable outcome doesn't surprise me at all. Even if someone coded up logic to look for surrounding clues that X is a secret, that other data ("X is a password") might only become available seconds or weeks later. For the foreseeable future, these idiot-savant systems (especially with append-only autocomplete at their core) will continue to be smart enough to get into trouble but not smart enough to get out of it.

dangus 2y ago
I don’t even know if this is a negative aspect of what Recall is intended to do. The whole point of Recall seems to be to allow you to…recall previous work you were doing. It reminds me a lot of whole-device backups. If you have some secrets in plain view then the backup itself is going to need to be secured. From the Microsoft website: > To use Recall you need to opt in to saving snapshots, which are screenshots of your activity. Snapshots and the contextual information derived from them are saved and encrypted to your local hard drive. Recall does not share snapshots or associated data with Microsoft or third parties, nor is it shared between different Windows users on the same device. Windows will ask for your permission before saving snapshots. You are always in control, and you can delete snapshots, pause or turn them off at any time. Any future options for the user to share data will require fully informed explicit action by the user.
- iAMkenough 2y ago
  It's doing what Microsoft intends, but not what customers intend. Entering my credit card or social security details in a secure form should not result in it being copied to a different folder on my local storage or added to my backups without my approval. Snapshots present a new way of circumventing data protection implemented by other apps and websites (if it was visible on your screen, that data is now copied somewhere else). Yes you can delete snapshots AFTER you've determined something sensitive has been inappropriately captured, but you are not warned or prompted to do so and at that point why even use the feature.
  - Terr_ 2y ago
    > what customers intend This customer, at least, is firmly entrenched into "if I wanted you to remember it I'd tell you so", which actually describes many everyday activities like saving a document with a filename, clicking "save" in my password-manager, making a browser bookmark, etc. In contrast, any assistant that is "constantly looking over my shoulder" (human or AI) to implicitly remember things needs to be trustworthy to (a) have my best-interests at heart, and (b) to understand what it's seeing and what my intent is. I don't think the current bleeding-edge is ready to provide both of those. P.S.: An interesting exception might be automatically-kept data like bash history or browser-history, however I feel those are very different from "anything I type anywhere", since each is a narrow and regimented kind of data/purpose. Both also have "forgetting" rules as a required feature. > but you are not warned or prompted And perhaps not even capable of noticing the problem, such as where a semi-transparent border-decor of a window overlaps the secret password text just enough that you can't see it but an aggressive OCR proess can pick it out. More generally, I fear a world where someone uses a botnet to make a funny picture go viral, and the picture contains subtle pixels to make each victim's Artificial Stupidity Assistant exfiltrate all their passwords back to the attacker.
    - dangus 2y ago
      Do you think you would describe yourself as above or below average in technical ability? For example, do you know what terms like "3-2-1 backup" and "swap file" mean? If you answered yes to either, I don't think you are the primary customer. You are the type of person who will turn this off and move on with their day. I think this argument you are making is essentially the same that a person would make for preferring a manual transmission car instead of automatic. But we all know that 90% of people just want the car to go when the right pedal is pressed down, they don't really care about their ability to tell the car exactly what to do. Personally I think the average person would be very excited to have a feature that can recall things they were working on or looking at that they forgot to save. I don't know how many times I've had to tell the non-tech inclined "sorry you didn't save it, it's gone." As far as capturing sensitive information, I'm not really sure it's all that much worse than anything else the non-technical people do. I think at all these attack vectors are no worse than someone convincing you to install conventional malware. Both require you to elevate permissions and dismiss strong warnings.
jazzyjackson 2y ago
Well it's just kind of silly that Microsoft is silo'd enough that they didn't just use data loss prevention that they've implemented elsewhere in the office stack. Microsoft outlook stops me from sending bank wire details over email. With a DLP policy, you can identify, monitor, and automatically protect sensitive items across: Microsoft 365 services such as Teams, Exchange, SharePoint, and OneDrive accounts Office applications such as Word, Excel, and PowerPoint Windows 10, Windows 11, and macOS (three latest released versions) endpoints non-Microsoft cloud apps on-premises file shares and on-premises SharePoint Fabric and Power BI workspaces Microsoft 365 Copilot (preview) DLP detects sensitive items by using deep content analysis, not by just a simple text scan. Content is analyzed: For primary data matches to keywords By the evaluation of regular expressions By internal function validation By secondary data matches that are in proximity to the primary data match DLP also uses machine learning algorithms and other methods to detect content that matches your DLP policies https://learn.microsoft.com/en-us/purview/dlp-learn-about-dlp https://learn.microsoft.com/en-us/purview/dlp-learn-about-dl...

by Terr_ 2y ago

dangus 2y ago

I don’t even know if this is a negative aspect of what Recall is intended to do. The whole point of Recall seems to be to allow you to…recall previous work you were doing. It reminds me a lot of whole-device backups. If you have some secrets in plain view then the backup itself is going to need to be secured. From the Microsoft website: > To use Recall you need to opt in to saving snapshots, which are screenshots of your activity. Snapshots and the contextual information derived from them are saved and encrypted to your local hard drive. Recall does not share snapshots or associated data with Microsoft or third parties, nor is it shared between different Windows users on the same device. Windows will ask for your permission before saving snapshots. You are always in control, and you can delete snapshots, pause or turn them off at any time. Any future options for the user to share data will require fully informed explicit action by the user.

iAMkenough 2y ago
It's doing what Microsoft intends, but not what customers intend. Entering my credit card or social security details in a secure form should not result in it being copied to a different folder on my local storage or added to my backups without my approval. Snapshots present a new way of circumventing data protection implemented by other apps and websites (if it was visible on your screen, that data is now copied somewhere else). Yes you can delete snapshots AFTER you've determined something sensitive has been inappropriately captured, but you are not warned or prompted to do so and at that point why even use the feature.
- Terr_ 2y ago
  > what customers intend This customer, at least, is firmly entrenched into "if I wanted you to remember it I'd tell you so", which actually describes many everyday activities like saving a document with a filename, clicking "save" in my password-manager, making a browser bookmark, etc. In contrast, any assistant that is "constantly looking over my shoulder" (human or AI) to implicitly remember things needs to be trustworthy to (a) have my best-interests at heart, and (b) to understand what it's seeing and what my intent is. I don't think the current bleeding-edge is ready to provide both of those. P.S.: An interesting exception might be automatically-kept data like bash history or browser-history, however I feel those are very different from "anything I type anywhere", since each is a narrow and regimented kind of data/purpose. Both also have "forgetting" rules as a required feature. > but you are not warned or prompted And perhaps not even capable of noticing the problem, such as where a semi-transparent border-decor of a window overlaps the secret password text just enough that you can't see it but an aggressive OCR proess can pick it out. More generally, I fear a world where someone uses a botnet to make a funny picture go viral, and the picture contains subtle pixels to make each victim's Artificial Stupidity Assistant exfiltrate all their passwords back to the attacker.
  - dangus 2y ago
    Do you think you would describe yourself as above or below average in technical ability? For example, do you know what terms like "3-2-1 backup" and "swap file" mean? If you answered yes to either, I don't think you are the primary customer. You are the type of person who will turn this off and move on with their day. I think this argument you are making is essentially the same that a person would make for preferring a manual transmission car instead of automatic. But we all know that 90% of people just want the car to go when the right pedal is pressed down, they don't really care about their ability to tell the car exactly what to do. Personally I think the average person would be very excited to have a feature that can recall things they were working on or looking at that they forgot to save. I don't know how many times I've had to tell the non-tech inclined "sorry you didn't save it, it's gone." As far as capturing sensitive information, I'm not really sure it's all that much worse than anything else the non-technical people do. I think at all these attack vectors are no worse than someone convincing you to install conventional malware. Both require you to elevate permissions and dismiss strong warnings.

jazzyjackson 2y ago

Well it's just kind of silly that Microsoft is silo'd enough that they didn't just use data loss prevention that they've implemented elsewhere in the office stack. Microsoft outlook stops me from sending bank wire details over email. With a DLP policy, you can identify, monitor, and automatically protect sensitive items across: Microsoft 365 services such as Teams, Exchange, SharePoint, and OneDrive accounts Office applications such as Word, Excel, and PowerPoint Windows 10, Windows 11, and macOS (three latest released versions) endpoints non-Microsoft cloud apps on-premises file shares and on-premises SharePoint Fabric and Power BI workspaces Microsoft 365 Copilot (preview) DLP detects sensitive items by using deep content analysis, not by just a simple text scan. Content is analyzed: For primary data matches to keywords By the evaluation of regular expressions By internal function validation By secondary data matches that are in proximity to the primary data match DLP also uses machine learning algorithms and other methods to detect content that matches your DLP policies https://learn.microsoft.com/en-us/purview/dlp-learn-about-dlp https://learn.microsoft.com/en-us/purview/dlp-learn-about-dl...