19 ms·

Screenshots as the Universal API

108 points by robinhouston 4y ago

drewbeck 4y ago
If someone wants to build a screenshot-based bookmark/reminder/everything tool with me … holler.
- mehal 4y ago
  How would you design it? What are your thoughts?
  - TedDoesntTalk 4y ago
    One example: take a screenshot of an email or flight schedule. Automatically add all dissolve details to my calendar.
- boffinAudio 4y ago
  Print-to-PDF, build your own local Internet .. best bookmark system ever, literally doesn't require anything to be installed, just put the resulting PDF's in folders and use pdfgrep and ag to your hearts content .. "ls -alF | grep thatarticlesubjectiremembervaguely"
- paulfsheridan 4y ago
  Hey dude, I am working on exactly this, would love to chat if you're free? My email is paul (at) lynq (dot) ai
- rvcdbn 4y ago
  this is also something I’ve been thinking about for a while. I messaged you on IG (I’m semistrict there)
  - paulfsheridan 4y ago
    Hey dude, would you mind shooting me an email too?
- spoiler 4y ago
  The Arc browser has a feature called Easels that does this; I quit like it even though it lacks some polish. I imagine they want to do more with it in the future The browser is in beta and macOS only currently, but a friend recently invited me. E-mail me (in profile) if you'd like an invite to explore what they've done with it. I can give out 5 invites I think, if anyone else wants it. So, first come first served!
  - drewbeck 4y ago
    Arc is great!
- sandkoan 4y ago
  I've actually been building something like this: a desktop app that basically records everything you see and hear, and makes it searchable. Maybe we could team up: my email is govind <dot> gnanakumar <at> outlook <dot> com.
pjc50 4y ago
Kind of the opposite of the semantic web. Not only will people not mark up their data in useful ways, often they're actively hostile to you using or remixing it, but the screenshot will always get through. Unless the display-driver level DRM gets involved.
- afandian 4y ago
  > the screenshot will always get through Dubious. Cloudflare and friends promote a culture of demonizing bots. For this to work you'd need to put as much energy into pretending to be a human as parsing the screenshot. At least with a dedicated API you declare that you aren't hostile to a user who wants to use the website on their own terms.
  - pjc50 4y ago
    > Cloudflare and friends promote a culture of demonizing bots What does this mean? Usually it's the website owners who are hostile to scraping for commercial reasons, or simply bandwidth, to which Cloudflare is the solution; it's not some prejudice which has to be marketed to people.
    - afandian 4y ago
      Here's one example of Cloudflare marketing. https://www.bigmarker.com/innovatus-digital/The-newest-approach-Stopping-bots-without-CAPTCHAs https://www.bigmarker.com/innovatus-digital/The-newest-appro... > We know a third of web traffic comes from bots with their insatiable appetite for attacks. From credential stuffing, to stealing inventory, to price and content scraping, stopping bots is critical to a strong web experience for customers that is not undermined by bots. This seems like classic demonizing tactics to me. Generalizing a whole class of users as an out-group, associating them with negative behaviour, painting them as criminals.
      - pjc50 4y ago
        > Generalizing a whole class of users as an out-group, associating them with negative behaviour Isn't this the reverse: the only criterion for this "out group" is behavior. Specifically that some people use a large number of automated clients to engage in behavior which the site owner regards (rightly or wrongly) as harmful. I can see that you're trying to draw racism analogies but that's not going to work. And ("stealing inventory") sometimes this is at the expense of other customers, who might want to buy at the RRP from the official website rather than have to go through scalpers.
obert 4y ago
https://rewind.ai https://rewind.ai right?
osrec 4y ago
This seems like a last resort API. I would much rather have the information in a more convenient form than a screenshot... Also, some apps (e.g. banking apps), do not allow you to take screenshots on certain devices.
dvh 4y ago
Isn't this what Wolfram alpha uses? Or used?
FartyMcFarter 4y ago
> Screenshots are always available Unfortunately not, certainly not on mobile. I don't know if this is the case on iPhone, but on Android some apps make it really hard or impossible to screenshot them. I think in recent Android versions they can even prevent the screen from getting mirrored to a connected PC / other casting device.
- baxtr 4y ago
  Super easy on iOS.
  - FartyMcFarter 4y ago
    Do you mean super easy to take a screenshot of any app, or super easy for apps to prevent that?
    - denkquer 4y ago
      impossible to prevent
      - baxtr 4y ago
        Not sure. Try screenshoting a Netflix movie
        azinman2 4y ago
        That’s DRM’d video, the one exception.
    - baxtr 4y ago
      I meant easy to do as user. Sorry.
- spoiler 4y ago
  Even on Windows and Mac (don't know about Linux) when browsing, you sometimes can't take screenshot of a portion on the website (only seen this with videos, though). We are not allowed to take a screenshot on a device we own, for content we paid for. What a world.
isodev 4y ago
> Screenshots are always available (similar to the era of web crawling) DRM enters the room… But on a serious note. I like the idea of using screenshots as a form of storage, but a lot of metadata is lost in the process like data hierarchy, data available in different visual states etc.
amelius 4y ago
> Screenshots-as-API solves a few problems They forgot to mention that the website owners may deliberately make it difficult to interface with the HTML directly, e.g. by changing the internal formatting structure every now and then.
- yunwal 4y ago
  At this point, AI also solves this problem with pretty reasonable accuracy. You can feed gpt3.5 some html and ask it to write a python script to parse all of the button text.
amelius 4y ago
I'd love to see an ad-blocker based on this. Perhaps I can even use it on my TV.
detrites 4y ago
An unexpected plot-twist for out-of-work digital artists, they can reboot their careers ...in online security testing.
choeger 4y ago
This is brilliant. "Hey we have this gazillion flop/s available, shall we try to put it to some good use?" - "Nah, let's just waste them recovering information that was lost because it's so much simpler to share a screenshot instead of text or even html." Sometimes, I wonder if the hardware industry sponsors a certain sunset of software developers to make sure every possible performance gain is immediately wasted. Electron was a strong hint, but this is a whole new level...
- koprulusector 4y ago
  I think they actually have a point. Parsing HTML isn’t trivial: aside from bad/invalid HTML (think: missing opening/closing tags, quotes, etc), there’s also a lot of content that requires javascript to render in the first place, for example, which means the page needs to be rendered and have access to window and DOM, etc. Then, of course, you have embedded objects, such as iframes, or that aren’t text that traditional parsing can’t easily identify/extract, even if everything is rendered OK. For example, a video, animations, interactivity, etc. Another contrived example which illustrates the point would be a site that uses images for buttons/links rather than text, or even non-semantic HTML such as a <div> and click handlers as "links", or buttons with <a> tags. I can think of several other examples. A "screenshot API" enables automation to capture "picture of a group of people celebrating" distinct from advertisements that might also appear on the page without the need to dig through CSS selectors/classes, domain name/query parameters, and handles cases where the image might be base64 embedded directly. Another example might be to simply/easily extract data from a Google Sheets/Excel table, which itself might include embedded images or non-text/HTML objects. Such an API could help accessibility by enabling screen readers for sites that weren't built to be accessible. I learned from this post that you can tap/copy objects from photos in iOS 16! I just did this for the first time and it’s CRAZY, using a photo of my basement and a bunch of shoe boxes, tools, etc. I pressure tapped the group of shoe boxes and hit copy, pasted into the Bear app (a markdown editor for Apple devices), and it pasted just the shoe boxes, perfectly clipped around the edges as if I’d used photoshop! I think ultimately, the point is that such an API which uses object detection, image-to-text, sentiment analysis, etc., on the backend could make trivial the tasks and edge cases that today require non-trivial effort and time, and could enrich the data prior to its retrieval.
mavu 4y ago
Hey! Nice! I can now read my corporate emails for the day, because I most certainly already read the dumbest thing I will read today.
- dd36 4y ago
  I disagree. Information is presented to humans visually. If your models can process visual information, it is the most robust of possible solutions.
hardware2win 4y ago
>Permissionless. Many applications won't allow you to export data. Screenshots are always available (similar to the era of web crawling). False. Some pdf readers and games will make your screenshoot black But you can always use your phone PS. I do wonder when they will add some watermark so phones will make the pictures black too
- JodieBenitez 4y ago
  Signal on Android blocks screenshots too.
- thyselius 4y ago
  It’s actually possible to at least make it harder to do screen grabs on iOS. But you can always take a photo of the phone’s screen using your other iPhone :) https://stackoverflow.com/questions/18680028/prevent-screen-capture-in-an-ios-app https://stackoverflow.com/questions/18680028/prevent-screen-...
carlosdp 4y ago
I was skeptical about this concept when I first heard it, until I actually tried some of the latest image models. Now, I’m fully bought in that this is correct. There’s a model you can use on hugging face, for example, where you can feed it any PDF or image of a document, ask it a query (“What is the total of the invoice?”), and it just spits out the right answer. Turns out decades of work trying to make universal data interoperability standards will most likely be replaced by screenshots and images! (Again, this is for inter-app data movement)
- stoned 4y ago
  Do you remember what the model was?
- citrin_ru 4y ago
  From an invoice in a standardized XML format (some countries require this by low) total amount can be extracted in milliseconds if not microseconds. How long it takes to do this from an image?
  - lostmsu 4y ago
    How long would it take you to get your plumber to convert his handwriting into a standardized XML?
- dd36 4y ago
  https://huggingface.co/spaces/ThePixOne/open_domain_qa https://huggingface.co/spaces/ThePixOne/open_domain_qa ?
GTP 4y ago
>"Easier to parse than highly complex layout formats." I disagree, the complexity is just hidden inside the model you have to train. What about a low resource device that could have difficulties in running the model? And how you handle the mistakes that the model will make?
c7b 4y ago
I suppose this is satire? I mean, how would I call such an API to make any changes to the application state? Create an image of the effect that I would like to see? Am I missing here, or is the joke just lost on me? If this is serious and the idea is to use this only for quick and dirty read-only access, I'll stick to my time-tested "select+copy" API for now. Usually does a better job at extracting what I care about (particularly for content longer than one screen page). Yes, app owners can make that harder if they want to, but the same is true for screenshots.
pipingdog 4y ago
I use this technique to build a personal dashboard. Rather than try to scrape data, then come up with a nice presentation for it, I just find a nice representation on the web for the data I want on the dashboard, then use Puppeteer[1] to automatically screenshot the specific DOM element that contains the thing I want. Works like a champ. [1]: https://github.com/puppeteer/puppeteer https://github.com/puppeteer/puppeteer
philjr 4y ago
Talked with a founder building something in this space that was inspired by Matt's post in October. lynq.ai (no affiliation, I just know Paul). I think when you look at this problem originally, you say - that's a bad idea. Why would you take structured data, output it to non-structured format and then have ML parse that. Lots of wasted CPU cycles all around. However, when you think about the complex dynamics of standards around documents, tens of years of digital formats, hundreds of standards, lack of adherence to those standards, proprietary formats, hundreds of years of print and legal documents, the argument is akin to self-driving cars. The state we have today around data & dashboards is a hugely emergent & dynamic system, just like our road transport infrastructure. We are closer to a machine being able to navigate the same way a human can, than we are to one simple (or one set of) standard that work more the way a machine would want to consume. Screenshots as a universal API simply meets the world where it is vs assuming the world is going to change towards something simpler and more elegant. I think part of the problem with how this comes across at first glance is how it's framed. "Screenshots" as an API evokes some dirty feelings for most of us in tech because the format is so unstructured. I think if you think about the idea of building something once that both a human and a machine consume from the same target (the UI), this makes a lot more sense in many ways even if it feels like there's an expensive level of indirection in there. Excited to see what'll happen here over time.
pipingdog 4y ago
I recently saw a post on Mastodon where someone figured out they could build a simple Automation[1] on their iPhone to assist them in avoiding foods that contain allergies. They take a picture of the label, then the automation OCRs the text and searches for $ALLERGEN. It seems the technology for making "screenshot APIs" a less zany proposition is emerging. [1]: https://support.apple.com/guide/iphone/create-an-automation-iph6d50ec543/ios https://support.apple.com/guide/iphone/create-an-automation-...
- pipingdog 4y ago
  Found it: https://fosstodon.org/@ross/109671294741113929 https://fosstodon.org/@ross/109671294741113929
singularity2001 4y ago
I already use it a lot for stupid things that don't let me copy text: Twitter, Reddit on mobile and some keyword protected PDFs. iOS recent builtin OCR is a boon! You don't even have to save it to photos. Sidenote: when will Apple finally organize my screenshots into a folder distinct from my family photos??
- azinman2 4y ago
  There already is a screenshots album in photos (under media types). It’s just mixed in your library if that’s what you meant.
  - singularity2001 4y ago
    Thanks, really helpful! Do you also know of a way to see (media type) "Photos" without those screenshots?
    - azinman2 4y ago
      I don’t think that’s possible?
truculent 4y ago
An existing implementation of this idea is screenotate: https://screenotate.com/ https://screenotate.com/ > Screenotate is an app for macOS and Windows that might help you with your screenshots. Every time you take a screenshot, Screenotate steps in to recognize and save the text inside (using Optical Character Recognition), along with the URL and the title of the place where you took the screenshot (where possible).
1000100_1000101 4y ago
No. Mobile screenshots are a horrendous way to pass information around. My father would send me screenshots of Amazon product listings and suggest I buy the item. There was never a visible URL, and typically no or only a small fraction of the product name. If he'd just sent a link, I could just click it, instead it became an investigation/research project. Better yet was when the listing was obviously not Amazon, and not a site I could identify, or had ever seen before. I'd get zero context clues. Perhaps an AI could figure it out, but again, I'd rather he just sent a link. When I get a screenshot, I'll have to submit it to some service, of which there will be a dozen different services, each requiring an account, each building a profile and tracking me based on whatever crap my dad sent me. No. Just no. Stop. I pre-emptively opt out.

108 points by robinhouston 4y ago

drewbeck 4y ago

If someone wants to build a screenshot-based bookmark/reminder/everything tool with me … holler.

mehal 4y ago
How would you design it? What are your thoughts?
- TedDoesntTalk 4y ago
  One example: take a screenshot of an email or flight schedule. Automatically add all dissolve details to my calendar.
boffinAudio 4y ago
Print-to-PDF, build your own local Internet .. best bookmark system ever, literally doesn't require anything to be installed, just put the resulting PDF's in folders and use pdfgrep and ag to your hearts content .. "ls -alF | grep thatarticlesubjectiremembervaguely"
paulfsheridan 4y ago
Hey dude, I am working on exactly this, would love to chat if you're free? My email is paul (at) lynq (dot) ai
rvcdbn 4y ago
this is also something I’ve been thinking about for a while. I messaged you on IG (I’m semistrict there)
- paulfsheridan 4y ago
  Hey dude, would you mind shooting me an email too?
spoiler 4y ago
The Arc browser has a feature called Easels that does this; I quit like it even though it lacks some polish. I imagine they want to do more with it in the future The browser is in beta and macOS only currently, but a friend recently invited me. E-mail me (in profile) if you'd like an invite to explore what they've done with it. I can give out 5 invites I think, if anyone else wants it. So, first come first served!
- drewbeck 4y ago
  Arc is great!
sandkoan 4y ago
I've actually been building something like this: a desktop app that basically records everything you see and hear, and makes it searchable. Maybe we could team up: my email is govind <dot> gnanakumar <at> outlook <dot> com.

pjc50 4y ago

Kind of the opposite of the semantic web. Not only will people not mark up their data in useful ways, often they're actively hostile to you using or remixing it, but the screenshot will always get through. Unless the display-driver level DRM gets involved.

afandian 4y ago
> the screenshot will always get through Dubious. Cloudflare and friends promote a culture of demonizing bots. For this to work you'd need to put as much energy into pretending to be a human as parsing the screenshot. At least with a dedicated API you declare that you aren't hostile to a user who wants to use the website on their own terms.
- pjc50 4y ago
  > Cloudflare and friends promote a culture of demonizing bots What does this mean? Usually it's the website owners who are hostile to scraping for commercial reasons, or simply bandwidth, to which Cloudflare is the solution; it's not some prejudice which has to be marketed to people.
  - afandian 4y ago
    Here's one example of Cloudflare marketing. https://www.bigmarker.com/innovatus-digital/The-newest-approach-Stopping-bots-without-CAPTCHAs https://www.bigmarker.com/innovatus-digital/The-newest-appro... > We know a third of web traffic comes from bots with their insatiable appetite for attacks. From credential stuffing, to stealing inventory, to price and content scraping, stopping bots is critical to a strong web experience for customers that is not undermined by bots. This seems like classic demonizing tactics to me. Generalizing a whole class of users as an out-group, associating them with negative behaviour, painting them as criminals.
    - pjc50 4y ago
      > Generalizing a whole class of users as an out-group, associating them with negative behaviour Isn't this the reverse: the only criterion for this "out group" is behavior. Specifically that some people use a large number of automated clients to engage in behavior which the site owner regards (rightly or wrongly) as harmful. I can see that you're trying to draw racism analogies but that's not going to work. And ("stealing inventory") sometimes this is at the expense of other customers, who might want to buy at the RRP from the official website rather than have to go through scalpers.

obert 4y ago

https://rewind.ai https://rewind.ai right?

osrec 4y ago

This seems like a last resort API. I would much rather have the information in a more convenient form than a screenshot... Also, some apps (e.g. banking apps), do not allow you to take screenshots on certain devices.

dvh 4y ago

Isn't this what Wolfram alpha uses? Or used?

FartyMcFarter 4y ago

> Screenshots are always available Unfortunately not, certainly not on mobile. I don't know if this is the case on iPhone, but on Android some apps make it really hard or impossible to screenshot them. I think in recent Android versions they can even prevent the screen from getting mirrored to a connected PC / other casting device.

baxtr 4y ago
Super easy on iOS.
- FartyMcFarter 4y ago
  Do you mean super easy to take a screenshot of any app, or super easy for apps to prevent that?
  - denkquer 4y ago
    impossible to prevent
    - baxtr 4y ago
      Not sure. Try screenshoting a Netflix movie
      - azinman2 4y ago
        That’s DRM’d video, the one exception.
  - baxtr 4y ago
    I meant easy to do as user. Sorry.
spoiler 4y ago
Even on Windows and Mac (don't know about Linux) when browsing, you sometimes can't take screenshot of a portion on the website (only seen this with videos, though). We are not allowed to take a screenshot on a device we own, for content we paid for. What a world.

isodev 4y ago

> Screenshots are always available (similar to the era of web crawling) DRM enters the room… But on a serious note. I like the idea of using screenshots as a form of storage, but a lot of metadata is lost in the process like data hierarchy, data available in different visual states etc.

amelius 4y ago

> Screenshots-as-API solves a few problems They forgot to mention that the website owners may deliberately make it difficult to interface with the HTML directly, e.g. by changing the internal formatting structure every now and then.

yunwal 4y ago
At this point, AI also solves this problem with pretty reasonable accuracy. You can feed gpt3.5 some html and ask it to write a python script to parse all of the button text.

amelius 4y ago

I'd love to see an ad-blocker based on this. Perhaps I can even use it on my TV.

detrites 4y ago

An unexpected plot-twist for out-of-work digital artists, they can reboot their careers ...in online security testing.

choeger 4y ago

This is brilliant. "Hey we have this gazillion flop/s available, shall we try to put it to some good use?" - "Nah, let's just waste them recovering information that was lost because it's so much simpler to share a screenshot instead of text or even html." Sometimes, I wonder if the hardware industry sponsors a certain sunset of software developers to make sure every possible performance gain is immediately wasted. Electron was a strong hint, but this is a whole new level...

koprulusector 4y ago
I think they actually have a point. Parsing HTML isn’t trivial: aside from bad/invalid HTML (think: missing opening/closing tags, quotes, etc), there’s also a lot of content that requires javascript to render in the first place, for example, which means the page needs to be rendered and have access to window and DOM, etc. Then, of course, you have embedded objects, such as iframes, or that aren’t text that traditional parsing can’t easily identify/extract, even if everything is rendered OK. For example, a video, animations, interactivity, etc. Another contrived example which illustrates the point would be a site that uses images for buttons/links rather than text, or even non-semantic HTML such as a <div> and click handlers as "links", or buttons with <a> tags. I can think of several other examples. A "screenshot API" enables automation to capture "picture of a group of people celebrating" distinct from advertisements that might also appear on the page without the need to dig through CSS selectors/classes, domain name/query parameters, and handles cases where the image might be base64 embedded directly. Another example might be to simply/easily extract data from a Google Sheets/Excel table, which itself might include embedded images or non-text/HTML objects. Such an API could help accessibility by enabling screen readers for sites that weren't built to be accessible. I learned from this post that you can tap/copy objects from photos in iOS 16! I just did this for the first time and it’s CRAZY, using a photo of my basement and a bunch of shoe boxes, tools, etc. I pressure tapped the group of shoe boxes and hit copy, pasted into the Bear app (a markdown editor for Apple devices), and it pasted just the shoe boxes, perfectly clipped around the edges as if I’d used photoshop! I think ultimately, the point is that such an API which uses object detection, image-to-text, sentiment analysis, etc., on the backend could make trivial the tasks and edge cases that today require non-trivial effort and time, and could enrich the data prior to its retrieval.

mavu 4y ago

Hey! Nice! I can now read my corporate emails for the day, because I most certainly already read the dumbest thing I will read today.

dd36 4y ago
I disagree. Information is presented to humans visually. If your models can process visual information, it is the most robust of possible solutions.

hardware2win 4y ago

>Permissionless. Many applications won't allow you to export data. Screenshots are always available (similar to the era of web crawling). False. Some pdf readers and games will make your screenshoot black But you can always use your phone PS. I do wonder when they will add some watermark so phones will make the pictures black too

JodieBenitez 4y ago
Signal on Android blocks screenshots too.
thyselius 4y ago
It’s actually possible to at least make it harder to do screen grabs on iOS. But you can always take a photo of the phone’s screen using your other iPhone :) https://stackoverflow.com/questions/18680028/prevent-screen-capture-in-an-ios-app https://stackoverflow.com/questions/18680028/prevent-screen-...

carlosdp 4y ago

I was skeptical about this concept when I first heard it, until I actually tried some of the latest image models. Now, I’m fully bought in that this is correct. There’s a model you can use on hugging face, for example, where you can feed it any PDF or image of a document, ask it a query (“What is the total of the invoice?”), and it just spits out the right answer. Turns out decades of work trying to make universal data interoperability standards will most likely be replaced by screenshots and images! (Again, this is for inter-app data movement)

stoned 4y ago
Do you remember what the model was?
citrin_ru 4y ago
From an invoice in a standardized XML format (some countries require this by low) total amount can be extracted in milliseconds if not microseconds. How long it takes to do this from an image?
- lostmsu 4y ago
  How long would it take you to get your plumber to convert his handwriting into a standardized XML?
dd36 4y ago
https://huggingface.co/spaces/ThePixOne/open_domain_qa https://huggingface.co/spaces/ThePixOne/open_domain_qa ?

GTP 4y ago

>"Easier to parse than highly complex layout formats." I disagree, the complexity is just hidden inside the model you have to train. What about a low resource device that could have difficulties in running the model? And how you handle the mistakes that the model will make?

c7b 4y ago

I suppose this is satire? I mean, how would I call such an API to make any changes to the application state? Create an image of the effect that I would like to see? Am I missing here, or is the joke just lost on me? If this is serious and the idea is to use this only for quick and dirty read-only access, I'll stick to my time-tested "select+copy" API for now. Usually does a better job at extracting what I care about (particularly for content longer than one screen page). Yes, app owners can make that harder if they want to, but the same is true for screenshots.

pipingdog 4y ago

I use this technique to build a personal dashboard. Rather than try to scrape data, then come up with a nice presentation for it, I just find a nice representation on the web for the data I want on the dashboard, then use Puppeteer[1] to automatically screenshot the specific DOM element that contains the thing I want. Works like a champ. [1]: https://github.com/puppeteer/puppeteer https://github.com/puppeteer/puppeteer

philjr 4y ago

Talked with a founder building something in this space that was inspired by Matt's post in October. lynq.ai (no affiliation, I just know Paul). I think when you look at this problem originally, you say - that's a bad idea. Why would you take structured data, output it to non-structured format and then have ML parse that. Lots of wasted CPU cycles all around. However, when you think about the complex dynamics of standards around documents, tens of years of digital formats, hundreds of standards, lack of adherence to those standards, proprietary formats, hundreds of years of print and legal documents, the argument is akin to self-driving cars. The state we have today around data & dashboards is a hugely emergent & dynamic system, just like our road transport infrastructure. We are closer to a machine being able to navigate the same way a human can, than we are to one simple (or one set of) standard that work more the way a machine would want to consume. Screenshots as a universal API simply meets the world where it is vs assuming the world is going to change towards something simpler and more elegant. I think part of the problem with how this comes across at first glance is how it's framed. "Screenshots" as an API evokes some dirty feelings for most of us in tech because the format is so unstructured. I think if you think about the idea of building something once that both a human and a machine consume from the same target (the UI), this makes a lot more sense in many ways even if it feels like there's an expensive level of indirection in there. Excited to see what'll happen here over time.

pipingdog 4y ago

I recently saw a post on Mastodon where someone figured out they could build a simple Automation[1] on their iPhone to assist them in avoiding foods that contain allergies. They take a picture of the label, then the automation OCRs the text and searches for $ALLERGEN. It seems the technology for making "screenshot APIs" a less zany proposition is emerging. [1]: https://support.apple.com/guide/iphone/create-an-automation-iph6d50ec543/ios https://support.apple.com/guide/iphone/create-an-automation-...

pipingdog 4y ago
Found it: https://fosstodon.org/@ross/109671294741113929 https://fosstodon.org/@ross/109671294741113929

singularity2001 4y ago

I already use it a lot for stupid things that don't let me copy text: Twitter, Reddit on mobile and some keyword protected PDFs. iOS recent builtin OCR is a boon! You don't even have to save it to photos. Sidenote: when will Apple finally organize my screenshots into a folder distinct from my family photos??

azinman2 4y ago
There already is a screenshots album in photos (under media types). It’s just mixed in your library if that’s what you meant.
- singularity2001 4y ago
  Thanks, really helpful! Do you also know of a way to see (media type) "Photos" without those screenshots?
  - azinman2 4y ago
    I don’t think that’s possible?

truculent 4y ago

An existing implementation of this idea is screenotate: https://screenotate.com/ https://screenotate.com/ > Screenotate is an app for macOS and Windows that might help you with your screenshots. Every time you take a screenshot, Screenotate steps in to recognize and save the text inside (using Optical Character Recognition), along with the URL and the title of the place where you took the screenshot (where possible).

1000100_1000101 4y ago

No. Mobile screenshots are a horrendous way to pass information around. My father would send me screenshots of Amazon product listings and suggest I buy the item. There was never a visible URL, and typically no or only a small fraction of the product name. If he'd just sent a link, I could just click it, instead it became an investigation/research project. Better yet was when the listing was obviously not Amazon, and not a site I could identify, or had ever seen before. I'd get zero context clues. Perhaps an AI could figure it out, but again, I'd rather he just sent a link. When I get a screenshot, I'll have to submit it to some service, of which there will be a dozen different services, each requiring an account, each building a profile and tracking me based on whatever crap my dad sent me. No. Just no. Stop. I pre-emptively opt out.