6 ms·

This is brilliant. "Hey we have this gazillion flop/s available, shall we try to put it to some good use?" - "Nah, let's just waste them recovering information

This is brilliant. "Hey we have this gazillion flop/s available, shall we try to put it to some good use?" - "Nah, let's just waste them recovering information that was lost because it's so much simpler to share a screenshot instead of text or even html." Sometimes, I wonder if the hardware industry sponsors a certain sunset of software developers to make sure every possible performance gain is immediately wasted. Electron was a strong hint, but this is a whole new level...

koprulusector 4y ago
I think they actually have a point. Parsing HTML isn’t trivial: aside from bad/invalid HTML (think: missing opening/closing tags, quotes, etc), there’s also a lot of content that requires javascript to render in the first place, for example, which means the page needs to be rendered and have access to window and DOM, etc. Then, of course, you have embedded objects, such as iframes, or that aren’t text that traditional parsing can’t easily identify/extract, even if everything is rendered OK. For example, a video, animations, interactivity, etc. Another contrived example which illustrates the point would be a site that uses images for buttons/links rather than text, or even non-semantic HTML such as a <div> and click handlers as "links", or buttons with <a> tags. I can think of several other examples. A "screenshot API" enables automation to capture "picture of a group of people celebrating" distinct from advertisements that might also appear on the page without the need to dig through CSS selectors/classes, domain name/query parameters, and handles cases where the image might be base64 embedded directly. Another example might be to simply/easily extract data from a Google Sheets/Excel table, which itself might include embedded images or non-text/HTML objects. Such an API could help accessibility by enabling screen readers for sites that weren't built to be accessible. I learned from this post that you can tap/copy objects from photos in iOS 16! I just did this for the first time and it’s CRAZY, using a photo of my basement and a bunch of shoe boxes, tools, etc. I pressure tapped the group of shoe boxes and hit copy, pasted into the Bear app (a markdown editor for Apple devices), and it pasted just the shoe boxes, perfectly clipped around the edges as if I’d used photoshop! I think ultimately, the point is that such an API which uses object detection, image-to-text, sentiment analysis, etc., on the backend could make trivial the tasks and edge cases that today require non-trivial effort and time, and could enrich the data prior to its retrieval.
- pwdisswordfish9 4y ago
  > Parsing HTML isn’t trivial: aside from bad/invalid HTML (think: missing opening/closing tags, quotes, etc), there’s also a lot of content that requires javascript to render in the first place, for example, which means the page needs to be rendered and have access to window and DOM, etc. Double standard. If you're going to make a fair comparison, then you need to compare like with like; you need to compare the subset of things about e.g. HTML that give you what you can also get with a screenshot. It makes no sense to hold the performance penalty of script execution against browser runtimes when (a) you don't have to execute any scripts to effect anything that gives you parity with a static image, and (b) you can't with static images do anything like what executable scripts enable. And whether or not parsing HTML is trivial (which is debatable), it's still not strictly greater than the computational resources that are needed for the kind of computer vision and widgetry that lets you e.g. select the text in a screenshot...
TechBro8615 4y ago
Perhaps you could think of it like operating at a different abstraction layer, a sort of overlay network for social communication. It's easier to screenshot new technology than it is to make it interoperable with old technology, so there will always be a lead time where screenshots are the dominant sharing method driving a new hype cycle. Look at ChatGPT for example - they launched with no share feature, and people were sharing screenshots of their conversations. It's probably still the most common method of sharing them, even after OpenAI added a share feature. Screenshots have the advantage of working everywhere and being cheap to produce. The loss of information is a solvable problem. In fact if you take a screenshot on an iPhone, there is already quite a bit of information stored along with the photo, but it's removed when using the WebKit <file> input to upload a photo to a website. Also, Safari has a "screenshot full page" option, meaning it already has a specific implementation of the screenshotting for its specific use case of taking a picture of a likely text-heavy Web page (which the same program taking the screenshot already rendered for the user anyway). On the receiving side, Apple's automatic device-local OCR, which already runs on every photo in your camera roll, could increase fidelity of information in a screenshot that you receive. Apple could add a feature for your device to locate the original URL of a screenshot you receive, using both the metadata of screenshots generated on other Apple devices (if shared), and the results of searching the Web for text in an image. This feature could have high accuracy for the limited cases where the content is available on the public web rather than on some ephemeral screen inside an app. But in fact, such cases are even more of a reason that screenshotting will remain a dominant way of storing and sharing information. They're completely decentralized and interoperable, with acceptable tradeoffs that double as advantages - like robustness (screenshot of content will survive longer than a server will continue serving that content), and privacy (the server of the original content doesn't get notified when you read it in a screenshot).

koprulusector 4y ago

I think they actually have a point. Parsing HTML isn’t trivial: aside from bad/invalid HTML (think: missing opening/closing tags, quotes, etc), there’s also a lot of content that requires javascript to render in the first place, for example, which means the page needs to be rendered and have access to window and DOM, etc. Then, of course, you have embedded objects, such as iframes, or that aren’t text that traditional parsing can’t easily identify/extract, even if everything is rendered OK. For example, a video, animations, interactivity, etc. Another contrived example which illustrates the point would be a site that uses images for buttons/links rather than text, or even non-semantic HTML such as a <div> and click handlers as "links", or buttons with <a> tags. I can think of several other examples. A "screenshot API" enables automation to capture "picture of a group of people celebrating" distinct from advertisements that might also appear on the page without the need to dig through CSS selectors/classes, domain name/query parameters, and handles cases where the image might be base64 embedded directly. Another example might be to simply/easily extract data from a Google Sheets/Excel table, which itself might include embedded images or non-text/HTML objects. Such an API could help accessibility by enabling screen readers for sites that weren't built to be accessible. I learned from this post that you can tap/copy objects from photos in iOS 16! I just did this for the first time and it’s CRAZY, using a photo of my basement and a bunch of shoe boxes, tools, etc. I pressure tapped the group of shoe boxes and hit copy, pasted into the Bear app (a markdown editor for Apple devices), and it pasted just the shoe boxes, perfectly clipped around the edges as if I’d used photoshop! I think ultimately, the point is that such an API which uses object detection, image-to-text, sentiment analysis, etc., on the backend could make trivial the tasks and edge cases that today require non-trivial effort and time, and could enrich the data prior to its retrieval.

pwdisswordfish9 4y ago
> Parsing HTML isn’t trivial: aside from bad/invalid HTML (think: missing opening/closing tags, quotes, etc), there’s also a lot of content that requires javascript to render in the first place, for example, which means the page needs to be rendered and have access to window and DOM, etc. Double standard. If you're going to make a fair comparison, then you need to compare like with like; you need to compare the subset of things about e.g. HTML that give you what you can also get with a screenshot. It makes no sense to hold the performance penalty of script execution against browser runtimes when (a) you don't have to execute any scripts to effect anything that gives you parity with a static image, and (b) you can't with static images do anything like what executable scripts enable. And whether or not parsing HTML is trivial (which is debatable), it's still not strictly greater than the computational resources that are needed for the kind of computer vision and widgetry that lets you e.g. select the text in a screenshot...

TechBro8615 4y ago

Perhaps you could think of it like operating at a different abstraction layer, a sort of overlay network for social communication. It's easier to screenshot new technology than it is to make it interoperable with old technology, so there will always be a lead time where screenshots are the dominant sharing method driving a new hype cycle. Look at ChatGPT for example - they launched with no share feature, and people were sharing screenshots of their conversations. It's probably still the most common method of sharing them, even after OpenAI added a share feature. Screenshots have the advantage of working everywhere and being cheap to produce. The loss of information is a solvable problem. In fact if you take a screenshot on an iPhone, there is already quite a bit of information stored along with the photo, but it's removed when using the WebKit <file> input to upload a photo to a website. Also, Safari has a "screenshot full page" option, meaning it already has a specific implementation of the screenshotting for its specific use case of taking a picture of a likely text-heavy Web page (which the same program taking the screenshot already rendered for the user anyway). On the receiving side, Apple's automatic device-local OCR, which already runs on every photo in your camera roll, could increase fidelity of information in a screenshot that you receive. Apple could add a feature for your device to locate the original URL of a screenshot you receive, using both the metadata of screenshots generated on other Apple devices (if shared), and the results of searching the Web for text in an image. This feature could have high accuracy for the limited cases where the content is available on the public web rather than on some ephemeral screen inside an app. But in fact, such cases are even more of a reason that screenshotting will remain a dominant way of storing and sharing information. They're completely decentralized and interoperable, with acceptable tradeoffs that double as advantages - like robustness (screenshot of content will survive longer than a server will continue serving that content), and privacy (the server of the original content doesn't get notified when you read it in a screenshot).