4 ms·
"GPT‑5.4 interprets screenshots of a browser interface and interacts with UI elements through coordinate-based clicking to send emails and schedule a calendar e
by mattas 7mo ago
"GPT‑5.4 interprets screenshots of a browser interface and interacts with UI elements through coordinate-based clicking to send emails and schedule a calendar event."
They show an example of 5.4 clicking around in Gmail to send an email.
I still think this is the wrong interface to be interacting with the internet. Why not use Gmail APIs? No need to do any screenshot interpretation or coordinate-based clicking.
- TheAceOfHearts 7mo agoI think the desire is that in the long-term AI should be able to use any human-made application to accomplish equivalent tasks. This email demo is proof that this capability is a high priority.
- spongebobstoes 7mo agonot everything has an API, or API use is limited. some UIs are more feature complete than their APIs some sites try to block programmatic use UI use can be recorded and audited by a non-technical person
- Jacques2Marais 7mo agoI guess a big chunk of their target market won't know how to use APIs.
- satvikpendem 7mo agoThe ideal of REST, the HTML and UI is the API.
- PaulHoule 7mo agoAPIs have never been a gift but rather have always been a take-away that lets you do less than you can with the web interface. It’s always been about drinking through a straw, paying NASA prices, and being limited in everything you can do. But people are intimidated by the complexity of writing web crawlers because management has been so traumatized by the cost of making GUI applications that they couldn’t believe how cheap it is to write crawlers and scrapers…. Until LLMs came along, and changed the perceived economics and created a permission structure. [1] AI is a threat to the “enshittification economy” because it lets us route around it. [1] that high cost of GUI development is one reason why scrapers are cheap… there is a good chance that the scraper you wrote 8 years ago still works because (a) they can’t afford to change their site and (b) if they could afford to change their site changing anything substantial about it is likely to unrecoverably tank their Google rankings so they won’t. A.I. might change the mechanics of that now that you Google traffic is likely to go to zero no matter what you do.
- disqard 7mo ago> AI is a threat to the “enshittification economy” because it lets us route around it. This is prescient -- I wonder if the Big Tech entities see it this way. Maybe, even if they do, they're 100% committed to speedrunning the current late-stage-cap wave, and therefore unable to do anything about it.
- PaulHoule 7mo agoThey are not a single thing. Google has a good model in the form of Gemini and they might figure they can win the AI race and if the web dies, the web dies. YouTube will still stick around. Facebook is not going to win the AI race with low I.Q. Llama but Zuck believed their business was cooked around the time it became a real business because their users would eventually age out and get tired of it. If I was him I'd be investing in anything that isn't cybernetic let it be gold bars or MMA studios. Microsoft? They bought Activision for $69 billion. I just can't explain their behavior rationally but they could do worse than their strategy of "put ChatGPT in front of laggards and hope that some of them rise to the challenge and become slop producers." Amazon is really a bricks-and-mortar play which has the freedom to invest in bricks-and-mortar because investors don't think they are a bricks-and-mortar play. Netflix? They're cooked as is all of Hollywood. Hollywood's gatekeeping-industrial strategy of producing as few franchise as possible will crack someday and our media market may wind up looking more like Japan, where somebody can write a low-rent light novel like https://en.wikipedia.org/wiki/Backstabbed_in_a_Backwater_Dungeon https://en.wikipedia.org/wiki/Backstabbed_in_a_Backwater_Dun... and J.C. Staff makes a terrible anime that convinces 20k Otaku to drop $150 on the light novels and another $150 on the manga (sorry, no way you can make a balanced game based on that premise!) and the cost structure is such that it is profitable.
- lostmsu 7mo ago> AI is a threat to the “enshittification economy” because it lets us route around it. I am not sure about that. We techies avoid enshittification because we recognize shit. Normies will just get their syncopatic enshittified AI that will tell them to continue buying into walled gardens.
- 7mo ago
- steve1977 7mo agoOne could argue that LLMs learning programming languages made for humans (i.e. most of them) is using the wrong interface as well. Why not use machine code?
- embedding-shape 7mo agoWhy would human language by the wrong interface when they're literally language models? Why would machine code be better when there is probably magnitude less of training material with machine code? You can also test this yourself easily, fire up two agents, ask one to use PL meant for humans, and one to write straight up machine code (or assembly even), and see which results you like best.
- BoredPositron 7mo agobecause they are inherently text based as is code?
- steve1977 7mo agoBut they are abstractions made to cater to human weaknesses.
- falkensmaize 7mo agoSo you want LLMs to write a bunch of black box code that humans won’t be able to read and reason about easily? That will definitely end well.
- Harvy 7mo agoIsn't that what LLMs are?
- falkensmaize 7mo agoNot if you can review the code.
- jstummbillig 7mo agoBecause the web and software more generally if full of not APIs and you do, in fact, need the clicking to work to make agents work generally
- modeless 7mo agoA world where AIs use APIs instead of UIs to do everything is a world where us humans will soon be helpless, as we'll have to ask the AIs to do everything for us and will have limited ability to observe and understand their work. I prefer that the AIs continue to use human-accessible tools, even if that's less efficient for them. As the price of intelligence trends toward zero, efficiency becomes relatively less important.
- npilk 7mo agoIt feels like building humanoid robots so they can use tools built for human hands. Not clear if it will pay off, but if it does then you get a bunch of flexibility across any task "for free". Of course APIs and CLIs also exist, but they don't necessarily have feature parity, so more development would be needed. Maybe that's the future though since code generation is so good - use AI to build scaffolding for agent interaction into every product.
- packetlost 7mo agoI don't see how an API couldn't have full parity with a web interface, the API is how you actually trigger a state transition in the vast majority of cases
- oliwary 7mo agoI think it's akin to self driving cars prioritizing nornal roads rather than implementing new infrastructure. Tricky, but if you get it right the whole world opens up, since you don't depend on others to adapt your system.
- coffeemug 7mo agoA model that gets good at computer use can be plugged in anywhere you have a human. A model that gets good at API use cannot. From the standpoint of diffusion into the economy/labor market, computer use is much higher value.
- f0e4c2f7 7mo agoLots of services have no desire to ever expose an API. This approach lets you step right over that. If an API is exposed you can just have the LLM write something against that.
- deleted 7mo ago[deleted]
- kristianp 7mo agoThis opens up a new question: how does bot detection work when the bot is using the computer via a gui?
- itintheory 7mo agoOn it's face, I'm not sure that's a new question. Bots using browser automation frameworks (puppeteer, selenium, playwright etc) have been around for a while. There are signals used in bot detection tools like cursor movement speed, accuracy, keyboard timing, etc. How those detection tools might update to support legitimate bot users does seem like an open question to me though.
- MattDaEskimo 7mo agoSame reason why Wikipedia deals with so many people scraping its web page instead of using their API: Optimizations are secondary to convenience
- bottlepalm 7mo agoThe vast majority of websites you visit don’t have usable APIs and very poor discovery of the those APIs. Screenshots on the other hand are documentation, API, and discovery all in one. And you’d be surprised how little context/tokens screenshots consumer compared to all the back and forth verbose json payloads of APIs
- LUmBULtERA 7mo ago>The vast majority of websites you visit don’t have usable APIs and very poor discovery of the those APIs. I think an important thing here is that a lot of websites/platforms don't want AIs to have direct API access, because they are afraid that AIs would take the customer "away" from the website/platform, making the consumer a customer of the AI rather than a customer of the website/platform. Therefore for AIs to be able to do what customers want them to do, they need their browsing to look just like the customer's browsing/browser.
- Gigachad 7mo agoAlso the fact that they don't want automated abuse. At this point a lot of services might just go app only so they can have a verified compute environment that is difficult to bot.
- bottlepalm 7mo agoThat's true, and it's always been like that, which is why the comment that AI should be using APIs is already dead in the water. In terms of gating a websites to humans by not providing APIs, that is quickly coming to a close.
- time0ut 7mo agoLowest common denominator.
- keyle 7mo agoThe 'AI' endgame is a robot that sits in your seat and does all of your tasks.
- sagarpatil 7mo agoOr CLI.