14 ms·
There are two things to note here: 1. A cross-application connectivity layer that pipes data and actions between apps 2. A natural language interface to contr
by deet 3y ago
There are two things to note here:
1. A cross-application connectivity layer that pipes data and actions between apps
2. A natural language interface to control #1
Thinking about them separately is useful, because although chat is the new UI hotness, #1 is valuable on its own and the two can potentially be deployed separately.
As presented here, I suspect the natural language interface will be faster and easier than buttons for operating the cross-app layer for complex queries, but potentially slower than operating buttons for simple things (like "start dark mode").
But personally, I believe #1 combined with some AI context awareness is more powerful of the features.
...
And btw, I left Apple last year to build a local-first and developer-extensible assistants for the Mac that's pretty similar. If this interests you, would love to chat (email in profile, as well as a waitlist).
- knrz 3y agoI was also thinking about how they could've integrated this into every app, but I figured that simple tasks would be more reliable across all apps if they did AI-recognition on screenshots of the desktop + injecting data like "Installed Apps", then by the end of it output virtual keyboard event or a virtual click at an X/Y coordinate. Just some afternoon postulating haha
- lostmsu 3y ago#1 has existed in Windows and other OSes for ages: accessibility APIs and UI test tooling (called UI Automation in Windows).
- awegio 3y agoIt looks like this is a high level interface more like Intents on Android, e.g. in the video it opens the Logo Templates page of Adobe Express probably without automated UI interaction.