5 ms·

Apple AI researcher boast on-device model that 'substantially outperforms' GPT-4

42 points by ksec 3y ago

terhechte 3y ago
The full context in the article says that this only applies to the specific task of analysing a screenshot and answering a user prompt based on the information in the screenshot (and some other things). It is fair to say that a small model that can run on the iPhone won't have the world knowledge to compete against GPT4 but it can be better for something like this specific task.
doctorpangloss 3y ago
All I want in my Apple product is to be able to choose my provider for AI services.
- eviks 3y ago
  And you'll be able to get any provider in this paradise as long as it's Siri
- s3p 3y ago
  What? You're more than welcome to download an app, or use any company's website.
  - troupo 3y ago
    And what will that app do? Nothing can answer the queries by default except Siri. There are extremely awkward workarounds by telling Siri to run a shortcut, but they are workarounds, and due to weird tech choices they don't work well: https://twitter.com/dmitriid/status/1640692625597505537 https://twitter.com/dmitriid/status/1640692625597505537
- jonplackett 3y ago
  Id like it if we had a fairly decent model that is Siri that runs locally and privately and without signal. BUT that can call a really powerful model as it sees fit, like ChatGPT plugins. Ideally I can have some choices about this external model, or if Apple can do a deal with OpenAI or Google to power this external model that that could be pretty sweet too.
- markbranly 3y ago
  Not Apple OS integrated but have you used Poe? Allows you to chat with ChatGPT, Gemini, etc all in one place. https://poe.com/ https://poe.com/ (I am completely unaffiliated with them)
- croddin 3y ago
  I'm sure the E.U. will eventually force them to let you replace Siri with another voice assistant if you live in the E.U.
mittermayr 3y ago
I recently noticed Apple stock is struggling quite a bit, and then I see things like this pop up more often recently, where the company (or "someone") talks about what's going to happen and what they're working on internally, rather than actual releases and demonstration of work "done". Would love to see a bit more "walk the walk" from Apple.
- kkzz99 3y ago
  It reads to me like the Google announcements. Platitudes to calm investors down so they can say "we are doing AI too see!". But I think Apple is really struggling, similar to Google. Dropping sales in China doesn't help either...
  - t888 3y ago
    It’s not an announcement, and the headline is a lie that has nothing to do with Apple. Apple published an academic paper, as they have been doing for years, showing a model that objectively outperforms GPT-4 on one narrow test. Nowhere do they claim to outperform GPT-4 in general. Assuming that this is somehow intended to manipulate the stock price is not a rational analysis of what is going on. Also, Apple has operated on a predicable release schedule for a decade now. They have already said that they will be announcing their AI work at WWDC. It’s unclear what anyone is expecting them to do differently or why it would make any sense. As for the stock price - the explanations as to why it’s depressed are twofold - one is regulatory action, and the other is frenzied investors who want to jump on the AI bandwagon. Dropping sales in China is a headline like a stopped clock. Usually wrong, sometimes right, and we only find out which at earnings time. Note that nobody’s AI product is doing great yet, so for the most part it’s all performative, something that Apple doesn’t engage in.
    - troupo 3y ago
      Also, unlike most other companies, Apple is really pushing for ML/AI on device. And while there are serious limitations for that, I really don't like to be beholden to opaque profit-at-all-costs companies like Google or OpenAI or...
ultimoo 3y ago
evals or didn't happen
htrp 3y ago
Section 7 Conclusion and Future Work We show that ReaLM outperforms previous approaches, and performs roughly as well as the state of-the-art LLM today, GPT-4, despite consisting of far fewer parameters, even for onscreen references despite being purely in the textual domain. It also outperforms GPT-4 for domain-specific user utterances, thus making ReaLM an ideal choice for a practical reference resolution system that can exist *on-device without compromising on performance.* ----------------------------- Emphasis mine. As far as I can tell, the model isn't exactly on-device yet.
deleted 3y ago
[deleted]
ChrisArchitect 3y ago
[dupe] Some more discussion: https://news.ycombinator.com/item?id=39901952 https://news.ycombinator.com/item?id=39901952

42 points by ksec 3y ago

The full context in the article says that this only applies to the specific task of analysing a screenshot and answering a user prompt based on the information in the screenshot (and some other things). It is fair to say that a small model that can run on the iPhone won't have the world knowledge to compete against GPT4 but it can be better for something like this specific task.

doctorpangloss 3y ago

All I want in my Apple product is to be able to choose my provider for AI services.

eviks 3y ago
And you'll be able to get any provider in this paradise as long as it's Siri
s3p 3y ago
What? You're more than welcome to download an app, or use any company's website.
- troupo 3y ago
  And what will that app do? Nothing can answer the queries by default except Siri. There are extremely awkward workarounds by telling Siri to run a shortcut, but they are workarounds, and due to weird tech choices they don't work well: https://twitter.com/dmitriid/status/1640692625597505537 https://twitter.com/dmitriid/status/1640692625597505537
jonplackett 3y ago
Id like it if we had a fairly decent model that is Siri that runs locally and privately and without signal. BUT that can call a really powerful model as it sees fit, like ChatGPT plugins. Ideally I can have some choices about this external model, or if Apple can do a deal with OpenAI or Google to power this external model that that could be pretty sweet too.
markbranly 3y ago
Not Apple OS integrated but have you used Poe? Allows you to chat with ChatGPT, Gemini, etc all in one place. https://poe.com/ https://poe.com/ (I am completely unaffiliated with them)
croddin 3y ago
I'm sure the E.U. will eventually force them to let you replace Siri with another voice assistant if you live in the E.U.

mittermayr 3y ago

I recently noticed Apple stock is struggling quite a bit, and then I see things like this pop up more often recently, where the company (or "someone") talks about what's going to happen and what they're working on internally, rather than actual releases and demonstration of work "done". Would love to see a bit more "walk the walk" from Apple.

kkzz99 3y ago
It reads to me like the Google announcements. Platitudes to calm investors down so they can say "we are doing AI too see!". But I think Apple is really struggling, similar to Google. Dropping sales in China doesn't help either...
- t888 3y ago
  It’s not an announcement, and the headline is a lie that has nothing to do with Apple. Apple published an academic paper, as they have been doing for years, showing a model that objectively outperforms GPT-4 on one narrow test. Nowhere do they claim to outperform GPT-4 in general. Assuming that this is somehow intended to manipulate the stock price is not a rational analysis of what is going on. Also, Apple has operated on a predicable release schedule for a decade now. They have already said that they will be announcing their AI work at WWDC. It’s unclear what anyone is expecting them to do differently or why it would make any sense. As for the stock price - the explanations as to why it’s depressed are twofold - one is regulatory action, and the other is frenzied investors who want to jump on the AI bandwagon. Dropping sales in China is a headline like a stopped clock. Usually wrong, sometimes right, and we only find out which at earnings time. Note that nobody’s AI product is doing great yet, so for the most part it’s all performative, something that Apple doesn’t engage in.
  - troupo 3y ago
    Also, unlike most other companies, Apple is really pushing for ML/AI on device. And while there are serious limitations for that, I really don't like to be beholden to opaque profit-at-all-costs companies like Google or OpenAI or...

ultimoo 3y ago

evals or didn't happen

htrp 3y ago

Section 7 Conclusion and Future Work We show that ReaLM outperforms previous approaches, and performs roughly as well as the state of-the-art LLM today, GPT-4, despite consisting of far fewer parameters, even for onscreen references despite being purely in the textual domain. It also outperforms GPT-4 for domain-specific user utterances, thus making ReaLM an ideal choice for a practical reference resolution system that can exist *on-device without compromising on performance.* ----------------------------- Emphasis mine. As far as I can tell, the model isn't exactly on-device yet.

deleted 3y ago

[deleted]

ChrisArchitect 3y ago

[dupe] Some more discussion: https://news.ycombinator.com/item?id=39901952 https://news.ycombinator.com/item?id=39901952