3 ms·

All I can say is, I asked Siri today (verbatim): What is 75 degrees fahrenheit in celsius, and what is 85 degrees in fahrenheit — and it offered a web search ab

All I can say is, I asked Siri today (verbatim): What is 75 degrees fahrenheit in celsius, and what is 85 degrees in fahrenheit — and it offered a web search about fahrenheit. The "and" completely disabled its most basic ability to do metric conversions. So, it's nice to see Apple is doing research and talking about it, but we're out here waiting, still waiting, for anything useful to make of it all on our thousand-dollar devices that literally connect us to the world and contain our entire life data. It's what I would've expected from one of the most valuable companies in the world.

basisword 1y ago
>> What is 75 degrees fahrenheit in celsius, and what is 85 degrees in fahrenheit Probably wouldn't have made a difference but the second half of that statement isn't exactly clear. 85 degrees what? I also think when you're chaining these two separate calculations together you get a problem when it comes to displaying the results.
- bronco21016 1y ago
  Meanwhile users have been conditioned to expect a system that understand the multiple queries and answers them appropriately.
  - JKCalhoun 1y ago
    True. But for most of us, only in the past year. I have a few friends/relatives who have still never conversed with an LLM.
- vosper 1y ago
  That exact phrase "What is 75 degrees fahrenheit in celsius, and what is 85 degrees in fahrenheit" given to ChatGPT produces the correct result (it infers that the second degrees must be Celsius) and ChatGPT gives me a nicely laid out formula for the math of the conversion. So yeah, Apple is way behind on this stuff.
- seydor 1y ago
  the fact is that gemini responds with this: 75 degrees Fahrenheit is 23.89 degrees Celsius, and 85 degrees Celsius is 185.00 degrees Fahrenheit.
losvedir 1y ago
> What is 75 degrees fahrenheit in celsius, and what is 85 degrees in fahrenheit Err, what? As a native English speaker human that's a pretty confusing question to me, too!
- hu3 1y ago
  First, most of the English speaking world is not native. "As of 2022, there were about 400 million native speakers of English. Including people who speak English as a second language, estimates of the total number of Anglophones vary from 1.5 billion to 2 billion." Second, all popular models I tested did well with that query, including Gemini on Android (aka "ok Google"), except Apple's. https://en.m.wikipedia.org/wiki/English-speaking_world https://en.m.wikipedia.org/wiki/English-speaking_world
  - manoweb 1y ago
    I am not sure why you go on the subject of English speaking world etc. Anyway, the models you tested with that query, which I am not sure why we think is a good benchmark, are local models running on a wireless device or they use datacenter and only convey the text back and forth?
    - lxgr 1y ago
      I'm fairly sure Siri still sends user voice samples to a data center. At least for a while, it used to use multipath TCP to decrease latency over multiple available network connections if I'm not misremembering. Some modern Apple devices support "local Siri", but it's a limited subset of both voice recognition performance and capabilities.
manoweb 1y ago
Your usage of Siri today (probably on an old version of iOS) frankly has nothing to do with the article we are discussing. Sorry to say this but it is going to take time. Comparing the performance of a chatgpt running in a big data center with a model running locally on a phone device... give it a few years.
- ninkendo 1y ago
  People have been giving Siri a few years for a decade now. Siri used to run in a data center (and still does for older hardware and things like HomePods) and it has never supported compound queries. Siri needs to be taken out back and shot. The problem with “upgrading” it is the pull to maintain backwards compatibility for every little thing Siri did, which leads them to try and incorporate existing Siri functionality (and existing Siri engineers) to work alongside any LLM. Which leads to disaster, and none of it works and just made it all slower. They’ve been trying to do an LLM assisted Siri for years now and it’s the most public facing disaster the company has had in a while. Time to start over.
  - mrheosuper 1y ago
    Those little things have been broken for a while now, it's best to bite the bullet and integrate LLM to Siri now.
  - lxgr 1y ago
    As a user, I'd gladly opt into a slightly less deeply integrated Siri that understands what I want from it. Build a crude router in front of it, if you must, or give it access to "the old Siri" as a tool it can call, and let the LLM decide whether to return its own or a Siri-generated response! I bet even smaller LLMs would be able to figure out, given a user input and Siri response pair, whether the request was resonably answered or whether the model itself could do better or at least explain that the request is out of capabilities for now.
    - ninkendo 1y ago
      > Build a crude router in front of it, if you must, or give it access to "the old Siri" as a tool it can call, and let the LLM decide whether to return its own or a Siri-generated response! Both of these approaches were tried internally, including even the ability for the LLM to rewrite siri-as-a-tool's response, and none of them shipped, because they all suck. Putting a router in front of it makes multi-turn conversation (when Siri asks for confirmation or disambiguation) a nightmare to implement, and siri-as-a-tool suffers from the same problem. What happens when legacy siri disambiguates? Does the LLM try to guess at an option? Does it proxy the prompt back to the user? What about all the "smart UI" like having a countdown timer with Siri saying "I'll send this" when sending a text message? Does that just pass through? When does the LLM know how/when to intervene in the responses the Siri tool is giving? This was all an integration nightmare and it's the main reason why none of it shipped. (Well, that and the LLM being underwhelming and the on-device models not being smart enough in the first place. It was just a slower, buggier siri without any new features.) The answer is that they need to renege on the entire promise of a "private" siri and admit that the only way they can get the experience they want is a _huge_ LLM running with a _ton_ of user context, in the cloud, and don't hinder it all with backwards compatibility with Siri. Give it a toolbox of things it can do with MCP to your device, bake in the stock tools with LoRA or whatever, and let it figure out the best user experience. If it's a frontier-quality LLM it'll be better than Siri on day one, without Apple having to really do anything other than figure out a good system prompt. The problem is, Apple doesn't want to admit the whole privacy story is a dead-end, so they're going to keep trying to pursue on-device models, and it's going to continue to be underwhelming and "not meeting our quality bar", for the foreseeable future.
al_borland 1y ago
You asked 2 questions in a system made for 1 question at a time. Split these up and Siri answers them fine. You’re holding it wrong.
- a_wild_dandan 1y ago
  "You haven't contorted your comically simple query enough to make the brittle tool work. Throw the chicken bones better next time."
  - al_borland 1y ago
    It’s been this way for over a decade. If someone hasn’t figured it out by now, that’s kind of on them. I’m not even sure why those two things would be asked as a single question. It seems like a very unnatural way to pose those two questions. Most humans would trip on that, especially if it was asked verbally.
    - lxgr 1y ago
      > It seems like a very unnatural way to pose those two questions. Most humans would trip on that I'd assume GP only gave an example. As a pretty frequent user, I can unfortunately only confirm that Siri trips over almost every multi-part question. This would be forgivable if there weren't multiple voice-based AI consumer products available that can handle these kinds of requests perfectly.
      - al_borland 1y ago
        And Apple has integrated one of them, ChatGPT, to do just that. If they wanted an LLM answer they could have got one. They went out of their way just to take shots at Apple.
        lxgr 1y ago
        I can’t talk to ChatGPT hands-free on my Apple devices, but I can to ChatGPT. Besides that, many people don’t install any apps, and Apple not pre-installing a reasonable LLM to cater to that market just seems incredibly out of character. And there’s enough credible reporting and personnel reshuffling happening to suggest that it’s not available yet because they failed to make it work, not because they didn’t try.
Angostura 1y ago
I just tried this on my phone and just got two pop ups with the conversions appear in quick succession.

basisword 1y ago

>> What is 75 degrees fahrenheit in celsius, and what is 85 degrees in fahrenheit Probably wouldn't have made a difference but the second half of that statement isn't exactly clear. 85 degrees what? I also think when you're chaining these two separate calculations together you get a problem when it comes to displaying the results.

bronco21016 1y ago
Meanwhile users have been conditioned to expect a system that understand the multiple queries and answers them appropriately.
- JKCalhoun 1y ago
  True. But for most of us, only in the past year. I have a few friends/relatives who have still never conversed with an LLM.
vosper 1y ago
That exact phrase "What is 75 degrees fahrenheit in celsius, and what is 85 degrees in fahrenheit" given to ChatGPT produces the correct result (it infers that the second degrees must be Celsius) and ChatGPT gives me a nicely laid out formula for the math of the conversion. So yeah, Apple is way behind on this stuff.
seydor 1y ago
the fact is that gemini responds with this: 75 degrees Fahrenheit is 23.89 degrees Celsius, and 85 degrees Celsius is 185.00 degrees Fahrenheit.

losvedir 1y ago

> What is 75 degrees fahrenheit in celsius, and what is 85 degrees in fahrenheit Err, what? As a native English speaker human that's a pretty confusing question to me, too!

hu3 1y ago
First, most of the English speaking world is not native. "As of 2022, there were about 400 million native speakers of English. Including people who speak English as a second language, estimates of the total number of Anglophones vary from 1.5 billion to 2 billion." Second, all popular models I tested did well with that query, including Gemini on Android (aka "ok Google"), except Apple's. https://en.m.wikipedia.org/wiki/English-speaking_world https://en.m.wikipedia.org/wiki/English-speaking_world
- manoweb 1y ago
  I am not sure why you go on the subject of English speaking world etc. Anyway, the models you tested with that query, which I am not sure why we think is a good benchmark, are local models running on a wireless device or they use datacenter and only convey the text back and forth?
  - lxgr 1y ago
    I'm fairly sure Siri still sends user voice samples to a data center. At least for a while, it used to use multipath TCP to decrease latency over multiple available network connections if I'm not misremembering. Some modern Apple devices support "local Siri", but it's a limited subset of both voice recognition performance and capabilities.

manoweb 1y ago

Your usage of Siri today (probably on an old version of iOS) frankly has nothing to do with the article we are discussing. Sorry to say this but it is going to take time. Comparing the performance of a chatgpt running in a big data center with a model running locally on a phone device... give it a few years.

ninkendo 1y ago
People have been giving Siri a few years for a decade now. Siri used to run in a data center (and still does for older hardware and things like HomePods) and it has never supported compound queries. Siri needs to be taken out back and shot. The problem with “upgrading” it is the pull to maintain backwards compatibility for every little thing Siri did, which leads them to try and incorporate existing Siri functionality (and existing Siri engineers) to work alongside any LLM. Which leads to disaster, and none of it works and just made it all slower. They’ve been trying to do an LLM assisted Siri for years now and it’s the most public facing disaster the company has had in a while. Time to start over.
- mrheosuper 1y ago
  Those little things have been broken for a while now, it's best to bite the bullet and integrate LLM to Siri now.
- lxgr 1y ago
  As a user, I'd gladly opt into a slightly less deeply integrated Siri that understands what I want from it. Build a crude router in front of it, if you must, or give it access to "the old Siri" as a tool it can call, and let the LLM decide whether to return its own or a Siri-generated response! I bet even smaller LLMs would be able to figure out, given a user input and Siri response pair, whether the request was resonably answered or whether the model itself could do better or at least explain that the request is out of capabilities for now.
  - ninkendo 1y ago
    > Build a crude router in front of it, if you must, or give it access to "the old Siri" as a tool it can call, and let the LLM decide whether to return its own or a Siri-generated response! Both of these approaches were tried internally, including even the ability for the LLM to rewrite siri-as-a-tool's response, and none of them shipped, because they all suck. Putting a router in front of it makes multi-turn conversation (when Siri asks for confirmation or disambiguation) a nightmare to implement, and siri-as-a-tool suffers from the same problem. What happens when legacy siri disambiguates? Does the LLM try to guess at an option? Does it proxy the prompt back to the user? What about all the "smart UI" like having a countdown timer with Siri saying "I'll send this" when sending a text message? Does that just pass through? When does the LLM know how/when to intervene in the responses the Siri tool is giving? This was all an integration nightmare and it's the main reason why none of it shipped. (Well, that and the LLM being underwhelming and the on-device models not being smart enough in the first place. It was just a slower, buggier siri without any new features.) The answer is that they need to renege on the entire promise of a "private" siri and admit that the only way they can get the experience they want is a _huge_ LLM running with a _ton_ of user context, in the cloud, and don't hinder it all with backwards compatibility with Siri. Give it a toolbox of things it can do with MCP to your device, bake in the stock tools with LoRA or whatever, and let it figure out the best user experience. If it's a frontier-quality LLM it'll be better than Siri on day one, without Apple having to really do anything other than figure out a good system prompt. The problem is, Apple doesn't want to admit the whole privacy story is a dead-end, so they're going to keep trying to pursue on-device models, and it's going to continue to be underwhelming and "not meeting our quality bar", for the foreseeable future.

al_borland 1y ago

You asked 2 questions in a system made for 1 question at a time. Split these up and Siri answers them fine. You’re holding it wrong.

a_wild_dandan 1y ago
"You haven't contorted your comically simple query enough to make the brittle tool work. Throw the chicken bones better next time."
- al_borland 1y ago
  It’s been this way for over a decade. If someone hasn’t figured it out by now, that’s kind of on them. I’m not even sure why those two things would be asked as a single question. It seems like a very unnatural way to pose those two questions. Most humans would trip on that, especially if it was asked verbally.
  - lxgr 1y ago
    > It seems like a very unnatural way to pose those two questions. Most humans would trip on that I'd assume GP only gave an example. As a pretty frequent user, I can unfortunately only confirm that Siri trips over almost every multi-part question. This would be forgivable if there weren't multiple voice-based AI consumer products available that can handle these kinds of requests perfectly.
    - al_borland 1y ago
      And Apple has integrated one of them, ChatGPT, to do just that. If they wanted an LLM answer they could have got one. They went out of their way just to take shots at Apple.
      - lxgr 1y ago
        I can’t talk to ChatGPT hands-free on my Apple devices, but I can to ChatGPT. Besides that, many people don’t install any apps, and Apple not pre-installing a reasonable LLM to cater to that market just seems incredibly out of character. And there’s enough credible reporting and personnel reshuffling happening to suggest that it’s not available yet because they failed to make it work, not because they didn’t try.

Angostura 1y ago

I just tried this on my phone and just got two pop ups with the conversions appear in quick succession.