5 ms·

It really feels like Apple does not have a real plan now and are just trying to catch up.

by chucke1992 2y ago

nxobject 2y ago
Yes – or a use case. Apple seems very happy to advertise that they keep their inference on-device.
- ein0p 2y ago
  There’s no way to do freeform chat on device. On-device models can realistically be used for constrained tasks like summarization, RAG, email/instant message replies, that kind of thing. You aren’t going to get anything GPT4 like unless your “device” can fit a model with at least tens, if not hundreds of billions of parameters. Which, to be fair, a number of higher end Apple SKUs can.
  - easton 2y ago
    I think Apple’s said before that they don’t really get the chat usecase, their priority is integrating ML backed features into the OS (text highlighting in images, etc). It seems that even the state of the art models can’t be used to make a more reliable Siri without another few breakthroughs, so it’s probable that these new AI features aren’t chat stuff.
  - anon373839 2y ago
    Oh, I think we’ve passed that point already. Llama 3 8B works incredibly well for chat. On the LMSYS English-only leaderboard, it’s only 3 points below GPT-4-0314. (!) It will hallucinate, as every other LLM will also do, so it shouldn’t be used as a store of knowledge. But grounded with context, it is astonishing in its capability. For that matter, Phi-3 at half the size is not bad at all.
    - ein0p 2y ago
      Yeah, if you’re OK with it making everything up, it “works incredibly well”. I’m not ok with that, so I actually prefer models where hallucination is minimized as much as possible because they encode massive amounts of knowledge. Points don’t matter. Of all SOTA models, GPT4, the largest one, hallucinates the least. Almost not at all, in fact.
      - anon373839 2y ago
        It doesn’t “make everything up”. On a hallucination benchmark [1] Llama 3 8B has a hallucination rate of 5.4%, compared to GPT4 at 3%, and GPT-4 Turbo at 2.5%. In absolute terms these just aren’t huge differences. [1] https://huggingface.co/spaces/vectara/Hallucination-evaluation-leaderboard https://huggingface.co/spaces/vectara/Hallucination-evaluati...
        ein0p 2y ago
        Benchmarks don’t tell the whole story. Obscure questions are unlikely to be in benchmarks. Large models simply encode more knowledge, which means that they are _way_ more likely to not hallucinate an answer to practical questions that are off the beaten path and that the model hasn’t seen before. Try asking your Llama 8B about some less commonly known details in some topic you know well, and then repeat the same with GPT4. Even Claude Opus hallucinates way more than GPT4, despite the claims that it matches it according to the various leaderboards and whatever.
        anon373839 2y ago
        I agree that larger models have more memorization capacity, but that isn't what find exciting about LLMs, personally. You can ask GPT-4 some esoteric question and get back an answer that's more likely to be truthful than Llama 3's, but either way, it's the same result: you have to look it up for yourself. What I think is much more interesting about LLMs is their ability to transform natural language inputs. Applications like open-book question answering, summarization, data extraction, and so forth. For these things, small models are at the point where they are commercially useful now. And if you do fine-tune them on your own domain, they will work better than larger general-purpose models.

It really feels like Apple does not have a real plan now and are just trying to catch up.

nxobject 2y ago

Yes – or a use case. Apple seems very happy to advertise that they keep their inference on-device.

ein0p 2y ago
There’s no way to do freeform chat on device. On-device models can realistically be used for constrained tasks like summarization, RAG, email/instant message replies, that kind of thing. You aren’t going to get anything GPT4 like unless your “device” can fit a model with at least tens, if not hundreds of billions of parameters. Which, to be fair, a number of higher end Apple SKUs can.
- easton 2y ago
  I think Apple’s said before that they don’t really get the chat usecase, their priority is integrating ML backed features into the OS (text highlighting in images, etc). It seems that even the state of the art models can’t be used to make a more reliable Siri without another few breakthroughs, so it’s probable that these new AI features aren’t chat stuff.
- anon373839 2y ago
  Oh, I think we’ve passed that point already. Llama 3 8B works incredibly well for chat. On the LMSYS English-only leaderboard, it’s only 3 points below GPT-4-0314. (!) It will hallucinate, as every other LLM will also do, so it shouldn’t be used as a store of knowledge. But grounded with context, it is astonishing in its capability. For that matter, Phi-3 at half the size is not bad at all.
  - ein0p 2y ago
    Yeah, if you’re OK with it making everything up, it “works incredibly well”. I’m not ok with that, so I actually prefer models where hallucination is minimized as much as possible because they encode massive amounts of knowledge. Points don’t matter. Of all SOTA models, GPT4, the largest one, hallucinates the least. Almost not at all, in fact.
    - anon373839 2y ago
      It doesn’t “make everything up”. On a hallucination benchmark [1] Llama 3 8B has a hallucination rate of 5.4%, compared to GPT4 at 3%, and GPT-4 Turbo at 2.5%. In absolute terms these just aren’t huge differences. [1] https://huggingface.co/spaces/vectara/Hallucination-evaluation-leaderboard https://huggingface.co/spaces/vectara/Hallucination-evaluati...
      - ein0p 2y ago
        Benchmarks don’t tell the whole story. Obscure questions are unlikely to be in benchmarks. Large models simply encode more knowledge, which means that they are _way_ more likely to not hallucinate an answer to practical questions that are off the beaten path and that the model hasn’t seen before. Try asking your Llama 8B about some less commonly known details in some topic you know well, and then repeat the same with GPT4. Even Claude Opus hallucinates way more than GPT4, despite the claims that it matches it according to the various leaderboards and whatever.
        anon373839 2y ago
        I agree that larger models have more memorization capacity, but that isn't what find exciting about LLMs, personally. You can ask GPT-4 some esoteric question and get back an answer that's more likely to be truthful than Llama 3's, but either way, it's the same result: you have to look it up for yourself. What I think is much more interesting about LLMs is their ability to transform natural language inputs. Applications like open-book question answering, summarization, data extraction, and so forth. For these things, small models are at the point where they are commercially useful now. And if you do fine-tune them on your own domain, they will work better than larger general-purpose models.