5 ms·

I'm running this on my iPhone 13 Pro Max as part of the Test Flight beta, and it's interesting. I don't believe I've ever run anything that's ever pushed my pho

I'm running this on my iPhone 13 Pro Max as part of the Test Flight beta, and it's interesting. I don't believe I've ever run anything that's ever pushed my phone this hard, and you can feel the heat. The text output performance is pretty inconsistent, it was very fast at first but slowed down considerably after a few answers. In terms of quality, it's prone to hallucinations, which is not unexpected based on the size and highly compressed nature of the model. "The marvel is not that the bear dances well, but that the bear dances at all."

junrushao1994 3y ago
Thanks for sharing! It's definitely a bit painstaking to get a real-world LLM running at all on an iPhone due to memory constraint. It's also quite compute-intense as it has 7B parameters, but we are glad that it's generating texts at reasonable speed! The model we are using is a quantized Vicuna-7b, which I believe is one of the best open-sourced models. Hallucination is a problem to all LLMs, but I believe research on model side would gradually alleviate this problem :-)
- Fr0styMatt88 3y ago
  WizardLM-7b would be a fantastic model to try out as well. Though that might be out of date tomorrow (or already!) given how many models are being released at the moment :)
david-gpu 3y ago
> I don't believe I've ever run anything that's ever pushed my phone this hard, and you can feel the heat. The text output performance is pretty inconsistent, it was very fast at first but slowed down considerably after a few answers. Those two events are causally related. The OS has to throttle down the CPU or else it will overheat and malfunction. It is one of the reasons why heavy number crunching is often performed on the cloud instead.
- capableweb 3y ago
  In my experience, heavy number crunching is more suitable to run on dedicated machines, rather than virtualized "cloud" cores. More consistent performance, no noisy neighbors and cheaper in the long term.
  - halflings 3y ago
    > cheaper in the long term Citation needed :)) There are economies of scale and various optimizations that are just not possible with dedicated machines. Anecdotal evidence: a company I worked in had a dedicated DC with hundreds/thousands of machines that mostly ran SQL queries on petabytes of data (any query would take ~5-30 minutes). Eye-watering budget and whole teams to maintain the cluster... They switched to GCP/BigQuery, got queries that ran in seconds at a fraction of the budget.
    - slimsag 3y ago
      The inverse is also true; I don't have to pay Google or AWS to deploy my software to dedicated consumer devices. The user already paid for it.
      - deleted 3y ago
        [deleted]
    - Negitivefrags 3y ago
      This anecdote is the opposite of the ones I usually hear. Economies of scale go in the other direction usually.
      - taneq 3y ago
        Those economies are for the cloud provider, not you. The cost to you is set at just less than their best guess at what you think self-hosting will cost you. Cloud hosting only makes sense in a case that your capacity needs are changing rapidly and unpredictably, or the case where you’re so big that the cloud hosting company is effectively a department. Any other time it will almost certainly be cheaper to self host.
kiratp 3y ago
You can do Stable Diffusion on your phone too, fully local. https://apps.apple.com/app/id6444050820 https://apps.apple.com/app/id6444050820
andrewaylett 3y ago
I'd suggest that everything output by an LLM is a hallucination, but it often enough hallucinates something that resembles reality.
- strudey 3y ago
  True for our minds too I think ;)
  - BoorishBears 3y ago
    It honestly worries me when I see people paint the whole "everything is a hallucination" angle as uniquely applying LLMs... If you're going to be that loose with your definition of hallucination, I'd really hope you apply it to yourself. That's a level of introspection that you need to avoid letting biases that all of us have deep down end up affecting your higher order function. We humans rely on a ball of biases, interpolations and extrapolations to function, LLMs don't have a monopoly on that.

junrushao1994 3y ago

Thanks for sharing! It's definitely a bit painstaking to get a real-world LLM running at all on an iPhone due to memory constraint. It's also quite compute-intense as it has 7B parameters, but we are glad that it's generating texts at reasonable speed! The model we are using is a quantized Vicuna-7b, which I believe is one of the best open-sourced models. Hallucination is a problem to all LLMs, but I believe research on model side would gradually alleviate this problem :-)

Fr0styMatt88 3y ago
WizardLM-7b would be a fantastic model to try out as well. Though that might be out of date tomorrow (or already!) given how many models are being released at the moment :)

david-gpu 3y ago

> I don't believe I've ever run anything that's ever pushed my phone this hard, and you can feel the heat. The text output performance is pretty inconsistent, it was very fast at first but slowed down considerably after a few answers. Those two events are causally related. The OS has to throttle down the CPU or else it will overheat and malfunction. It is one of the reasons why heavy number crunching is often performed on the cloud instead.

capableweb 3y ago
In my experience, heavy number crunching is more suitable to run on dedicated machines, rather than virtualized "cloud" cores. More consistent performance, no noisy neighbors and cheaper in the long term.
- halflings 3y ago
  > cheaper in the long term Citation needed :)) There are economies of scale and various optimizations that are just not possible with dedicated machines. Anecdotal evidence: a company I worked in had a dedicated DC with hundreds/thousands of machines that mostly ran SQL queries on petabytes of data (any query would take ~5-30 minutes). Eye-watering budget and whole teams to maintain the cluster... They switched to GCP/BigQuery, got queries that ran in seconds at a fraction of the budget.
  - slimsag 3y ago
    The inverse is also true; I don't have to pay Google or AWS to deploy my software to dedicated consumer devices. The user already paid for it.
    - deleted 3y ago
      [deleted]
  - Negitivefrags 3y ago
    This anecdote is the opposite of the ones I usually hear. Economies of scale go in the other direction usually.
    - taneq 3y ago
      Those economies are for the cloud provider, not you. The cost to you is set at just less than their best guess at what you think self-hosting will cost you. Cloud hosting only makes sense in a case that your capacity needs are changing rapidly and unpredictably, or the case where you’re so big that the cloud hosting company is effectively a department. Any other time it will almost certainly be cheaper to self host.

kiratp 3y ago

You can do Stable Diffusion on your phone too, fully local. https://apps.apple.com/app/id6444050820 https://apps.apple.com/app/id6444050820

andrewaylett 3y ago

I'd suggest that everything output by an LLM is a hallucination, but it often enough hallucinates something that resembles reality.

strudey 3y ago
True for our minds too I think ;)
- BoorishBears 3y ago
  It honestly worries me when I see people paint the whole "everything is a hallucination" angle as uniquely applying LLMs... If you're going to be that loose with your definition of hallucination, I'd really hope you apply it to yourself. That's a level of introspection that you need to avoid letting biases that all of us have deep down end up affecting your higher order function. We humans rely on a ball of biases, interpolations and extrapolations to function, LLMs don't have a monopoly on that.