3 ms·

Local LLM models can be many GBs each. One can have many of them.

by realityfactchex 2y ago

wtallis 2y ago
Is that something you actually do with your phone?
- userbinator 2y ago
  Look at all the hype around AI assistants lately. Having those run locally is definitely much better from a privacy perspective.
  - wtallis 2y ago
    Many models of many GB each on a phone with only 8GB of DRAM doesn't make sense. Even on Android phones that have more DRAM than that, there's not enough performance for large models to be useful, let alone multiple models consuming tens of GB of storage.
    - realityfactchex 2y ago
      > Many models of many GB each on a phone with only 8GB of DRAM doesn't make sense. > there's not enough performance for large models to be useful For iPhone, the Private LLM app offers 44 models in its easy-install menu right now (on 15 promax), with avg model size about 2.7 GB. Total GB is ~120 GB to install and compare them all. People could argue about what being useful means, but asking these different models the same question can give quite different (and indeed thought-provokingly different) results. They all have either different foundations or different fine tuning or something. The same prompt gives rather different answers depending on what model you pick. It's fun that way. So to compare their outputs, one would need 120GB free at once, then install them all, or else do some very inefficient and weird dance of uninstall-then-reinstall n times while picking and choosing which ones to try out for a given prompt. And that's just for one offline "gen AI" app. There might be others. This can be done offline, while on a walk, or whatever, without hitting any servers at all, in the palm of one's hand. Performance is good enough on iPhone for a bunch of tasks/prompts. So, IDK. Does part of that not make sense? These models not being run run at once, and don't need to be in RAM at once. They're being stored on disk at once to be used when wanted. They initialize one at a time when in use. The large disk size is for storing the models. The phone can then use the models when wanted. (Granted, it's not the "same" as hitting huge versions of the models sitting on some beefy infra, but we're talking about mobile devices here and actual current uses for having a bunch of local storage available on them.)
      - wtallis 2y ago
        So what I'm hearing is that you don't have a use case for having 44 LLMs on your phone simultaneously, aside from testing different LLMs for the sake of testing different LLMs. There's no way a user would want to keep all 44 LLMs around indefinitely; nobody could possibly remember which one they prefer for which type of query across that many models. I can believe there are uses for having a handful of models around and for trying out new ones when they become available, but I can't see many users bothering to thoroughly evaluate dozens of models at once.
        skeaker 2y ago
        We're getting away from the point. What does it matter what the specific use case is? Who are you to be the arbiter of a valid or invalid use case? This one guy didn't justify himself to fulfill your arbitrary personal requirements; I guess that means nobody should even have the option of removable storage and that we should all be okay with paying an extra few hundred dollars for five more gigabytes of space on our phones (or whatever the insane going rate is now).
        wtallis 2y ago
        > What does it matter what the specific use case is? Recall where the thread started: https://news.ycombinator.com/item?id=41631782 https://news.ycombinator.com/item?id=41631782 A question about what use cases people have for large amounts of storage on phones, aside from photos and videos that are easily offloaded to the cloud. When someone replies that they need tons of storage for LLMs, it's quite reasonable to want to understand where that need comes from, how much storage is actually needed for that use case, and whether it's anything that could become popular enough to influence product design decisions or if it's something that will only ever be done by HN nerds. If the best use cases HN can come up with for more storage on phones are things that are unlikely to go mainstream, then we should be assuming that phone manufacturers aren't going to be feeling much pressure to change their storage strategies. Complaints about product design choices rarely produce interesting discussion when they can be reduced down to the complainer not making the distinction between their personal usage patterns and those of the target market in general. LLMs are still a very new class of workloads for phones, and what an early-adopter HN user is doing with LLMs on a phone could be a preview of an emerging trend that may go mainstream—very on-topic for HN discussion. But in this case it looks more like a use case that will forever remain an outlier. (It's totally okay to be an outlier and do things with your gadgets that hardly anyone else would choose to do or even think of doing.)
- realityfactchex 2y ago
  Yes, and it's awesome.

Local LLM models can be many GBs each. One can have many of them.

wtallis 2y ago

Is that something you actually do with your phone?

userbinator 2y ago
Look at all the hype around AI assistants lately. Having those run locally is definitely much better from a privacy perspective.
- wtallis 2y ago
  Many models of many GB each on a phone with only 8GB of DRAM doesn't make sense. Even on Android phones that have more DRAM than that, there's not enough performance for large models to be useful, let alone multiple models consuming tens of GB of storage.
  - realityfactchex 2y ago
    > Many models of many GB each on a phone with only 8GB of DRAM doesn't make sense. > there's not enough performance for large models to be useful For iPhone, the Private LLM app offers 44 models in its easy-install menu right now (on 15 promax), with avg model size about 2.7 GB. Total GB is ~120 GB to install and compare them all. People could argue about what being useful means, but asking these different models the same question can give quite different (and indeed thought-provokingly different) results. They all have either different foundations or different fine tuning or something. The same prompt gives rather different answers depending on what model you pick. It's fun that way. So to compare their outputs, one would need 120GB free at once, then install them all, or else do some very inefficient and weird dance of uninstall-then-reinstall n times while picking and choosing which ones to try out for a given prompt. And that's just for one offline "gen AI" app. There might be others. This can be done offline, while on a walk, or whatever, without hitting any servers at all, in the palm of one's hand. Performance is good enough on iPhone for a bunch of tasks/prompts. So, IDK. Does part of that not make sense? These models not being run run at once, and don't need to be in RAM at once. They're being stored on disk at once to be used when wanted. They initialize one at a time when in use. The large disk size is for storing the models. The phone can then use the models when wanted. (Granted, it's not the "same" as hitting huge versions of the models sitting on some beefy infra, but we're talking about mobile devices here and actual current uses for having a bunch of local storage available on them.)
    - wtallis 2y ago
      So what I'm hearing is that you don't have a use case for having 44 LLMs on your phone simultaneously, aside from testing different LLMs for the sake of testing different LLMs. There's no way a user would want to keep all 44 LLMs around indefinitely; nobody could possibly remember which one they prefer for which type of query across that many models. I can believe there are uses for having a handful of models around and for trying out new ones when they become available, but I can't see many users bothering to thoroughly evaluate dozens of models at once.
      - skeaker 2y ago
        We're getting away from the point. What does it matter what the specific use case is? Who are you to be the arbiter of a valid or invalid use case? This one guy didn't justify himself to fulfill your arbitrary personal requirements; I guess that means nobody should even have the option of removable storage and that we should all be okay with paying an extra few hundred dollars for five more gigabytes of space on our phones (or whatever the insane going rate is now).
        wtallis 2y ago
        > What does it matter what the specific use case is? Recall where the thread started: https://news.ycombinator.com/item?id=41631782 https://news.ycombinator.com/item?id=41631782 A question about what use cases people have for large amounts of storage on phones, aside from photos and videos that are easily offloaded to the cloud. When someone replies that they need tons of storage for LLMs, it's quite reasonable to want to understand where that need comes from, how much storage is actually needed for that use case, and whether it's anything that could become popular enough to influence product design decisions or if it's something that will only ever be done by HN nerds. If the best use cases HN can come up with for more storage on phones are things that are unlikely to go mainstream, then we should be assuming that phone manufacturers aren't going to be feeling much pressure to change their storage strategies. Complaints about product design choices rarely produce interesting discussion when they can be reduced down to the complainer not making the distinction between their personal usage patterns and those of the target market in general. LLMs are still a very new class of workloads for phones, and what an early-adopter HN user is doing with LLMs on a phone could be a preview of an emerging trend that may go mainstream—very on-topic for HN discussion. But in this case it looks more like a use case that will forever remain an outlier. (It's totally okay to be an outlier and do things with your gadgets that hardly anyone else would choose to do or even think of doing.)
realityfactchex 2y ago
Yes, and it's awesome.