2 ms·

Any M-series Mac can already run local models with decent performance.

mvdtnz 3y ago
1. Not models that are in any way competitive with things like OpenAI. Not even close. 2. A typical user is not even remotely close to using hardware like that. You live in a bubble.
- theshrike79 3y ago
  #1 Depends on what you're competing with. I, for example, like to use "AI" to assist me in writing NSFW short stories. Can't do that with any of the public ones without hitting the guard rails HARD. Even with extreme prompt massaging, the AI will start to moralise and editorialise the actions so much it's affecting the output. As for #2, MacBooks are so ubiquitous it's a meme on itself. You can go into any classroom or fancy coffee place and I can bet you over 80% of the laptops there have an Apple logo and a good portion of those are new enough to have an M-series processor in it. On the PC side you're right, even a semi-high end Intel/AMD CPU is not powerful enough to be practical for LLMs. You need a dedicated NVidia GPU for that and those are only in "gaming" laptops.
  - PeterStuer 3y ago
    > "AI" to assist me in writing NSFW short stories. Can't do that with any of the public ones without hitting the guard rails HARD It is actually not trivial to have a GPT-2 conversation that does not descend into NSFW. It will eagerly jump to guide you there at even the most remote opportunity.
    - theshrike79 3y ago
      With a local LLM yes, using any of the public services like gemini or OpenAI PGT3.5 or 4, it's currently nearly impossible. It used to be possible to prompt engineer 3.5 to not moralise on every single thing, but can't be done anymore
      - PeterStuer 3y ago
        You can run it on Huggingface https://transformer.huggingface.co/doc/gpt2-large https://transformer.huggingface.co/doc/gpt2-large
nonameiguess 3y ago
I guess it probably won't matter in the long run as hardware requirements continue to get lesser, but the "overwhelming majority of users" don't have anything close to an M-series Mac right now. More like a Galaxy S3, if that.
- theshrike79 3y ago
  I'm predicting that phones will get AI-optimised chips in the next 2-3 years, enough to run specific optimised models on-device. They'll get specific models trained to do things that make sense locally + the ability to fetch data from the cloud if needed.

1. Not models that are in any way competitive with things like OpenAI. Not even close. 2. A typical user is not even remotely close to using hardware like that. You live in a bubble.

theshrike79 3y ago
#1 Depends on what you're competing with. I, for example, like to use "AI" to assist me in writing NSFW short stories. Can't do that with any of the public ones without hitting the guard rails HARD. Even with extreme prompt massaging, the AI will start to moralise and editorialise the actions so much it's affecting the output. As for #2, MacBooks are so ubiquitous it's a meme on itself. You can go into any classroom or fancy coffee place and I can bet you over 80% of the laptops there have an Apple logo and a good portion of those are new enough to have an M-series processor in it. On the PC side you're right, even a semi-high end Intel/AMD CPU is not powerful enough to be practical for LLMs. You need a dedicated NVidia GPU for that and those are only in "gaming" laptops.
- PeterStuer 3y ago
  > "AI" to assist me in writing NSFW short stories. Can't do that with any of the public ones without hitting the guard rails HARD It is actually not trivial to have a GPT-2 conversation that does not descend into NSFW. It will eagerly jump to guide you there at even the most remote opportunity.
  - theshrike79 3y ago
    With a local LLM yes, using any of the public services like gemini or OpenAI PGT3.5 or 4, it's currently nearly impossible. It used to be possible to prompt engineer 3.5 to not moralise on every single thing, but can't be done anymore
    - PeterStuer 3y ago
      You can run it on Huggingface https://transformer.huggingface.co/doc/gpt2-large https://transformer.huggingface.co/doc/gpt2-large

nonameiguess 3y ago

I guess it probably won't matter in the long run as hardware requirements continue to get lesser, but the "overwhelming majority of users" don't have anything close to an M-series Mac right now. More like a Galaxy S3, if that.

theshrike79 3y ago
I'm predicting that phones will get AI-optimised chips in the next 2-3 years, enough to run specific optimised models on-device. They'll get specific models trained to do things that make sense locally + the ability to fetch data from the cloud if needed.