5 ms·

Is anyone blown away by how fast we got to running something this powerful locally? I know it's easy to get burnt out on llms but this is pretty incredible. I

Is anyone blown away by how fast we got to running something this powerful locally? I know it's easy to get burnt out on llms but this is pretty incredible. I genuinely think we're only 2 years away from full custom local voice to voice llm assistants that grow with you like JOI in BR2049 and it's going to change how we think about being human and being social, and how we grow up.

simonw 2y ago
It's incredible. I've been experimenting with running local LLMs for nearly two years now, ever since the first LLaMA release back in March 2023. About six months ago I had mostly lost interest in them. They were fun to play around with but the quality difference between the ones I could run on my MacBook and the ones I could access via an online API felt insurmountable. This has completely changed in the second half of 2024. The models I can run locally had a leap in quality - they feel genuinely GPT-4 class now. They're not as good as the best hosted models (GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet) but they're definitely good enough to be extremely useful. This started with the Qwen 2 and 2.5 series, but I also rate Llama 3.3 70B and now Phi-4 as GPT-4 class models that run on my laptop. I wrote more about this here: https://simonwillison.net/2024/Dec/31/llms-in-2024/#some-of-those-gpt-4-models-run-on-my-laptop https://simonwillison.net/2024/Dec/31/llms-in-2024/#some-of-...
- OhioMan2943 2y ago
  I'm in complete agreement with your more recent timeline piece (the negative one), and as a younger user (22 year old student) I'm actively relocating this year to somewhere slightly more rural with a focus on physical/knowledge combined work to secure a good quality of life nearly solely because of how fast our timelines are. A 'word calculator' this effective is the best substitute that we have for a logic calculator. And the fact that it's enough in 90% of situations is terrifying as it is transformative, as is the fact no one is awake to it. Exponential power scaling in an unstable world feels like it only makes it exponentially more unstable though.
  - simonw 2y ago
    I should emphasize that I really don't think the dystopian version of this is likely to happen - the one where "AGI/ASI" puts every human out of work and society collapses. Human beings have agency, and we are very good at rolling with the punches. We've survived waves of automation for hundreds of years. I'm much more confident that we will continue to find ways to use these things as tools that elevate us, not replace us. I really hope the dystopian version doesn't come to pass!
    - actsasbuffoon 2y ago
      I’m on the opposite end of the spectrum. I’m almost certain that this is going to end extremely badly for the majority of humanity, and for programmers in particular. I think there’s a less than 5% chance that this goes well, and that’s only if we get a series of things to go extremely well. And frankly, we’re tracking along the extremely bad path so far.
      - OhioMan2943 2y ago
        We barely survived one nuclear arms race, and this could give every nation state a new type of power weapon every 5ish years through the inevitable scaling in energy and weapons. I agree we're on one of the worst timelines for AI/AGI/ASI with the world actively being run into the ground by short sighted 'dementia-ocracies' and every security risk about to increase dramatically.
    - imiric 2y ago
      > We've survived waves of automation for hundreds of years. I'm much more confident that we will continue to find ways to use these things as tools that elevate us, not replace us. The difference with past technological breakthroughs is that they augmented what humans could do, but didn't have the potential to replace human labor altogether as AI does. They were disruptive, but humans were able to adapt to new career paths as they became available. Lamplighters were replaced by electrical lighting, but that created jobs for electrical engineers. Carriage drivers were replaced by car drivers; human computers by programmers, and so on. The invention of AI is a tipping point for technology. The only jobs where human labor will be valued over machine labor are those that machines are not good at yet, and those where human creativity is a key component. And the doors are quickly closing on both of those as well. Realistically, the only jobs that will have some form of longevity (~20 years?) are those of humans that build and program AI machines. But eventually even those will be better accomplished by other AI machines. So, I'm really curious why you see AI as the same kind of technology we've invented before, and why you're so confident that humanity will be able to overcome the key existential problems AI introduces, which we haven't even begun to address. I don't see myself as a pessimist, but can't help noticing that we're careening towards a future we're not prepared to handle.
SamPatt 2y ago
Yes, and for image and video generation too. Hunyuan (open source video) has been remarkable. Flux dev makes some incredible images. The fact that it's still only going to get better from here is hard to imagine.
api 2y ago
I’ve thought for a while that Joi in BR2049 was less dystopian than what we will probably do with AI. She doesn’t constantly prompt K to buy more credits (like a mobile game) to continue engaging with her or deepen their relationship. (“If you really love me…”) I’ve been expecting that this is how our industry would operate given the customer hostile psychologically abusive hellscape of social and mobile. Of course there’s still time. She appears to be a local model runnable on a small device without cloud.
- brookst 2y ago
  I expect AI to be like any other tech: some fantastic uses that advance humanity and improve the world, some terrible uses that abuse, manipulate, oppress. I don’t see anything in the tech that indicates a singular pattern that will be “good” or “bad”.
  - ericrosedev 2y ago
    The enshitification trend seems pretty dominant, and pretty bad for users / good for investors
cloudking 2y ago
Not related to local LLMs, but JOI from BR2049 is essentially what Replika is striving for: https://replika.com/ https://replika.com/ Infact during the onboarding process they ask the user to choose which AI companion movie they related to the most: Her, BR2049 or Ex-Machina. The experience is then tailored to align closer to the movie chosen. It's quite a terrible app from a product design perspective: filled with dark patterns (like sending the user blurred images to "unlock*) and upsells, but it's become successful amongst the masses that have adopted it, which I find fascinating. 30m+ users https://en.wikipedia.org/wiki/Replika#:~:text=Replika%20became%20available%20to%20the,users%20had%20surpassed%2030%20million https://en.wikipedia.org/wiki/Replika#:~:text=Replika%20beca....
mark_l_watson 2y ago
I am blown away: a year ago I bought a M2 32G Mac to run local models. It seems like what I can run locally now just one year later is 10x more useful for NLP, data wrangling, RAG, experimenting with agents, etc. BTW, a few days ago I published a book on using Ollama. Here is a link to read it online https://leanpub.com/ollama/read https://leanpub.com/ollama/read
- hn8726 2y ago
  Which models do you recommend for that amount of memory?
  - rpastuszak 2y ago
    I asked the same question a few days back and I'm keeping the responses here: https://bsky.app/profile/potato.horse/post/3lejngewfmc2n https://bsky.app/profile/potato.horse/post/3lejngewfmc2n
  - mark_l_watson 2y ago
    For reasoning: qwq:latest (19G file) For coding: qwen2.5-coder:14b (9G file) Misc. experiments, runs fast: llama3.2:latest )2 G file)
yeahwhatever10 2y ago
How can a model "grow with you"? Do current models do this other than adding the full conversation to the context window?

simonw 2y ago

It's incredible. I've been experimenting with running local LLMs for nearly two years now, ever since the first LLaMA release back in March 2023. About six months ago I had mostly lost interest in them. They were fun to play around with but the quality difference between the ones I could run on my MacBook and the ones I could access via an online API felt insurmountable. This has completely changed in the second half of 2024. The models I can run locally had a leap in quality - they feel genuinely GPT-4 class now. They're not as good as the best hosted models (GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet) but they're definitely good enough to be extremely useful. This started with the Qwen 2 and 2.5 series, but I also rate Llama 3.3 70B and now Phi-4 as GPT-4 class models that run on my laptop. I wrote more about this here: https://simonwillison.net/2024/Dec/31/llms-in-2024/#some-of-those-gpt-4-models-run-on-my-laptop https://simonwillison.net/2024/Dec/31/llms-in-2024/#some-of-...

OhioMan2943 2y ago
I'm in complete agreement with your more recent timeline piece (the negative one), and as a younger user (22 year old student) I'm actively relocating this year to somewhere slightly more rural with a focus on physical/knowledge combined work to secure a good quality of life nearly solely because of how fast our timelines are. A 'word calculator' this effective is the best substitute that we have for a logic calculator. And the fact that it's enough in 90% of situations is terrifying as it is transformative, as is the fact no one is awake to it. Exponential power scaling in an unstable world feels like it only makes it exponentially more unstable though.
- simonw 2y ago
  I should emphasize that I really don't think the dystopian version of this is likely to happen - the one where "AGI/ASI" puts every human out of work and society collapses. Human beings have agency, and we are very good at rolling with the punches. We've survived waves of automation for hundreds of years. I'm much more confident that we will continue to find ways to use these things as tools that elevate us, not replace us. I really hope the dystopian version doesn't come to pass!
  - actsasbuffoon 2y ago
    I’m on the opposite end of the spectrum. I’m almost certain that this is going to end extremely badly for the majority of humanity, and for programmers in particular. I think there’s a less than 5% chance that this goes well, and that’s only if we get a series of things to go extremely well. And frankly, we’re tracking along the extremely bad path so far.
    - OhioMan2943 2y ago
      We barely survived one nuclear arms race, and this could give every nation state a new type of power weapon every 5ish years through the inevitable scaling in energy and weapons. I agree we're on one of the worst timelines for AI/AGI/ASI with the world actively being run into the ground by short sighted 'dementia-ocracies' and every security risk about to increase dramatically.
  - imiric 2y ago
    > We've survived waves of automation for hundreds of years. I'm much more confident that we will continue to find ways to use these things as tools that elevate us, not replace us. The difference with past technological breakthroughs is that they augmented what humans could do, but didn't have the potential to replace human labor altogether as AI does. They were disruptive, but humans were able to adapt to new career paths as they became available. Lamplighters were replaced by electrical lighting, but that created jobs for electrical engineers. Carriage drivers were replaced by car drivers; human computers by programmers, and so on. The invention of AI is a tipping point for technology. The only jobs where human labor will be valued over machine labor are those that machines are not good at yet, and those where human creativity is a key component. And the doors are quickly closing on both of those as well. Realistically, the only jobs that will have some form of longevity (~20 years?) are those of humans that build and program AI machines. But eventually even those will be better accomplished by other AI machines. So, I'm really curious why you see AI as the same kind of technology we've invented before, and why you're so confident that humanity will be able to overcome the key existential problems AI introduces, which we haven't even begun to address. I don't see myself as a pessimist, but can't help noticing that we're careening towards a future we're not prepared to handle.

SamPatt 2y ago

Yes, and for image and video generation too. Hunyuan (open source video) has been remarkable. Flux dev makes some incredible images. The fact that it's still only going to get better from here is hard to imagine.

api 2y ago

I’ve thought for a while that Joi in BR2049 was less dystopian than what we will probably do with AI. She doesn’t constantly prompt K to buy more credits (like a mobile game) to continue engaging with her or deepen their relationship. (“If you really love me…”) I’ve been expecting that this is how our industry would operate given the customer hostile psychologically abusive hellscape of social and mobile. Of course there’s still time. She appears to be a local model runnable on a small device without cloud.

brookst 2y ago
I expect AI to be like any other tech: some fantastic uses that advance humanity and improve the world, some terrible uses that abuse, manipulate, oppress. I don’t see anything in the tech that indicates a singular pattern that will be “good” or “bad”.
- ericrosedev 2y ago
  The enshitification trend seems pretty dominant, and pretty bad for users / good for investors

cloudking 2y ago

Not related to local LLMs, but JOI from BR2049 is essentially what Replika is striving for: https://replika.com/ https://replika.com/ Infact during the onboarding process they ask the user to choose which AI companion movie they related to the most: Her, BR2049 or Ex-Machina. The experience is then tailored to align closer to the movie chosen. It's quite a terrible app from a product design perspective: filled with dark patterns (like sending the user blurred images to "unlock*) and upsells, but it's become successful amongst the masses that have adopted it, which I find fascinating. 30m+ users https://en.wikipedia.org/wiki/Replika#:~:text=Replika%20became%20available%20to%20the,users%20had%20surpassed%2030%20million https://en.wikipedia.org/wiki/Replika#:~:text=Replika%20beca....

mark_l_watson 2y ago

I am blown away: a year ago I bought a M2 32G Mac to run local models. It seems like what I can run locally now just one year later is 10x more useful for NLP, data wrangling, RAG, experimenting with agents, etc. BTW, a few days ago I published a book on using Ollama. Here is a link to read it online https://leanpub.com/ollama/read https://leanpub.com/ollama/read

hn8726 2y ago
Which models do you recommend for that amount of memory?
- rpastuszak 2y ago
  I asked the same question a few days back and I'm keeping the responses here: https://bsky.app/profile/potato.horse/post/3lejngewfmc2n https://bsky.app/profile/potato.horse/post/3lejngewfmc2n
- mark_l_watson 2y ago
  For reasoning: qwq:latest (19G file) For coding: qwen2.5-coder:14b (9G file) Misc. experiments, runs fast: llama3.2:latest )2 G file)

yeahwhatever10 2y ago

How can a model "grow with you"? Do current models do this other than adding the full conversation to the context window?