4 ms·

This isn't correct. It's 100% feasible to run very capable models on consumer devices. I wrote up a feasibility investigation last year: https://fleetwood.dev/

This isn't correct. It's 100% feasible to run very capable models on consumer devices. I wrote up a feasibility investigation last year: https://fleetwood.dev/posts/a-case-for-client-side-machine-learning https://fleetwood.dev/posts/a-case-for-client-side-machine-l...

frozenlettuce 4y ago
I ran alpaca 7B in a 2015 Acer laptop (it was slow, but worked!)
crazygringo 4y ago
But there's a big difference between running "very capable" models that use most available resources and consume tons of power and run slowly... ...versus running the best models available, in a few seconds, without using up the memory the main app you're using needs for running. These are all mainly going to be run remotely for general consumer usage for quite a while I think.
- FL33TW00D 4y ago
  How many applications on your machine right now are GPU accelerated? There is a chunk of silicon that spends almost all its time doing nothing.
  - crazygringo 4y ago
    Sure but it's not that powerful. It's not a gaming rig. Also it had better spend almost all its time doing nothing or it would kill my battery. Same as with my CPU. The main point still stands though -- it's pretty useless if it takes a couple minutes to do what a server can do in a couple seconds.
    - earthboundkid 4y ago
      How do you think the server does it in a couple of seconds? Maybe we should get rid of gaming rigs because servers are fast, right?
  - pdntspa 4y ago
    Every single one of them since both windows and mac use the GPU for window management I miss Aero, that shit was so cool...
moffkalast 4y ago
> very capable models Well that's the problem though, those models don't come any close to being useful at all. At least not yet. And they also run much slower. As compute increases in general, there will be larger and more capable state of the art models and it'll make more sense to just use those instead of trying to run some local one that won't give you any useful answers. Data centers will always have a few orders of magnitude more horsepower than your average laptop, even with some kind of inference accelerator card.
- FL33TW00D 4y ago
  Stable diffusion isn't useful at all?
  - moffkalast 4y ago
    Do people use it for anything practical? Making stock photos maybe? I haven't really had a proper use case for it and all the random things I tried to make with it weren't good enough to use with anything. Could be useful for making concepts for real artists, but last I heard they were all too busy boycotting it. Also, not an LLM.
    - UncleEntity 4y ago
      > I haven't really had a proper use case for it and all the random things I tried to make with it weren't good enough to use with anything. Sounds a lot like most of my early programming experiments… Though I’ve heard on good authority that the early programmers looked past being able to calculate ballistic charts and have done some interesting things with these “computer” things.
    - zirgs 4y ago
      There are some models that can generate tileable textures, for example.
      - moffkalast 4y ago
        Ah interesting yeah, found this one: https://replicate.com/tommoore515/material_stable_diffusion https://replicate.com/tommoore515/material_stable_diffusion Trying out some prompts, maybe last I used SD my mistake was going with a lower resolution to speed up generation. I literally cannot get this one to make anything that isn't a weird blob at 256px and lower, but at 512px it works fine? Weird that it's so resolution dependant. I guess some proper stuff can be made at 1024px and above.
circuit10 4y ago
“very capable” and “state of the art” aren’t the same thing. In this case the gap is huge
- FL33TW00D 4y ago
  This is true, but a good subset of tasks can be accomplished by much smaller models. Copilot is 12B!

frozenlettuce 4y ago

I ran alpaca 7B in a 2015 Acer laptop (it was slow, but worked!)

crazygringo 4y ago

But there's a big difference between running "very capable" models that use most available resources and consume tons of power and run slowly... ...versus running the best models available, in a few seconds, without using up the memory the main app you're using needs for running. These are all mainly going to be run remotely for general consumer usage for quite a while I think.

FL33TW00D 4y ago
How many applications on your machine right now are GPU accelerated? There is a chunk of silicon that spends almost all its time doing nothing.
- crazygringo 4y ago
  Sure but it's not that powerful. It's not a gaming rig. Also it had better spend almost all its time doing nothing or it would kill my battery. Same as with my CPU. The main point still stands though -- it's pretty useless if it takes a couple minutes to do what a server can do in a couple seconds.
  - earthboundkid 4y ago
    How do you think the server does it in a couple of seconds? Maybe we should get rid of gaming rigs because servers are fast, right?
- pdntspa 4y ago
  Every single one of them since both windows and mac use the GPU for window management I miss Aero, that shit was so cool...

moffkalast 4y ago

> very capable models Well that's the problem though, those models don't come any close to being useful at all. At least not yet. And they also run much slower. As compute increases in general, there will be larger and more capable state of the art models and it'll make more sense to just use those instead of trying to run some local one that won't give you any useful answers. Data centers will always have a few orders of magnitude more horsepower than your average laptop, even with some kind of inference accelerator card.

FL33TW00D 4y ago
Stable diffusion isn't useful at all?
- moffkalast 4y ago
  Do people use it for anything practical? Making stock photos maybe? I haven't really had a proper use case for it and all the random things I tried to make with it weren't good enough to use with anything. Could be useful for making concepts for real artists, but last I heard they were all too busy boycotting it. Also, not an LLM.
  - UncleEntity 4y ago
    > I haven't really had a proper use case for it and all the random things I tried to make with it weren't good enough to use with anything. Sounds a lot like most of my early programming experiments… Though I’ve heard on good authority that the early programmers looked past being able to calculate ballistic charts and have done some interesting things with these “computer” things.
  - zirgs 4y ago
    There are some models that can generate tileable textures, for example.
    - moffkalast 4y ago
      Ah interesting yeah, found this one: https://replicate.com/tommoore515/material_stable_diffusion https://replicate.com/tommoore515/material_stable_diffusion Trying out some prompts, maybe last I used SD my mistake was going with a lower resolution to speed up generation. I literally cannot get this one to make anything that isn't a weird blob at 256px and lower, but at 512px it works fine? Weird that it's so resolution dependant. I guess some proper stuff can be made at 1024px and above.

circuit10 4y ago

“very capable” and “state of the art” aren’t the same thing. In this case the gap is huge

FL33TW00D 4y ago
This is true, but a good subset of tasks can be accomplished by much smaller models. Copilot is 12B!