5 ms·

This doesn't seem technically feasible to me. The state of the art will for a long time require a lot more hardware to run than it's available on a consumer dev

This doesn't seem technically feasible to me. The state of the art will for a long time require a lot more hardware to run than it's available on a consumer device. Beyond which, inference also benefits from parallelization, not just training, so being able to batch requests is a benefit, and more likely when access is offered via an API.

FL33TW00D 4y ago
This isn't correct. It's 100% feasible to run very capable models on consumer devices. I wrote up a feasibility investigation last year: https://fleetwood.dev/posts/a-case-for-client-side-machine-learning https://fleetwood.dev/posts/a-case-for-client-side-machine-l...
- frozenlettuce 4y ago
  I ran alpaca 7B in a 2015 Acer laptop (it was slow, but worked!)
- crazygringo 4y ago
  But there's a big difference between running "very capable" models that use most available resources and consume tons of power and run slowly... ...versus running the best models available, in a few seconds, without using up the memory the main app you're using needs for running. These are all mainly going to be run remotely for general consumer usage for quite a while I think.
  - FL33TW00D 4y ago
    How many applications on your machine right now are GPU accelerated? There is a chunk of silicon that spends almost all its time doing nothing.
    - crazygringo 4y ago
      Sure but it's not that powerful. It's not a gaming rig. Also it had better spend almost all its time doing nothing or it would kill my battery. Same as with my CPU. The main point still stands though -- it's pretty useless if it takes a couple minutes to do what a server can do in a couple seconds.
      - earthboundkid 4y ago
        How do you think the server does it in a couple of seconds? Maybe we should get rid of gaming rigs because servers are fast, right?
    - pdntspa 4y ago
      Every single one of them since both windows and mac use the GPU for window management I miss Aero, that shit was so cool...
- moffkalast 4y ago
  > very capable models Well that's the problem though, those models don't come any close to being useful at all. At least not yet. And they also run much slower. As compute increases in general, there will be larger and more capable state of the art models and it'll make more sense to just use those instead of trying to run some local one that won't give you any useful answers. Data centers will always have a few orders of magnitude more horsepower than your average laptop, even with some kind of inference accelerator card.
  - FL33TW00D 4y ago
    Stable diffusion isn't useful at all?
    - moffkalast 4y ago
      Do people use it for anything practical? Making stock photos maybe? I haven't really had a proper use case for it and all the random things I tried to make with it weren't good enough to use with anything. Could be useful for making concepts for real artists, but last I heard they were all too busy boycotting it. Also, not an LLM.
      - UncleEntity 4y ago
        > I haven't really had a proper use case for it and all the random things I tried to make with it weren't good enough to use with anything. Sounds a lot like most of my early programming experiments… Though I’ve heard on good authority that the early programmers looked past being able to calculate ballistic charts and have done some interesting things with these “computer” things.
      - zirgs 4y ago
        There are some models that can generate tileable textures, for example.
        moffkalast 4y ago
        Ah interesting yeah, found this one: https://replicate.com/tommoore515/material_stable_diffusion https://replicate.com/tommoore515/material_stable_diffusion Trying out some prompts, maybe last I used SD my mistake was going with a lower resolution to speed up generation. I literally cannot get this one to make anything that isn't a weird blob at 256px and lower, but at 512px it works fine? Weird that it's so resolution dependant. I guess some proper stuff can be made at 1024px and above.
- circuit10 4y ago
  “very capable” and “state of the art” aren’t the same thing. In this case the gap is huge
  - FL33TW00D 4y ago
    This is true, but a good subset of tasks can be accomplished by much smaller models. Copilot is 12B!
brucethemoose2 4y ago
I can already run Vicuna(llama) 7B on my 2020, 14" PC laptop at ~3.5 tokens/sec, and more speed can definitely be squeezed out. Most future laptops and phones will ship with NPUs next to the CPU silicon. Once they get enabled in software, that means a 16GB machine can run a 13B model, or a 7B model with room for other heavy apps. As for the benefits of batching and centralization, that is true, but its somewhat countered by the high cost of server accelerators and the high profit margins of cloud services.
- int_19h 4y ago
  It's not just the compute, you need fast memory too. And 7B and 13B are nowhere near enough to get you GPT-3.5 level of performance, which is where it becomes actually interesting. We'll get there eventually but I don't think it's right around the corner or anything like that.
  - brucethemoose2 4y ago
    Setting the M series aside, the AMD 7000 laptops already have reasonably fast memory. Faster than some old GPUs. And that trend is accelerating. The latest rumor is that Intel is bringing back the eDRAM cache next (which means it was in planning long before the generative ai craze), and more stacked/on package memory is just around the corner.
    - lhl 4y ago
      While 7000U laptops have yet to be benchmarked, dual-channel DDR5/quad-channel LPDDR5 systems top out at about 60GB/s. (The M1/M2 by comparison is a 100GB/s, and doubles for Pro, Ultra, and Max up to 800GB/s). As a point of reference, top end consumer GPUs like the RTX 4090 are at about 1000GB/s. My understanding is things like V-Cache, eDRAM have limited benefits for dense transformers, as they need to cycle through all/most of the parameters when running.
Art9681 4y ago
This technology will be embedded into every OS within 2 years. People don't generally need a "super" model like GPT3/4. It will be perfectly acceptable and common to have the model change context, sync with whatever model/training data is necessary to be an expert in that context only, and associated contexts..., and prompt it in a specific domain. Client devices and internet connections are fast enough to do this in near real time today. The platforms to do all of this are being built right now by every company that creates software otherwise they will fail within 5 years. It's an inconvenient truth, for better or worse.

FL33TW00D 4y ago

This isn't correct. It's 100% feasible to run very capable models on consumer devices. I wrote up a feasibility investigation last year: https://fleetwood.dev/posts/a-case-for-client-side-machine-learning https://fleetwood.dev/posts/a-case-for-client-side-machine-l...

frozenlettuce 4y ago
I ran alpaca 7B in a 2015 Acer laptop (it was slow, but worked!)
crazygringo 4y ago
But there's a big difference between running "very capable" models that use most available resources and consume tons of power and run slowly... ...versus running the best models available, in a few seconds, without using up the memory the main app you're using needs for running. These are all mainly going to be run remotely for general consumer usage for quite a while I think.
- FL33TW00D 4y ago
  How many applications on your machine right now are GPU accelerated? There is a chunk of silicon that spends almost all its time doing nothing.
  - crazygringo 4y ago
    Sure but it's not that powerful. It's not a gaming rig. Also it had better spend almost all its time doing nothing or it would kill my battery. Same as with my CPU. The main point still stands though -- it's pretty useless if it takes a couple minutes to do what a server can do in a couple seconds.
    - earthboundkid 4y ago
      How do you think the server does it in a couple of seconds? Maybe we should get rid of gaming rigs because servers are fast, right?
  - pdntspa 4y ago
    Every single one of them since both windows and mac use the GPU for window management I miss Aero, that shit was so cool...
moffkalast 4y ago
> very capable models Well that's the problem though, those models don't come any close to being useful at all. At least not yet. And they also run much slower. As compute increases in general, there will be larger and more capable state of the art models and it'll make more sense to just use those instead of trying to run some local one that won't give you any useful answers. Data centers will always have a few orders of magnitude more horsepower than your average laptop, even with some kind of inference accelerator card.
- FL33TW00D 4y ago
  Stable diffusion isn't useful at all?
  - moffkalast 4y ago
    Do people use it for anything practical? Making stock photos maybe? I haven't really had a proper use case for it and all the random things I tried to make with it weren't good enough to use with anything. Could be useful for making concepts for real artists, but last I heard they were all too busy boycotting it. Also, not an LLM.
    - UncleEntity 4y ago
      > I haven't really had a proper use case for it and all the random things I tried to make with it weren't good enough to use with anything. Sounds a lot like most of my early programming experiments… Though I’ve heard on good authority that the early programmers looked past being able to calculate ballistic charts and have done some interesting things with these “computer” things.
    - zirgs 4y ago
      There are some models that can generate tileable textures, for example.
      - moffkalast 4y ago
        Ah interesting yeah, found this one: https://replicate.com/tommoore515/material_stable_diffusion https://replicate.com/tommoore515/material_stable_diffusion Trying out some prompts, maybe last I used SD my mistake was going with a lower resolution to speed up generation. I literally cannot get this one to make anything that isn't a weird blob at 256px and lower, but at 512px it works fine? Weird that it's so resolution dependant. I guess some proper stuff can be made at 1024px and above.
circuit10 4y ago
“very capable” and “state of the art” aren’t the same thing. In this case the gap is huge
- FL33TW00D 4y ago
  This is true, but a good subset of tasks can be accomplished by much smaller models. Copilot is 12B!

brucethemoose2 4y ago

I can already run Vicuna(llama) 7B on my 2020, 14" PC laptop at ~3.5 tokens/sec, and more speed can definitely be squeezed out. Most future laptops and phones will ship with NPUs next to the CPU silicon. Once they get enabled in software, that means a 16GB machine can run a 13B model, or a 7B model with room for other heavy apps. As for the benefits of batching and centralization, that is true, but its somewhat countered by the high cost of server accelerators and the high profit margins of cloud services.

int_19h 4y ago
It's not just the compute, you need fast memory too. And 7B and 13B are nowhere near enough to get you GPT-3.5 level of performance, which is where it becomes actually interesting. We'll get there eventually but I don't think it's right around the corner or anything like that.
- brucethemoose2 4y ago
  Setting the M series aside, the AMD 7000 laptops already have reasonably fast memory. Faster than some old GPUs. And that trend is accelerating. The latest rumor is that Intel is bringing back the eDRAM cache next (which means it was in planning long before the generative ai craze), and more stacked/on package memory is just around the corner.
  - lhl 4y ago
    While 7000U laptops have yet to be benchmarked, dual-channel DDR5/quad-channel LPDDR5 systems top out at about 60GB/s. (The M1/M2 by comparison is a 100GB/s, and doubles for Pro, Ultra, and Max up to 800GB/s). As a point of reference, top end consumer GPUs like the RTX 4090 are at about 1000GB/s. My understanding is things like V-Cache, eDRAM have limited benefits for dense transformers, as they need to cycle through all/most of the parameters when running.

Art9681 4y ago

This technology will be embedded into every OS within 2 years. People don't generally need a "super" model like GPT3/4. It will be perfectly acceptable and common to have the model change context, sync with whatever model/training data is necessary to be an expert in that context only, and associated contexts..., and prompt it in a specific domain. Client devices and internet connections are fast enough to do this in near real time today. The platforms to do all of this are being built right now by every company that creates software otherwise they will fail within 5 years. It's an inconvenient truth, for better or worse.