5 ms·

“This new technique” is they load each layer into memory, process the result of that layer, and then clear it and load the next layer. For the cases where this

by svnt 3y ago

“This new technique” is they load each layer into memory, process the result of that layer, and then clear it and load the next layer. For the cases where this even works, as one commenter said: “I try it on a GTX 1060 6Gb on windows, but I don’t think it’s supposed to be that slow? It took me 13 hours to generate one sentence, RAM was around 9.5Gb and only 6% of GPU in use. It works but I don’t know what I’m doing wrong”

james-revisoai 3y ago
I appreciate your and the other commentators scepticism, and agree for the premise of generating text. However to play devils advocate - Can't inference of single layers be quite useful sometimes for embeddings? and the larger models could embed capabilities only achievable at large parameter size? (i.e. you don't need to finetune the last layer like the BERT days or necessarily use a certain layer for relevant embeddings, you could use the first layer outputs and finetune a custom matrix projection to make some useful embeddings that might include properties of the input that the larger models show capabilities for that smaller models don't?) (ofc this assumes that those capabilities considered "emergent" at larger sizes do not require later layers which they probably do). There is a weird situation where we have the sentence transformer, the bge, e5 and etc kind of embeddings, and then a big jump up to generative model embeddings the model providers provide, but not much widespread adoption of e.g GPT-J or neo-20B embeddings (even though at time of release, they had notable usecases over sentence transformers)
- sp332 3y ago
  You can save some space, but it still takes more than one layer to do an embedding.
- svnt 3y ago
  There must be some real use for it for sure. Maybe as a result of this effort someone else has their attention called to it and picks up the idea loosely and runs with it in an unexpected way. One of the benefits of lack of contextual or historical understanding is the lack of barriers to action. One of the downsides is you’re likely to do a lot of things of questionable value.
makomk 3y ago
Yeah, I'm sure the performance of this must be pretty bad. As I understand it, non-batched LLM inference is generally limited by the bandwidth required to read the entire weights from GPU memory for each token produced. This trick instead moves the bandwidth bottleneck to loading those weights into the GPU from wherever has enough space to store them, probably some kind of SSD, and that has much lower bandwidth than GPU RAM.
- andy99 3y ago
  For CPU inference, performance is generally memory bandwidth limited. A modern cpu can do the matmuls (all of which are matrix-vector) faster than the data can be loaded in from memory. If the model is all in RAM and being moved one layer at a time to a GPU, would that be faster than getting it to the CPU? What about if the model is on an SSD. The point being, compute is not the bottleneck so I wonder if this provides speedup compared to an optimized model run on a CPU?
  - faeriechangling 3y ago
    It would potentially be slower because pcie bandwidth is likely to be lower than memory bandwidth. The way to get a speed up is to split the load across both the CPU and GPU since LLMs can be split into layers. This also increases the size the model can be.
  - andy99 3y ago
    Just want to add, this approach is the exact opposite of conventional wisdom around cache hierarchy. Here we are loading data in from the slowest source, using it once, and throwing it away. I'm not sure what can be changed with a sequential model like and llm, just that this is almost the textbook slowest way to run the model.
  - gpm 3y ago
    Tried this awhile ago on the llama models, with my hardware 1. The GPU was still faster, though the difference was smaller than I really expected. 2. Almost all the time was spent on compute, not loading weights to/from the GPU. For models I could fit on my GPU the difference in performance in loading one layer at a time vs loading them all was completely negligible, and the difference in memory usage (constraining what else I could do with the computer simultaneously) significant. 3. Loading from SSD was a huge bottleneck when I tried that Hardware: RTX 2070 super, 5800x
moffkalast 3y ago
> 13 hours to generate one sentence This is hysterical, they're literally swapping from VRAM to RAM and certainly to disk as well since a 70B GGUF is at least 30-40 GB in total. I suppose it's slightly better than not being able to to it at all, but jesus christ this is just hilariously bad.
godelski 3y ago
> “This new technique” Which also has a name that they even use, model sharding. Which is well known to be very slow because you increase the number of IO operations to the GPU. I don't even understand the post because they mention sharding is slow and then say they do sharding but as if they don't. Even the comment in their code calls it sharding. Unless I'm missing something, it's pretty fucking deceptive to call this a new technique. Can we stop with the AI hype? The field is exciting enough that we don't need to be selling snakeoil and selling everything as if it is 10x better than it is. For anyone interested in model sharding you should look into pytorch's gradient checkpointing documentation and/or fairseq.
- benreesman 3y ago
  To echo and in no way contradict the parent’s point, there have been versions of this going on forever. Without getting into a bunch of stuff about FDSP or Accelerate or even the NCCL/All-Reduce type ops (which I obviously hope the interested check out) I’ll signal boost my favorite tech talk ever by maybe the coolest CS prof living: https://youtu.be/l5JqUvTdZts?si=9HlNWHTlRbiatHD1 https://youtu.be/l5JqUvTdZts?si=9HlNWHTlRbiatHD1 I’m biased because I’m also Ben, but I’m not “Ben Rekt” who is a musician and charismatic and way too cool to care about SV.
  - godelski 3y ago
    Btw, that link isn't working for me, this is though https://www.youtube.com/watch?v=l5JqUvTdZts https://www.youtube.com/watch?v=l5JqUvTdZts Hogwild is a really cool routine. There's a lot of work going into FDSP, parallelism, and all kinds of things at the low level for optimizing communication, maximizing parallelization, writing cuda kernels, and so on. I think there's a lot of unsung heros of ML working in these areas that people never even see (including researchers). So many people quote Sutton's Bitter Lesson but these are the people that made that landscape even possible. Without these people working at the low level we wouldn't be able to process large models or large datasets. The fame issue is always a bit weird to me. Like I think more people know of Vaswani (AIAYN) or Dosovitskiy (ViT paper) than those that know Kingma, who wrote the Adam optimizer. Both these works use Adam as well as Rombach's LDM (Stable Diffusion). I'm only mentioning Kingma because I'm more aware of that work but am certain there's tons of work that could be better highlighted (actually illustrating this point).

by svnt 3y ago

james-revisoai 3y ago

I appreciate your and the other commentators scepticism, and agree for the premise of generating text. However to play devils advocate - Can't inference of single layers be quite useful sometimes for embeddings? and the larger models could embed capabilities only achievable at large parameter size? (i.e. you don't need to finetune the last layer like the BERT days or necessarily use a certain layer for relevant embeddings, you could use the first layer outputs and finetune a custom matrix projection to make some useful embeddings that might include properties of the input that the larger models show capabilities for that smaller models don't?) (ofc this assumes that those capabilities considered "emergent" at larger sizes do not require later layers which they probably do). There is a weird situation where we have the sentence transformer, the bge, e5 and etc kind of embeddings, and then a big jump up to generative model embeddings the model providers provide, but not much widespread adoption of e.g GPT-J or neo-20B embeddings (even though at time of release, they had notable usecases over sentence transformers)

sp332 3y ago
You can save some space, but it still takes more than one layer to do an embedding.
svnt 3y ago
There must be some real use for it for sure. Maybe as a result of this effort someone else has their attention called to it and picks up the idea loosely and runs with it in an unexpected way. One of the benefits of lack of contextual or historical understanding is the lack of barriers to action. One of the downsides is you’re likely to do a lot of things of questionable value.

makomk 3y ago

Yeah, I'm sure the performance of this must be pretty bad. As I understand it, non-batched LLM inference is generally limited by the bandwidth required to read the entire weights from GPU memory for each token produced. This trick instead moves the bandwidth bottleneck to loading those weights into the GPU from wherever has enough space to store them, probably some kind of SSD, and that has much lower bandwidth than GPU RAM.

andy99 3y ago
For CPU inference, performance is generally memory bandwidth limited. A modern cpu can do the matmuls (all of which are matrix-vector) faster than the data can be loaded in from memory. If the model is all in RAM and being moved one layer at a time to a GPU, would that be faster than getting it to the CPU? What about if the model is on an SSD. The point being, compute is not the bottleneck so I wonder if this provides speedup compared to an optimized model run on a CPU?
- faeriechangling 3y ago
  It would potentially be slower because pcie bandwidth is likely to be lower than memory bandwidth. The way to get a speed up is to split the load across both the CPU and GPU since LLMs can be split into layers. This also increases the size the model can be.
- andy99 3y ago
  Just want to add, this approach is the exact opposite of conventional wisdom around cache hierarchy. Here we are loading data in from the slowest source, using it once, and throwing it away. I'm not sure what can be changed with a sequential model like and llm, just that this is almost the textbook slowest way to run the model.
- gpm 3y ago
  Tried this awhile ago on the llama models, with my hardware 1. The GPU was still faster, though the difference was smaller than I really expected. 2. Almost all the time was spent on compute, not loading weights to/from the GPU. For models I could fit on my GPU the difference in performance in loading one layer at a time vs loading them all was completely negligible, and the difference in memory usage (constraining what else I could do with the computer simultaneously) significant. 3. Loading from SSD was a huge bottleneck when I tried that Hardware: RTX 2070 super, 5800x

moffkalast 3y ago

> 13 hours to generate one sentence This is hysterical, they're literally swapping from VRAM to RAM and certainly to disk as well since a 70B GGUF is at least 30-40 GB in total. I suppose it's slightly better than not being able to to it at all, but jesus christ this is just hilariously bad.

godelski 3y ago

> “This new technique” Which also has a name that they even use, model sharding. Which is well known to be very slow because you increase the number of IO operations to the GPU. I don't even understand the post because they mention sharding is slow and then say they do sharding but as if they don't. Even the comment in their code calls it sharding. Unless I'm missing something, it's pretty fucking deceptive to call this a new technique. Can we stop with the AI hype? The field is exciting enough that we don't need to be selling snakeoil and selling everything as if it is 10x better than it is. For anyone interested in model sharding you should look into pytorch's gradient checkpointing documentation and/or fairseq.

benreesman 3y ago
To echo and in no way contradict the parent’s point, there have been versions of this going on forever. Without getting into a bunch of stuff about FDSP or Accelerate or even the NCCL/All-Reduce type ops (which I obviously hope the interested check out) I’ll signal boost my favorite tech talk ever by maybe the coolest CS prof living: https://youtu.be/l5JqUvTdZts?si=9HlNWHTlRbiatHD1 https://youtu.be/l5JqUvTdZts?si=9HlNWHTlRbiatHD1 I’m biased because I’m also Ben, but I’m not “Ben Rekt” who is a musician and charismatic and way too cool to care about SV.
- godelski 3y ago
  Btw, that link isn't working for me, this is though https://www.youtube.com/watch?v=l5JqUvTdZts https://www.youtube.com/watch?v=l5JqUvTdZts Hogwild is a really cool routine. There's a lot of work going into FDSP, parallelism, and all kinds of things at the low level for optimizing communication, maximizing parallelization, writing cuda kernels, and so on. I think there's a lot of unsung heros of ML working in these areas that people never even see (including researchers). So many people quote Sutton's Bitter Lesson but these are the people that made that landscape even possible. Without these people working at the low level we wouldn't be able to process large models or large datasets. The fame issue is always a bit weird to me. Like I think more people know of Vaswani (AIAYN) or Dosovitskiy (ViT paper) than those that know Kingma, who wrote the Adam optimizer. Both these works use Adam as well as Rombach's LDM (Stable Diffusion). I'm only mentioning Kingma because I'm more aware of that work but am certain there's tons of work that could be better highlighted (actually illustrating this point).