43 ms·

This EfficientVectorRouting paper (and related papers) is fascinating. It seems to have endless potential applications. I'm surprised I had not heard of it befo

This EfficientVectorRouting paper (and related papers) is fascinating. It seems to have endless potential applications. I'm surprised I had not heard of it before. I'm excited to experiment with novel model structures incorporating it. What's the catch? I'm guessing they're fairly expensive layers, since they're based on EM? Although it looks like it's using incremental EM (iEM), which perhaps makes it less expensive? May I ask what you are using it for?

cs702 4y ago
There's no catch, really, but like everything else, there are applications and uses for which there are better tools. For example, if you need a layer for modeling n² relationships, you'd be much better off using something like FlashAttention than this routing method. I've found it to be a useful tool. It's become my go-to solution whenever I need to combine data, hidden states, or model outputs of different sizes/dimensions/modalities into something that works on the first try. It can "glue models together into systems" without much hassle. I mean, you still have to tinker and tweak to get things to work well, but no more than usual. Most recently, I've been using it to build models with access to long-term memory, more or less like this: Concatenate a large number of sequences of previously computed transformer hidden states. Apply a couple of routing layers to shrink them into a single short sequence of fixed size. Attach the shrunk sequence as a form of long-term memory to the current sequence of final transformer hidden states. Slap a couple of routing layers on top to predict things of interest. Finally, train only the routing layers (leave all the transformer stuff frozen). You can of course save the long sequences of previous transformer hidden states in a key-value store and fetch them as needed :-)
- ilaksh 4y ago
  This is a really dumb question but could EfficientVectorRouting be used for text completion somehow?
  - cs702 4y ago
    In principle, yes, you can, by using something like this on the last layer https://github.com/glassroom/heinsen_routing#sequence-to-vector https://github.com/glassroom/heinsen_routing#sequence-to-vec... ... but keep in mind it will not match the training efficiency of triangular masking in autoregressive transformers. You'd have to train the model one time-step at a time, vs. all time-steps in parallel with a masked autoregressive transformer. If you're looking to make context longer, a more practical approach is to route multiple past sequences of frozen pretrained transformer hidden states to a much shorter sequence, say, of length C, concatenate that with the current final sequence of hidden states, say, of length N, apply masked autoregressive attention on the C+N hidden states and then a feedforward layer to predict all next tokens in parallel, and last, drop the first C predictions, i.e., use only the trailing N predicted next tokens for training and inference.

cs702 4y ago

There's no catch, really, but like everything else, there are applications and uses for which there are better tools. For example, if you need a layer for modeling n² relationships, you'd be much better off using something like FlashAttention than this routing method. I've found it to be a useful tool. It's become my go-to solution whenever I need to combine data, hidden states, or model outputs of different sizes/dimensions/modalities into something that works on the first try. It can "glue models together into systems" without much hassle. I mean, you still have to tinker and tweak to get things to work well, but no more than usual. Most recently, I've been using it to build models with access to long-term memory, more or less like this: Concatenate a large number of sequences of previously computed transformer hidden states. Apply a couple of routing layers to shrink them into a single short sequence of fixed size. Attach the shrunk sequence as a form of long-term memory to the current sequence of final transformer hidden states. Slap a couple of routing layers on top to predict things of interest. Finally, train only the routing layers (leave all the transformer stuff frozen). You can of course save the long sequences of previous transformer hidden states in a key-value store and fetch them as needed :-)

ilaksh 4y ago
This is a really dumb question but could EfficientVectorRouting be used for text completion somehow?
- cs702 4y ago
  In principle, yes, you can, by using something like this on the last layer https://github.com/glassroom/heinsen_routing#sequence-to-vector https://github.com/glassroom/heinsen_routing#sequence-to-vec... ... but keep in mind it will not match the training efficiency of triangular masking in autoregressive transformers. You'd have to train the model one time-step at a time, vs. all time-steps in parallel with a masked autoregressive transformer. If you're looking to make context longer, a more practical approach is to route multiple past sequences of frozen pretrained transformer hidden states to a much shorter sequence, say, of length C, concatenate that with the current final sequence of hidden states, say, of length N, apply masked autoregressive attention on the C+N hidden states and then a feedforward layer to predict all next tokens in parallel, and last, drop the first C predictions, i.e., use only the trailing N predicted next tokens for training and inference.