3 ms·
Does anyone understand why they are using B x H instead of B x S x H? Why is the context size and batch size represented as a single parameter?
by jasonjmcghee 3y ago
Does anyone understand why they are using B x H instead of B x S x H?
Why is the context size and batch size represented as a single parameter?
- numeri 3y agoI would have to go back and reread the paper to be sure, but FF layers are applied position-wise, meaning independently and in parallel on all input tokens/positions. Because of that, I could imagine contexts where the sequence dimension isn't relevant, i.e., for computational complexity.