3 ms·
This is true. We use backpropagation not because it’s the only way or because it’s biologically plausible (the brain doesn’t have any backward passes) but becau
by janalsncm 3y ago
This is true. We use backpropagation not because it’s the only way or because it’s biologically plausible (the brain doesn’t have any backward passes) but because it works. Neural networks aren’t special because of any sort of connection to the brain, we use them because we have hardware (GPUs) which can train them pretty quickly.
I feel the same way about transformers vs RNNs: even if RNNs are more “correct” in some sense of having theoretically infinite memory it takes forever to train them so transformers won. And then we developed techniques like Long LoRA which make theoretical disadvantages functionally irrelevant.
- bilsbie 3y ago> developed techniques like Long LoRA which make theoretical disadvantages functionally irrelevant. How’s that?
- janalsncm 3y agoContext windows used to be tiny e.g. 768 or 1024. That meant sliding windows of limited context if you had any reasonably sized input. If your context window is 32k or even 100k, a lot of inputs will fit into the context window entirely. The reason huge context windows weren't possible in the past is that memory requirements were quadratic with input length. Long LoRA lets us use less memory for our context windows, or use the same memory footprint for larger context windows.
- erikerikson 3y agoI agree that doing effective things is effective and we should do effective things when they prove valuable. These things here are generally simplifications or even distillations which can be faster and more efficient which are excellent attributes of solutions. My objection isn't that we develop great systems but that we don't forget that other branches in the solution space exist. Particularly here because the more rich enough may help yield the next generation of solutions.