3 ms·
it makes you wonder if it may be more efficient to spend all the weights on one layer, and have a repeating stack of the same layer, one would presume this axis
by DoctorOetker 3mo ago
it makes you wonder if it may be more efficient to spend all the weights on one layer, and have a repeating stack of the same layer, one would presume this axis has already been explored with metaparameter sweeps?
- smallerize 3mo agoThat's called a recurrent neural network (RNN).
- DoctorOetker 3mo agodistinction is that GPT allows a lot of parallel computation compared to RNN, but I see how your remark indicates a convergence towards RNNs indeed.