4 ms·
Is that necessarily the case for massive LLMs, or could there be a more refined grouping?
by armatav 3y ago
Is that necessarily the case for massive LLMs, or could there be a more refined grouping?
- niemandhier 3y agoI am not sure if the origin of polysyntacicity is fully understood. From a physics perspective it’s entropy: There are just more local minima that have neurons code multiple things. I suspect that dropout and similar tricks also play a role in this: Removing connections during training means that pathways need to be redundant somewhat.
- armatav 3y agoThat's super interesting, I wonder if dropout applied in a certain pattern of segregation could induce more neuron dependence, instead of less. Or do some anti-dropout where a dedicated portion of the model is used. Not sure why you'd want that, but it might be interesting to explore.