4 ms·
I am not sure if the origin of polysyntacicity is fully understood. From a physics perspective it’s entropy: There are just more local minima that have neurons
by niemandhier 3y ago
I am not sure if the origin of polysyntacicity is fully understood.
From a physics perspective it’s entropy:
There are just more local minima that have neurons code multiple things.
I suspect that dropout and similar tricks also play a role in this:
Removing connections during training means that pathways need to be redundant somewhat.
- armatav 3y agoThat's super interesting, I wonder if dropout applied in a certain pattern of segregation could induce more neuron dependence, instead of less. Or do some anti-dropout where a dedicated portion of the model is used. Not sure why you'd want that, but it might be interesting to explore.