3 ms·
Two quibbles: (1) neural nets don’t necessarily operate on language, (2) they only loosely model biological neurons in that they operate in discrete space. All
by catlifeonmars 3y ago
Two quibbles: (1) neural nets don’t necessarily operate on language, (2) they only loosely model biological neurons in that they operate in discrete space. All that is to say that any similarities are purely incidental without accounting for these two facts.
- sjwhevvvvvsj 3y agoWell, in some sense they don’t operate on language at all - but on mathematical representation of tokens derived from language. I’m sure from your comment you are aware of the distinction, but it is an interesting concept for people to keep in mind.
- metricspaces 3y ago[dead]
- pmayrgundter 3y ago1) agreed. it's exciting seeing the same basic architecture broadly applied 2) not sure what you mean by "operate in discrete space" I'd emphasize the potential similarity to biological recurrence. Deep ANNs don't need to have this explicitly (tho e.g. LSTM has explicit recurrence), but it is known that recurrent NNs can be emulated by unrolling, in a process similar to function currying. In this mode, a learned network would learn to recognize certain inputs and carry them across to other parts of the network that can be copies of the originator, thus achieving functional equivalence to self feedback, or neighbor feedback. It takes a lot of layers and nodes in theory, but ofc modern nets are getting very big.
- catlifeonmars 3y agoI didn’t phrase (2) particularly well. I meant that digital neural networks typically use floating point numbers, which can represent only discrete values. Do aliasing effects show up in the behavior of these networks that wouldn’t apply for continuously variable systems, or perhaps more appropriate for biological NNs, temporal coding via spike trains?