3 ms·

>For example, I was in a presentation by a gentleman who works at DeepMind last year and who works on neuro-symbolic integration Sure, neural networks don't op

>For example, I was in a presentation by a gentleman who works at DeepMind last year and who works on neuro-symbolic integration Sure, neural networks don't operate on proper variables and so in the context of neuro-symbolic processing I'm sure this is a significant hurdle. But in general, abstract representations is part-and-parcel of what makes deep learning powerful. And such an abstract representation is all that's needed for a neural arithmetic unit. Here[1] is a study on GPT-2 that demonstrates its middle layers develop a representation of syntax and part-of-speech, the sorts of abstract representations that would be needed to develop a mechanism to do abstract arithmetic. >What I mean is, you seem to have a theory about how GPT-3 works. Where does it come from? Studies like the one mentioned, and reasonable extrapolation from knowledge of DL and other transformer architectures. We are not totally ignorant on how GPT-3 works. [1] https://aletheap.github.io/posts/2020/07/looking-for-grammar/ https://aletheap.github.io/posts/2020/07/looking-for-grammar...

YeGoblynQueenne 6y ago
My comment above discussed the inability of neural networks (and propositional, attribute-value learners in general) to represent variables. I'm sorry, but I can't see how your comment or the post you link to show that neural networks can represent variables. I do not quite understand the relation between "abstract representations that would be needed to develop a mechanism to do abstract arithmetic" and variables. I'm also not sure what you mean by "abstract arithmetic", or what mechanisms you mean. Can you please explain? Also, I had thought we shared an understanding that the ability to represent primitive recursive functions (which presupose the ability to represent variables and recursion) is necessary to represent arithmetic. Your above comment now makes me doubt this, also. Can you clarify? Finally, the link above is a blog post. I wouldn't call it a study. But, can you say where in that post I can find the theory about GPT-3's function that you express above?

YeGoblynQueenne 6y ago

My comment above discussed the inability of neural networks (and propositional, attribute-value learners in general) to represent variables. I'm sorry, but I can't see how your comment or the post you link to show that neural networks can represent variables. I do not quite understand the relation between "abstract representations that would be needed to develop a mechanism to do abstract arithmetic" and variables. I'm also not sure what you mean by "abstract arithmetic", or what mechanisms you mean. Can you please explain? Also, I had thought we shared an understanding that the ability to represent primitive recursive functions (which presupose the ability to represent variables and recursion) is necessary to represent arithmetic. Your above comment now makes me doubt this, also. Can you clarify? Finally, the link above is a blog post. I wouldn't call it a study. But, can you say where in that post I can find the theory about GPT-3's function that you express above?