4 ms·

Well here is my (a bit cynical) take on it. In the beginning, there was the matrix multiply. A simple neural network is a chain of matrix multiplies. Let's say

Well here is my (a bit cynical) take on it. In the beginning, there was the matrix multiply. A simple neural network is a chain of matrix multiplies. Let's say you have your data A1 and weights W1 in a matrix. You produce A2 as A1xW1. Then you produce A3 as A2xW2, and so on. There are other operations in there like non-linearities (so that you can actually learn something interesting) and fancy batch norms, but let's forget about those for now. The problem with this is, it's not very expressive. Let's say your A1 matrix has just 2 values, and you want the output to be their product. Can you learn a weight matrix that performs multiplication of these inputs? No you can't. Multiplication must be simulated by piecing together piecewise linear functions. To perform multiplication, the weight matrix W would also need to be produced by the network. Transformers do basically that. In the product A*W you replace A with (AxW1), W with (AxW2), and multiply those together: (AxW1)x(AxW2) And then do it once more for good measure: (AxW1)x(AxW2)x(AxW3). Boom, Nobel prize. Now your network can multiply, not just add. OK it's actually a bit more complicated, there is for example a softmax in the middle to perform normalisation, which in general helps during numerical optimisation: softmax((AxW1)x(AxW2))x(AxW3). There are then fancy explanations that try to retrospectively justify this as a "differentiable lookup table" or somesuch nonsense, calling the 3 parts "key", "query" and "value", which help make your paper more popular. But the basic idea is not so complicated. A Transformer then uses this operation as a building block (running them in parallel an in sequence) to build giant networks that can do really cool things. Maybe you can teach networks to divide next and then you get the next Nobel prize.

seydor 3y ago
Are there any papers using more than 3 linearly transformed vectors?
mcdougal 3y ago
Are transformers - a hack devised/stumbled upon by AI workers or - a theoretical concept concocted by a mathematician who has been thinking about what NNs do or, - a set of techniques pipe-lined together by clever programmers who work with NNs? Or... - something else? I mean, if transformers really do something rational, then there should be a straightforward rational mathematical statement of the problem, a clear and clean expression of what they do! What I see instead is a lot of complex cumbersome description and terminological noise: no clear problem statement, lots of steps, lots of moving parts, and downhill from there. Now I'd be the first to admit that, if I believed I understood and could reproduce intelligence or language [and is intelligence merely language? An argument to that effect can be made - see Helen Keller], then if required I'd be prone to provide dense, noisy and incorrect explanations aplenty to potential competitors and even to honest inquisitive people. I would do that b/c revelation of the truth would destroy my competitive advantage. IOW I see every reason for ChatGPT et al developers and corporations to guide outsiders astray at this time. Developing something like ChatGPT is like running an exposed Manhattan Project - everyone wants Da Bomb and you don't want them to have it - instead you want to lead them completely astray. Seems to be succeeding: certainly as far as I'm concerned. Here's a simple test: has anyone within reach of my words made a version of these systems from scratch that does anything like the fullblown ChatGPT does and that (s)he will reveal? Current times would indicate that we won't learn how it all works until someone leaks it or someone else figures it out (like happened to Heisenberg and Schrodinger). My bet is on the latter. And that's the guy/gal who should get the Fields medal or Nobel Prize. [I'd bet a mathematician will do it]. Meanwhile as we twiddle our matrices much effort must be afoot to infiltrate the ChatGPT working groups and get the goods and also to keep the current ChatGPT worker bees from flying off to other hives and revealing secrets. This may be one of the few times when tech workers' jobs seriously shorten their lives. - Wandering in the desert...

seydor 3y ago

Are there any papers using more than 3 linearly transformed vectors?

mcdougal 3y ago

Are transformers - a hack devised/stumbled upon by AI workers or - a theoretical concept concocted by a mathematician who has been thinking about what NNs do or, - a set of techniques pipe-lined together by clever programmers who work with NNs? Or... - something else? I mean, if transformers really do something rational, then there should be a straightforward rational mathematical statement of the problem, a clear and clean expression of what they do! What I see instead is a lot of complex cumbersome description and terminological noise: no clear problem statement, lots of steps, lots of moving parts, and downhill from there. Now I'd be the first to admit that, if I believed I understood and could reproduce intelligence or language [and is intelligence merely language? An argument to that effect can be made - see Helen Keller], then if required I'd be prone to provide dense, noisy and incorrect explanations aplenty to potential competitors and even to honest inquisitive people. I would do that b/c revelation of the truth would destroy my competitive advantage. IOW I see every reason for ChatGPT et al developers and corporations to guide outsiders astray at this time. Developing something like ChatGPT is like running an exposed Manhattan Project - everyone wants Da Bomb and you don't want them to have it - instead you want to lead them completely astray. Seems to be succeeding: certainly as far as I'm concerned. Here's a simple test: has anyone within reach of my words made a version of these systems from scratch that does anything like the fullblown ChatGPT does and that (s)he will reveal? Current times would indicate that we won't learn how it all works until someone leaks it or someone else figures it out (like happened to Heisenberg and Schrodinger). My bet is on the latter. And that's the guy/gal who should get the Fields medal or Nobel Prize. [I'd bet a mathematician will do it]. Meanwhile as we twiddle our matrices much effort must be afoot to infiltrate the ChatGPT working groups and get the goods and also to keep the current ChatGPT worker bees from flying off to other hives and revealing secrets. This may be one of the few times when tech workers' jobs seriously shorten their lives. - Wandering in the desert...