5 ms·

What does "differentiable primitives" mean here?

by andoando 2y ago

CobrastanJorji 2y ago
Continuous mathematical functions which have derivatives.
- deleted 2y ago
  [deleted]
  - deleted 2y ago
    [deleted]
xanderlewis 2y ago
I think it’s referring to ‘primitive functions’ in the sense that they’re the building blocks of more complicated functions. If f and g are differentiable, f+g, fg, f/g (as long as g is never zero)… and so on are differentiable too. Importantly, f composed with g is also differentiable, and so since the output of the whole network as a function of its input is a composition of these ‘primitives’ it’s differentiable too. The actual primitive functions in this case would be things like the weighted sums of activations in the previous layer to get the activation of a given layer, and the actual ‘activation functions’ (traditionally something like a sigmoid function; these days a ReLU) associated with each layer. ‘Primitives’ is also sometimes used as a synonym for antiderivatives, but I don’t think that’s what it means here. Edit: it just occurred to me from a comment below that you might have meant to ask what the ‘differentiable’ part means. See https://en.wikipedia.org/wiki/Differentiable_function https://en.wikipedia.org/wiki/Differentiable_function.
- andoando 2y ago
  Is this function composition essentially lambda calculus then?
  - OJFord 2y ago
    Function composition is just f(g(x)), considered as a single function that's the composition of f and g; it has the domain of f and the range of g. In lambda calculus terminology it's an 'application' (with a function argument).
  - xanderlewis 2y ago
    Composition here just means what it does for any two functions: the value of the ‘composition’ of f and g at x is defined to be f applied to g applied to x. In symbols, its: f∘g := f(g(x)) for each x in the domain of f. It may seem obvious, but the fact that this new thing is also a function (that is, its value is well-defined for every input) is actually a very useful thing indeed and leads to… well, most of mathematics. You can certainly do function composition in lambda calculus: in fact, the act of composition itself is a higher order function (takes functions and returns a function) and you can certainly express it formally with lambda terms and such. It’s not really got anything to do with any particular language or model of computation though.
    - andoando 2y ago
      I didn't form my question too well. I understand all that. What I am asking is, are these function compositions equivalent to equivalent/similar to functions in lambda calculus? I guess my question, is what are the primitive functions here doing?
      - xanderlewis 2y ago
        Well, yes, to the extent that functions are functions are functions (they’re just associations or mappings or whatever you want to call them). Maybe your question boils down to asking something more general like: what’s the difference between functions to a computer scientist (or a programmer) and functions to a mathematician? That is, are ‘functions’ in C (or lambda calculus), say, the same ‘functions’ we talk about in calculus? The answer to that is: in this case, because these are quite simple functions (sums and products and compositions thereof) they’re the same. In general, they’re a bit different. The difference is basically the difference between functional programming and ‘traditional’ programming. If you have state/‘side effects’ of functions, then your function won’t be a function in the sense of mathematics; if the return value of your function depends entirely on the input and doesn’t return different values depending on whatever else is happening in the program, then it will be. Since you’re asking about lambda calculus in particular, the answer is that they’re the same because lambda calculus doesn’t have state. It’s ‘purely functional’ in that sense. >I guess my question, is what are the primitive functions here doing? I’m not really sure what you mean. They’re doing what functions always do. Every computer program is abstractly a (partial) function. Does that help, or have I misunderstood?
dirkc 2y ago
When I did "AI" it would have meant the sigmoid function, these days it's something like ReLU.
esafak 2y ago
functions, which you can compose to increase their expressiveness, and run gradient descent on to train. The success of deep learning is basically attributable to composable (expressive), differentiable (learnable) functions. The "deep" moniker alludes to the compositionality.

What does "differentiable primitives" mean here?

CobrastanJorji 2y ago

Continuous mathematical functions which have derivatives.

deleted 2y ago
[deleted]
- deleted 2y ago
  [deleted]

xanderlewis 2y ago

I think it’s referring to ‘primitive functions’ in the sense that they’re the building blocks of more complicated functions. If f and g are differentiable, f+g, fg, f/g (as long as g is never zero)… and so on are differentiable too. Importantly, f composed with g is also differentiable, and so since the output of the whole network as a function of its input is a composition of these ‘primitives’ it’s differentiable too. The actual primitive functions in this case would be things like the weighted sums of activations in the previous layer to get the activation of a given layer, and the actual ‘activation functions’ (traditionally something like a sigmoid function; these days a ReLU) associated with each layer. ‘Primitives’ is also sometimes used as a synonym for antiderivatives, but I don’t think that’s what it means here. Edit: it just occurred to me from a comment below that you might have meant to ask what the ‘differentiable’ part means. See https://en.wikipedia.org/wiki/Differentiable_function https://en.wikipedia.org/wiki/Differentiable_function.

andoando 2y ago
Is this function composition essentially lambda calculus then?
- OJFord 2y ago
  Function composition is just f(g(x)), considered as a single function that's the composition of f and g; it has the domain of f and the range of g. In lambda calculus terminology it's an 'application' (with a function argument).
- xanderlewis 2y ago
  Composition here just means what it does for any two functions: the value of the ‘composition’ of f and g at x is defined to be f applied to g applied to x. In symbols, its: f∘g := f(g(x)) for each x in the domain of f. It may seem obvious, but the fact that this new thing is also a function (that is, its value is well-defined for every input) is actually a very useful thing indeed and leads to… well, most of mathematics. You can certainly do function composition in lambda calculus: in fact, the act of composition itself is a higher order function (takes functions and returns a function) and you can certainly express it formally with lambda terms and such. It’s not really got anything to do with any particular language or model of computation though.
  - andoando 2y ago
    I didn't form my question too well. I understand all that. What I am asking is, are these function compositions equivalent to equivalent/similar to functions in lambda calculus? I guess my question, is what are the primitive functions here doing?
    - xanderlewis 2y ago
      Well, yes, to the extent that functions are functions are functions (they’re just associations or mappings or whatever you want to call them). Maybe your question boils down to asking something more general like: what’s the difference between functions to a computer scientist (or a programmer) and functions to a mathematician? That is, are ‘functions’ in C (or lambda calculus), say, the same ‘functions’ we talk about in calculus? The answer to that is: in this case, because these are quite simple functions (sums and products and compositions thereof) they’re the same. In general, they’re a bit different. The difference is basically the difference between functional programming and ‘traditional’ programming. If you have state/‘side effects’ of functions, then your function won’t be a function in the sense of mathematics; if the return value of your function depends entirely on the input and doesn’t return different values depending on whatever else is happening in the program, then it will be. Since you’re asking about lambda calculus in particular, the answer is that they’re the same because lambda calculus doesn’t have state. It’s ‘purely functional’ in that sense. >I guess my question, is what are the primitive functions here doing? I’m not really sure what you mean. They’re doing what functions always do. Every computer program is abstractly a (partial) function. Does that help, or have I misunderstood?

dirkc 2y ago

When I did "AI" it would have meant the sigmoid function, these days it's something like ReLU.

esafak 2y ago

functions, which you can compose to increase their expressiveness, and run gradient descent on to train. The success of deep learning is basically attributable to composable (expressive), differentiable (learnable) functions. The "deep" moniker alludes to the compositionality.