4 ms·

To some extent, it is a matter of taste. Personally, I love functional notation (technically, in PyTorch its chaining) e.g.: x.matmul(y).pow(2).sum() This way

by stared 6y ago

To some extent, it is a matter of taste. Personally, I love functional notation (technically, in PyTorch its chaining) e.g.: x.matmul(y).pow(2).sum() This way we can rite a lot of things, and we don't need to make up a new combination of punctuation marks and special characters for an operation. For example, while one can write in Python: torch.sum((x @ y) 2) I consider it less readable. I mean, here maybe it is fine, but once it gets longer, more complicated, or we want to add new operations, it turns into a mess. Vide "style" section in: https://github.com/stared/thinking-in-tensors-writing-in-pytorch https://github.com/stared/thinking-in-tensors-writing-in-pyt...

enriquto 6y ago
Both of your examples are equally ugly and equally unreadable to me. Why cannot it be something like Σ(x*y)^2 ? Why the extraneous words "torch", "matmul" and the ridiculous operator @ ?
- dunefox 6y ago
  I'm of the same opinion. In Julia it would be sum(A * B) ^ 2 If you define sum as Σ then you can do Σ(A * B) ^ 2
  - enriquto 6y ago
    yes I like Julia very much because it seems to put the natural instincts of mathematicians upfront. When programming in Julia you do not have the impression of fighting against the language continuously. Everything that you want to do is accessible, and it is alright if you implement your computation in the most natural way possible. If I try to do that in python, I always end with an ugly mess of numba, numpy and calls to C code used at different places for different reasons.
  - mkl 6y ago
    I think you mean sum((A*B)^2)? That's one character shorter than pytorch if you import sum out of it, and Σ = torch.sum would be fine in Python too.
    - dunefox 6y ago
      No, the parent example was sum(A * B)^2. I don't think you can use * if you want to backprop, for example in layer regularisation. You'd need to use only PyTorch operations such as torch.mul(A, B).pow(2).sum(). The point here isn't that PyTorch can't look similar to Julia in this small example, rather that I can just use regular, concise Julia syntax - unlike in Python + PyTorch where I need to use PyTorch constructs that are outside of Python.
      - mkl 6y ago
        Well, there's the notational ambiguity! In maths notation, Σ(x * y)^2 would mean Σ((x * y)^2), but in most programming notation, treating Σ as a function, it would be as you say. I'm going with the original formula in https://news.ycombinator.com/item?id=23508661 https://news.ycombinator.com/item?id=23508661. I don't know PyTorch, but regular Python, Numpy, Sympy, etc. seem very similar to Julia in this instance.
        dunefox 6y ago
        You're right, that was my mistake - but I wouldn't say that it's extremely ambiguous in this case, I just wasn't consequent enough in reading the formula. > I don't know PyTorch, but regular Python, Numpy, Sympy, etc. seem very similar to Julia in this instance. If you need PyTorch to record the operations for the backward pass later on I believe you need to use PyTorch versions of *, +, etc.: torch.mul, torch.sub, torch.add, etc. In Julia you can just use built in functions and let Flux handle the backward pass.
- stared 6y ago
  Σ(x*y)^2 is the closest to the traditional notation, which does not mean that it is the best one. I have a strong preference for notations that can be read consistently from left to right (vide pipe operators, chaining, etc). A litmus test is if when I read something I use the word 'of'. If you do anything more complicated that only vector-matrix operations (i.e. a lot of quantum information, all deep learning), with all multiplications you need to specify dimensions somehow. Having only two operators for multiplication is not enough.
- uryga 6y ago
  why is "@" so bad? i mean it's a huge glyph but it's just an operator like +-×÷•, you get used to it
  - dunefox 6y ago
    It's bad because it's a special operator for this operation because Python is not made for scientific programming. That's exactly why you have np.matmul, np.dot, etc. all over the place.
mkl 6y ago
Looks like the two asterisks cancelled each other out in torch.sum((x @ y)**2)

by stared 6y ago

enriquto 6y ago

Both of your examples are equally ugly and equally unreadable to me. Why cannot it be something like Σ(x*y)^2 ? Why the extraneous words "torch", "matmul" and the ridiculous operator @ ?

dunefox 6y ago
I'm of the same opinion. In Julia it would be sum(A * B) ^ 2 If you define sum as Σ then you can do Σ(A * B) ^ 2
- enriquto 6y ago
  yes I like Julia very much because it seems to put the natural instincts of mathematicians upfront. When programming in Julia you do not have the impression of fighting against the language continuously. Everything that you want to do is accessible, and it is alright if you implement your computation in the most natural way possible. If I try to do that in python, I always end with an ugly mess of numba, numpy and calls to C code used at different places for different reasons.
- mkl 6y ago
  I think you mean sum((A*B)^2)? That's one character shorter than pytorch if you import sum out of it, and Σ = torch.sum would be fine in Python too.
  - dunefox 6y ago
    No, the parent example was sum(A * B)^2. I don't think you can use * if you want to backprop, for example in layer regularisation. You'd need to use only PyTorch operations such as torch.mul(A, B).pow(2).sum(). The point here isn't that PyTorch can't look similar to Julia in this small example, rather that I can just use regular, concise Julia syntax - unlike in Python + PyTorch where I need to use PyTorch constructs that are outside of Python.
    - mkl 6y ago
      Well, there's the notational ambiguity! In maths notation, Σ(x * y)^2 would mean Σ((x * y)^2), but in most programming notation, treating Σ as a function, it would be as you say. I'm going with the original formula in https://news.ycombinator.com/item?id=23508661 https://news.ycombinator.com/item?id=23508661. I don't know PyTorch, but regular Python, Numpy, Sympy, etc. seem very similar to Julia in this instance.
      - dunefox 6y ago
        You're right, that was my mistake - but I wouldn't say that it's extremely ambiguous in this case, I just wasn't consequent enough in reading the formula. > I don't know PyTorch, but regular Python, Numpy, Sympy, etc. seem very similar to Julia in this instance. If you need PyTorch to record the operations for the backward pass later on I believe you need to use PyTorch versions of *, +, etc.: torch.mul, torch.sub, torch.add, etc. In Julia you can just use built in functions and let Flux handle the backward pass.
stared 6y ago
Σ(x*y)^2 is the closest to the traditional notation, which does not mean that it is the best one. I have a strong preference for notations that can be read consistently from left to right (vide pipe operators, chaining, etc). A litmus test is if when I read something I use the word 'of'. If you do anything more complicated that only vector-matrix operations (i.e. a lot of quantum information, all deep learning), with all multiplications you need to specify dimensions somehow. Having only two operators for multiplication is not enough.
uryga 6y ago
why is "@" so bad? i mean it's a huge glyph but it's just an operator like +-×÷•, you get used to it
- dunefox 6y ago
  It's bad because it's a special operator for this operation because Python is not made for scientific programming. That's exactly why you have np.matmul, np.dot, etc. all over the place.

mkl 6y ago

Looks like the two asterisks cancelled each other out in torch.sum((x @ y)**2)