6 ms·

I might be persuaded to see things from your point of view if you can show that Julia's tensors meet my criteria. I might be wrong about this - I do not know mu

I might be persuaded to see things from your point of view if you can show that Julia's tensors meet my criteria. I might be wrong about this - I do not know much about Julia and it might very well be capable of this. But these are the kinds of things that if it were capable of, it would advertise. If it can do it I'll apologize and change the line to show Julia in a more positive light. The reason why I started talking about inlining of monads here is because ultimately they are just functions, and Spiral's tensors are not part of the language, but just functions as well. So if it cannot do monads then it probably cannot do tensors to satisfaction either. It was pointed out to me that tensor slicing could serve as substitute for partial application. What I am wondering next is the degree of flexibility Julia's tensors have in the context of GPU kernels. Let me illustrate this with a short example. The following is how the inplace map Cuda kernel is made in Spiral. It seems simple, but I'll go at it from the top to show the characteristics and flexibility that Spiral's tensors have. met map' w f in out = inl in, out = zip in, zip out assert (in.dim = out.dim) "The input and output dimensions must be equal." inl in = flatten in |> to_dev_tensor inl out = flatten out |> to_dev_tensor inl in_a :: () = in.dim inl blockDim = 128 inl gridDim = min 64 (divup (s in_a) blockDim) w.run { blockDim gridDim kernel = cuda // Lexical scoping rocks. grid_for {blockDim gridDim} .x in_a {body=inl {i} -> inl out = out i inl in = in i out .set (f in.get out.get) } } The first line is `inl in, out = zip in, zip out`. Since the inputs to to the map function can be arbitrary data structures, the `zip` iterates over it and zips the tensors in it into a single one. In Spiral the tensors have a tuple of arrays layout by default, and it is absolutely trivial to switch between AOT and TOA representations. In fact you can mix and match tensors that have both layouts internally. Zipping tensors in Spiral just merges them into one with the TOA layout though the subtensors could be AOT. Dealing with these layout transformations is a significant issue in numerical computation that Spiral solves completely. The reason why I have an impression that Julia cannot do this is not because I know for sure. Rather I do not think this can be done within the confines of a parametric type system. Rather I think it would require an intensional type system like the Spiral has. inl in = flatten in |> to_dev_tensor inl out = flatten out |> to_dev_tensor `flatten` checks that the tensor is contiguous and flattens it into a 1d tensor. The `to_dev_tensor` is just some formality that I could not get rid of. It takes the pointers out from the references holding them which allows the tensors to be captured lexically before being passed into the kernel. Without this a type error would occur. kernel = cuda // Lexical scoping rocks. grid_for {blockDim gridDim} .x in_a {body=inl {i} -> inl out = out i inl in = in i out .set (f in.get out.get) } The actual kernel itself is quite small and trivial. `cuda` is just a parser abbreviation for `inl {blockDim gridDim} ->`. In other words it is not a special language feature, but a regular function. When I was doing this in F#, I had to not pass in every single argument of the function by hand - meaning the tensor pointers, their offsets, their sizes, their dimension ranges for each and every tensor, I also had to make wrapper classes on top of that. If a map operation needed more arguments, then I would have needed to copy paste a kernel, adjust it and the wrapper and repeat the testing process all over again, all this with very little type safety. Spiral is not quite on the level of Futhark in terms of ergonomics, but it is a vast improvement over what came before. Note that there is no need for a type annotations anywhere - the function is as generic as it could possibly be in a statically typed language. Because of inlining guarantees all of this has zero overhead. Compared to Spiral, where do Julia and its tensors stand on the axis of generality? Would it be possible to write a simple map as simply as this?

eigenspace 8y ago
I actually don't know anything about these sorts of things. My complaint is merely that you claimed julia is not even worth comparing with when you don't actually know how tensors are implemented in julia or if whatever it is you're doing here is possible in julia. Instead you compare to PyTorch and assume julia has as shit semantics as PyTorch which is false. Perhaps someone who knows more can comment on whether or not julia meets your criteria here but either way, I think your presentation is misleading.
- abstractcontrol 8y ago
  I cannot change my attitude otherwise I would never get anything done. The only principle that I hold dear, and the only way I can truly be fair is that if I am actually wrong I will change my mind. I would not be so rough on Julia if it didn't present itself as a direct competitor to Spiral. Since it presents itself as a numerical computation language capable of replacing C++ and dominating machine learning and GPU programming I will hold it to the same standard I would take Spiral. Any numerical computation language should have tensors generic in dimension, layout and type. That is the point I am trying to hammer home. If you think that Julia's tensors are better than PyTorch's and feel that my equal treatment of them is unfair - I honestly feel that is the wrong way to think about this. Rather than fighting to be the king of a very small hill, it would be better to try climbing a much larger mountain instead. Look inward and be unhappy at what the language has now. Envy Spiral's wonderful tensors. Wouldn't Julia be a much better language if it had its inlining guarantees? And maybe add to that the intensional polymorphism that would allow it to implement them. Why if that happened, I'd lose my reason for working on the language and have to do something else like study reinforcement learning for example. How horrible would that be? :)
  - ChrisRackauckas 8y ago
    >Any numerical computation language should have tensors generic in dimension, layout and type. That is the point I am trying to hammer home. If you think that Julia's tensors are better than PyTorch's and feel that my equal treatment of them is unfair - I honestly feel that is the wrong way to think about this. Rather than fighting to be the king of a very small hill, it would be better to try climbing a much larger mountain instead. Look inward and be unhappy at what the language has now. It's still silly to compare Julia's tensors to PyTorch's because they are generic in dimension, layout, and type unlike PyTorch's and these seem to be exactly the pieces that you're ragging on. This means that your beef against PyTorch is fine, but it doesn't necessarily apply to Julia. I am curious if you can go into more detail about Julia's implementation instead of talking about possible Julia problems through an explanation of PyTorch. >Wouldn't Julia be a much better language if it had its inlining guarantees? Not necessarily. Julia is a dynamic interactive language which is made to be used through a REPL. Although you can increasingly statically compile packages, the standard way to use it is interactive and in this case you do have to watch your compile-times closely. Julia does have opt-in inlining guarantees through the `@inline` macro. It's opt-in instead of opt-out in order to stop unnecessary compile-time growth (inlining isn't always that helpful). To decide when inlining should occur, Julia uses a cost model: https://github.com/JuliaLang/julia/pull/22210 https://github.com/JuliaLang/julia/pull/22210 This then tries to only inline what's necessary to do for performance. Of course, using the fact that Julia can inline what it needs allows things like CUDANative.jl to force inlining and get the kernels it needs. >And maybe add to that the intensional polymorphism that would allow it to implement them. This sounds like a good idea. We should take this over to our community and see whether it would work well in the language. I think the last thing to address is: >Since it presents itself as a numerical computation language capable of replacing C++ and dominating machine learning and GPU programming I will hold it to the same standard I would take Spiral. Not really. Julia is dynamic and interactive. C++ with its full templating is very powerful but can allow its compile-times to explode because its main use case is not with a REPL (and templated C++ code compile times are...). While Julia's generics and speed have increased the domain of what's possible with a dynamic language, it is still made for a much different domain than C++. I would say it's not really in direct competition with Spiral either since the interactive workflow is just so different. Julia's static binary generation is growing in power but will never be as strong as something made for static compilation, and C++/Spiral won't feel like an on-demand calculator where most users are punching commands and getting instant feedback. To me, Spiral looks like a cool alternative to C++/Rust/Go in the scientific computing domain, and package ecosystems that mix Spiral+Julia could be quite powerful.

eigenspace 8y ago

I actually don't know anything about these sorts of things. My complaint is merely that you claimed julia is not even worth comparing with when you don't actually know how tensors are implemented in julia or if whatever it is you're doing here is possible in julia. Instead you compare to PyTorch and assume julia has as shit semantics as PyTorch which is false. Perhaps someone who knows more can comment on whether or not julia meets your criteria here but either way, I think your presentation is misleading.

abstractcontrol 8y ago
I cannot change my attitude otherwise I would never get anything done. The only principle that I hold dear, and the only way I can truly be fair is that if I am actually wrong I will change my mind. I would not be so rough on Julia if it didn't present itself as a direct competitor to Spiral. Since it presents itself as a numerical computation language capable of replacing C++ and dominating machine learning and GPU programming I will hold it to the same standard I would take Spiral. Any numerical computation language should have tensors generic in dimension, layout and type. That is the point I am trying to hammer home. If you think that Julia's tensors are better than PyTorch's and feel that my equal treatment of them is unfair - I honestly feel that is the wrong way to think about this. Rather than fighting to be the king of a very small hill, it would be better to try climbing a much larger mountain instead. Look inward and be unhappy at what the language has now. Envy Spiral's wonderful tensors. Wouldn't Julia be a much better language if it had its inlining guarantees? And maybe add to that the intensional polymorphism that would allow it to implement them. Why if that happened, I'd lose my reason for working on the language and have to do something else like study reinforcement learning for example. How horrible would that be? :)
- ChrisRackauckas 8y ago
  >Any numerical computation language should have tensors generic in dimension, layout and type. That is the point I am trying to hammer home. If you think that Julia's tensors are better than PyTorch's and feel that my equal treatment of them is unfair - I honestly feel that is the wrong way to think about this. Rather than fighting to be the king of a very small hill, it would be better to try climbing a much larger mountain instead. Look inward and be unhappy at what the language has now. It's still silly to compare Julia's tensors to PyTorch's because they are generic in dimension, layout, and type unlike PyTorch's and these seem to be exactly the pieces that you're ragging on. This means that your beef against PyTorch is fine, but it doesn't necessarily apply to Julia. I am curious if you can go into more detail about Julia's implementation instead of talking about possible Julia problems through an explanation of PyTorch. >Wouldn't Julia be a much better language if it had its inlining guarantees? Not necessarily. Julia is a dynamic interactive language which is made to be used through a REPL. Although you can increasingly statically compile packages, the standard way to use it is interactive and in this case you do have to watch your compile-times closely. Julia does have opt-in inlining guarantees through the `@inline` macro. It's opt-in instead of opt-out in order to stop unnecessary compile-time growth (inlining isn't always that helpful). To decide when inlining should occur, Julia uses a cost model: https://github.com/JuliaLang/julia/pull/22210 https://github.com/JuliaLang/julia/pull/22210 This then tries to only inline what's necessary to do for performance. Of course, using the fact that Julia can inline what it needs allows things like CUDANative.jl to force inlining and get the kernels it needs. >And maybe add to that the intensional polymorphism that would allow it to implement them. This sounds like a good idea. We should take this over to our community and see whether it would work well in the language. I think the last thing to address is: >Since it presents itself as a numerical computation language capable of replacing C++ and dominating machine learning and GPU programming I will hold it to the same standard I would take Spiral. Not really. Julia is dynamic and interactive. C++ with its full templating is very powerful but can allow its compile-times to explode because its main use case is not with a REPL (and templated C++ code compile times are...). While Julia's generics and speed have increased the domain of what's possible with a dynamic language, it is still made for a much different domain than C++. I would say it's not really in direct competition with Spiral either since the interactive workflow is just so different. Julia's static binary generation is growing in power but will never be as strong as something made for static compilation, and C++/Spiral won't feel like an on-demand calculator where most users are punching commands and getting instant feedback. To me, Spiral looks like a cool alternative to C++/Rust/Go in the scientific computing domain, and package ecosystems that mix Spiral+Julia could be quite powerful.