4 ms·

I recall an example where Julia had to use macros in order to achieve loop unrolling. I'll dig it up if you want. This is something can be done quite naturally

I recall an example where Julia had to use macros in order to achieve loop unrolling. I'll dig it up if you want. This is something can be done quite naturally in Spiral due to its type system. I find that people often say macros are about syntax, but what they really are is some combination of a parser and a partial evaluator - in other words, their main use seems to doing compile-time function evaluation. Spiral does not have arbitrary CTFE, but can perform any functionally-pure, non side-effecting computation in its type system plus memoization of function calls. One thing of practical interest that makes me doubt Julia's inlining capabilities would be monads. https://github.com/pao/Monads.jl/blob/master/doc/index.rst https://github.com/pao/Monads.jl/blob/master/doc/index.rst "Monads.jl provides a powerful, if relatively slow" The key word here is slow. I haven't seen a single language as of yet apart from Spiral that can inline all their overheads away. They are pervasive in Haskell, and Haskell cannot do it. Scala wants to do it, but it can't. F# and Ocaml cannot optimize their overheads away. If Julia could then it would advertise it. This is something that is subject of academic research in the field of partial evaluation. Speaking as a functional programmer, as a language Julia feels barely functional to me. It has first class functions, but it does not have pervasive partial application and syntax that makes such a style comfortable like functional languages tend to have. It feels more like an imperative language. And speaking as a systems programmer, I know that without inlining guarantees you cannot have optimized functional constructs. It is not just monads. By tracking all the factions and their bodies to exactness and forgoing all heap allocation unless explicitly stated, it becomes much easier to do language interop. It is possible to make functions that capture variables from lexical scope and then pass as arguments to a function that passes them through to the GPU. Julia talks about speed, but barely talks about inlining so I am dubious about its claims. I could go on, but I think this is a large enough sample of my views on Julia. If Julia wants me to use it, then it needs to speak more to my values as a programmer.

ChrisRackauckas 8y ago
I agree that while Julia has a lot of functional programming elements, it's not strictly a functional programming language. It also is built for things where heap-allocated arrays are necessary: you're not going to be scientific computing with 100GB stack-allocated arrays. Having the base arrays be stack-allocated and then having StaticArrays for stack-allocated arrays works well, though that means you do want to use imperative styles a lot when managing the large array operations to cut down on allocations. Spiral and Julia are just very different and cater to different paradigms and applications.
eigenspace 8y ago
I think that when you say >There is no need to consider how tensors are made in a language made for numeric computation like Julia. and then proceed to compare to PyTorch instead of julia is incredibly misleading and unfair to julia. I fail to see how anything you've brought up here excuses that.
- ChrisRackauckas 8y ago
  Indeed, I don't see the relationship between the PyTorch and Julia implementations.
- abstractcontrol 8y ago
  I might be persuaded to see things from your point of view if you can show that Julia's tensors meet my criteria. I might be wrong about this - I do not know much about Julia and it might very well be capable of this. But these are the kinds of things that if it were capable of, it would advertise. If it can do it I'll apologize and change the line to show Julia in a more positive light. The reason why I started talking about inlining of monads here is because ultimately they are just functions, and Spiral's tensors are not part of the language, but just functions as well. So if it cannot do monads then it probably cannot do tensors to satisfaction either. It was pointed out to me that tensor slicing could serve as substitute for partial application. What I am wondering next is the degree of flexibility Julia's tensors have in the context of GPU kernels. Let me illustrate this with a short example. The following is how the inplace map Cuda kernel is made in Spiral. It seems simple, but I'll go at it from the top to show the characteristics and flexibility that Spiral's tensors have. met map' w f in out = inl in, out = zip in, zip out assert (in.dim = out.dim) "The input and output dimensions must be equal." inl in = flatten in |> to_dev_tensor inl out = flatten out |> to_dev_tensor inl in_a :: () = in.dim inl blockDim = 128 inl gridDim = min 64 (divup (s in_a) blockDim) w.run { blockDim gridDim kernel = cuda // Lexical scoping rocks. grid_for {blockDim gridDim} .x in_a {body=inl {i} -> inl out = out i inl in = in i out .set (f in.get out.get) } } The first line is `inl in, out = zip in, zip out`. Since the inputs to to the map function can be arbitrary data structures, the `zip` iterates over it and zips the tensors in it into a single one. In Spiral the tensors have a tuple of arrays layout by default, and it is absolutely trivial to switch between AOT and TOA representations. In fact you can mix and match tensors that have both layouts internally. Zipping tensors in Spiral just merges them into one with the TOA layout though the subtensors could be AOT. Dealing with these layout transformations is a significant issue in numerical computation that Spiral solves completely. The reason why I have an impression that Julia cannot do this is not because I know for sure. Rather I do not think this can be done within the confines of a parametric type system. Rather I think it would require an intensional type system like the Spiral has. inl in = flatten in |> to_dev_tensor inl out = flatten out |> to_dev_tensor `flatten` checks that the tensor is contiguous and flattens it into a 1d tensor. The `to_dev_tensor` is just some formality that I could not get rid of. It takes the pointers out from the references holding them which allows the tensors to be captured lexically before being passed into the kernel. Without this a type error would occur. kernel = cuda // Lexical scoping rocks. grid_for {blockDim gridDim} .x in_a {body=inl {i} -> inl out = out i inl in = in i out .set (f in.get out.get) } The actual kernel itself is quite small and trivial. `cuda` is just a parser abbreviation for `inl {blockDim gridDim} ->`. In other words it is not a special language feature, but a regular function. When I was doing this in F#, I had to not pass in every single argument of the function by hand - meaning the tensor pointers, their offsets, their sizes, their dimension ranges for each and every tensor, I also had to make wrapper classes on top of that. If a map operation needed more arguments, then I would have needed to copy paste a kernel, adjust it and the wrapper and repeat the testing process all over again, all this with very little type safety. Spiral is not quite on the level of Futhark in terms of ergonomics, but it is a vast improvement over what came before. Note that there is no need for a type annotations anywhere - the function is as generic as it could possibly be in a statically typed language. Because of inlining guarantees all of this has zero overhead. Compared to Spiral, where do Julia and its tensors stand on the axis of generality? Would it be possible to write a simple map as simply as this?

ChrisRackauckas 8y ago

I agree that while Julia has a lot of functional programming elements, it's not strictly a functional programming language. It also is built for things where heap-allocated arrays are necessary: you're not going to be scientific computing with 100GB stack-allocated arrays. Having the base arrays be stack-allocated and then having StaticArrays for stack-allocated arrays works well, though that means you do want to use imperative styles a lot when managing the large array operations to cut down on allocations. Spiral and Julia are just very different and cater to different paradigms and applications.

eigenspace 8y ago

I think that when you say >There is no need to consider how tensors are made in a language made for numeric computation like Julia. and then proceed to compare to PyTorch instead of julia is incredibly misleading and unfair to julia. I fail to see how anything you've brought up here excuses that.

ChrisRackauckas 8y ago
Indeed, I don't see the relationship between the PyTorch and Julia implementations.
abstractcontrol 8y ago
I might be persuaded to see things from your point of view if you can show that Julia's tensors meet my criteria. I might be wrong about this - I do not know much about Julia and it might very well be capable of this. But these are the kinds of things that if it were capable of, it would advertise. If it can do it I'll apologize and change the line to show Julia in a more positive light. The reason why I started talking about inlining of monads here is because ultimately they are just functions, and Spiral's tensors are not part of the language, but just functions as well. So if it cannot do monads then it probably cannot do tensors to satisfaction either. It was pointed out to me that tensor slicing could serve as substitute for partial application. What I am wondering next is the degree of flexibility Julia's tensors have in the context of GPU kernels. Let me illustrate this with a short example. The following is how the inplace map Cuda kernel is made in Spiral. It seems simple, but I'll go at it from the top to show the characteristics and flexibility that Spiral's tensors have. met map' w f in out = inl in, out = zip in, zip out assert (in.dim = out.dim) "The input and output dimensions must be equal." inl in = flatten in |> to_dev_tensor inl out = flatten out |> to_dev_tensor inl in_a :: () = in.dim inl blockDim = 128 inl gridDim = min 64 (divup (s in_a) blockDim) w.run { blockDim gridDim kernel = cuda // Lexical scoping rocks. grid_for {blockDim gridDim} .x in_a {body=inl {i} -> inl out = out i inl in = in i out .set (f in.get out.get) } } The first line is `inl in, out = zip in, zip out`. Since the inputs to to the map function can be arbitrary data structures, the `zip` iterates over it and zips the tensors in it into a single one. In Spiral the tensors have a tuple of arrays layout by default, and it is absolutely trivial to switch between AOT and TOA representations. In fact you can mix and match tensors that have both layouts internally. Zipping tensors in Spiral just merges them into one with the TOA layout though the subtensors could be AOT. Dealing with these layout transformations is a significant issue in numerical computation that Spiral solves completely. The reason why I have an impression that Julia cannot do this is not because I know for sure. Rather I do not think this can be done within the confines of a parametric type system. Rather I think it would require an intensional type system like the Spiral has. inl in = flatten in |> to_dev_tensor inl out = flatten out |> to_dev_tensor `flatten` checks that the tensor is contiguous and flattens it into a 1d tensor. The `to_dev_tensor` is just some formality that I could not get rid of. It takes the pointers out from the references holding them which allows the tensors to be captured lexically before being passed into the kernel. Without this a type error would occur. kernel = cuda // Lexical scoping rocks. grid_for {blockDim gridDim} .x in_a {body=inl {i} -> inl out = out i inl in = in i out .set (f in.get out.get) } The actual kernel itself is quite small and trivial. `cuda` is just a parser abbreviation for `inl {blockDim gridDim} ->`. In other words it is not a special language feature, but a regular function. When I was doing this in F#, I had to not pass in every single argument of the function by hand - meaning the tensor pointers, their offsets, their sizes, their dimension ranges for each and every tensor, I also had to make wrapper classes on top of that. If a map operation needed more arguments, then I would have needed to copy paste a kernel, adjust it and the wrapper and repeat the testing process all over again, all this with very little type safety. Spiral is not quite on the level of Futhark in terms of ergonomics, but it is a vast improvement over what came before. Note that there is no need for a type annotations anywhere - the function is as generic as it could possibly be in a statically typed language. Because of inlining guarantees all of this has zero overhead. Compared to Spiral, where do Julia and its tensors stand on the axis of generality? Would it be possible to write a simple map as simply as this?