6 ms·

Sorry about not seeing your reply yesterday, HN is not as helpful as Reddit so I have to scan for them manually which can be error prone. I just went through yo

Sorry about not seeing your reply yesterday, HN is not as helpful as Reddit so I have to scan for them manually which can be error prone. I just went through your talk. It was definitely interesting. Despite my confidence in the high-performance abstractions that Spiral allows, I do not have much experience in doing high performance optimizations myself so such field reports motivate my efforts. So Julia can definitely do flexible data structure layouts, but looking at your code I am starting to understand why it took an expert such as yourself to show me this example. Had you not told me that the link is to a map kernel I would have difficulty figuring it out on my own. This shifts my objections from 'Julia cannot do it', to 'macro heavy code is quite hard to grasp'. In Spiral you could have written all of this without the need for macros. Spiral does have (text) macros, but their purpose is to do language interop and not abstraction. The use of macros is a typical anti-pattern in languages whose type systems are too weak for the problem at hand - I am decently sure that with sufficient macro magic any language that has them would have been capable of performing these same optimizations. One immediate benefit of a powerful type system that I'd like to point out is that Spiral allows reflection over tuples, which is a lot more elegant solution to a problem of a function having variable arguments than the C-style varargs mechanism that I see Julia using in the examples you've shown me. Also, in Spiral reflection is done using pattern matching rather than if statements. Another benefit is that Spiral has no need for separate static array machinery. Because literals can be a part of a variable's type and if not, unless explicitly prevented the literals tend to be propagated through function call boundaries (join points). Assuming they are known at compile time the inlining of tensor dimensions in loops is actually the default behavior in Spiral. I've yet to do benchmarking to see how helpful that is, but I am sure it would help reduce register pressure. Your examples are not quite enough to get me to apologize for my rude behavior, but are enough to get me to shift my views a little so I'll change the offending sentence so it reflects reality more accurately.

simondanisch 8y ago
On Julia 0.7, the broadcast kernel will look much more like this (and actually the version used in CuArrays/CLArrays currently also doesn't use those macros): https://github.com/jrevels/MixedModeBroadcastAD.jl/blob/master/src/MixedModeBroadcastAD.jl#L80 https://github.com/jrevels/MixedModeBroadcastAD.jl/blob/mast... let I = @cuda_index(shape) @inbounds dest[I] = bc[I] end There are several things to point out here: 1) The macro @cuda_index doesn't really need to be a macro, but like this it's the shortest way to check the index and return when out of range (in CUDA/OpenCL you often need to schedule a loop with more iterations than you need). 2) the `dest[I] = bc[I]` does all the unzipping of arguments and dispatch to the correct fast getindex for a certain layout of dest/bc via the type system + tuples of the arguments (not using C-style varargs). This simple function makes it possible to get all the advantages of julia's lazy fused broadcasting to work. Feel free to checkout this great blog article to see what's possible with the new lazy broadcasting: https://julialang.org/blog/2018/05/extensible-broadcast-fusion https://julialang.org/blog/2018/05/extensible-broadcast-fusi... You can also look directly at the broadcasting code in Julia Base, which is already a bit bloated to deal with lots of edge cases, but is still pretty readable in parts: https://github.com/JuliaLang/julia/blob/master/base/broadcast.jl#L539 https://github.com/JuliaLang/julia/blob/master/base/broadcas...
- abstractcontrol 8y ago
  I like how the broadcasting comes out in the end in Julia. Besides that Julia has quite many niceties that I could only wish Spiral had and I do think that they matter significantly even if they are not vital. This is definitely praiseworthy work. Since I haven't done so and I really should have, if you or anyone else still looking at this thread are curious how GPU programming without macros looks like then take a peek at this: https://github.com/mrakgr/The-Spiral-Language/blob/3a94ca644160310cb494c92ebf8c6a5ed9836f5f/Learning/CudaModule.fs#L1289 https://github.com/mrakgr/The-Spiral-Language/blob/3a94ca644... Comparing Spiral and Julia code that has been linked so far, I feel that no one would ever guess without knowing ahead of time that Spiral is a static language and Julia dynamic based on these examples. As an example of this in action, here is how layer normalization is implemented using the 'seq_broadcast' kernel which does a sequence of reductions and broadcasts in registers. It is also used to implement softmax for example. https://github.com/mrakgr/The-Spiral-Language/blob/3a94ca644160310cb494c92ebf8c6a5ed9836f5f/Learning/LearningModule.fs#L171 https://github.com/mrakgr/The-Spiral-Language/blob/3a94ca644...
  - simondanisch 8y ago
    >GPU programming without macros looks like then take a peek at this: I'm not sure if you missed my examples, or if there is some other missunderstanding going on ;) GPU code in Julia works 100% without macros, but you are free to use them, so people do ;) Also, you can pretty much write the gpu kernels like pseydo code, not sure how much simpler it can get. With your linked spiral code, I can't even really tell where the algorithm starts and where the setup code ends - which of course might easily be attributed to my unfamiliarity with Spiral ;) I wrote an article some time ago about generic programming with Julia's gpu infrastructure: https://techburst.io/writing-extendable-and-hardware-agnostic-gpu-libraries-b21c145a8dad https://techburst.io/writing-extendable-and-hardware-agnosti... If you can make concrete examples of how things can be simpler than this, I'd be delighted to hear them :) Cuda only (for no reason actually) and with lots of macros, here is another article about gpu programing in Julia: https://mikeinnes.github.io/2017/08/24/cudanative.html https://mikeinnes.github.io/2017/08/24/cudanative.html Do you have any benchmarks for the softmax kernel? If that kernel has optimal performance, it would be quite interesting. If it's sup par, it looks much longer than a simple version.
StefanKarpinski 8y ago
You keep making assertions based on completely false assumptions about Julia and its type system. It might come across better if you posed these assertions as questions instead. A few counterpoints... > One immediate benefit of a powerful type system that I'd like to point out is that Spiral allows reflection over tuples, which is a lot more elegant solution to a problem of a function having variable arguments than the C-style varargs mechanism that I see Julia using in the examples you've shown me. * Julia's type system can reflect on tuples just fine at both compile time and run time and we do so all the time. When tuple size and contents are statically knowable, there is no run time overhead. * Julia's varargs are represented as tuples, which can be and are reflected on by the compiler. Julia's varargs has nothing to do with C varargs except superficial usage similarity. > Another benefit is that Spiral has no need for separate static array machinery. Because literals can be a part of a variable's type and if not, unless explicitly prevented the literals tend to be propagated through function call boundaries (join points). * The fact that Julia's default array types doesn't include static dimensions is a design _choice_ not a type system limitation. Specializing on the exact dimensions of an array is _not_ usually what you want to do in general computational work. If it _is_ what you want to do, you can use the StaticArrays package. * Literals can be type parameters in Julia too—this is done all the time. Indeed, as I said elsewhere in this thread, type parameters can be other types, integers, floats, symbols, tuples, and recursive constructions thereof. * The StaticArrays package is implemented in pure Julia, demonstrating that the type system is perfectly capable of reflecting on tuples and having dimension sizes and tuples of them as type parameters—that's how the static array types are implemented. The compiler generates completely unrolled, fully inlined code for static array operations since the dimensions are known at compile time. > Assuming they are known at compile time the inlining of tensor dimensions in loops is actually the default behavior in Spiral. I've yet to do benchmarking to see how helpful that is, but I am sure it would help reduce register pressure. I understand that Spiral is targeted at a context (GPUs) where static arrays are a good default. The fact that Julia does not take that approach as a default is not a type system limitation (as proven by the existence of StaticArrays), it's a choice based on the fact that static arrays are not the right default for general numerical computing. There are many situations where being forced to know array dimensions at compile time is not helpful or even actively problematic. > Your examples are not quite enough to get me to apologize for my rude behavior, but are enough to get me to shift my views a little so I'll change the offending sentence so it reflects reality more accurately. I'm unclear on why any examples are necessary to apologize for rudeness. Being right doesn't justify being rude. As far as I can tell, the relationship between Spiral and Julia's approaches is roughly: * Spiral is static and primarily focused on statically known, fixed sized arrays and generating good code for GPUs. * Julia is dynamic, and primarily focused on flexibility along with C-level performance on CPUs. * Julia partially covers Spirals target territory with StaticArrays and GPU code generation capabilities. * Spiral does not, on the other hand address (or aim) to the more general array computing that Julia supports. Given the substantial overlap in capabilities, it seems like dismissing Julia out of hand for the application areas you're interested is both unwarranted and unwise. You may want to be a bit less dismissive and understand it and learn from it instead. In particular, Spiral's type system is not more powerful than Julia's. They are different since Spiral's is static and forces types to be known and checked at compile time (as far as I understand). Julia's type system relaxes this, allowing types to be unknown until much later—and as a tradeoff, checked much later. I have yet to see anything that Spiral's type system can express that cannot be expressed in Julia's. Of course, we're also cheating since Julia doesn't do type checking and therefore having a highly expressive type system doesn't really cost us much.

simondanisch 8y ago

On Julia 0.7, the broadcast kernel will look much more like this (and actually the version used in CuArrays/CLArrays currently also doesn't use those macros): https://github.com/jrevels/MixedModeBroadcastAD.jl/blob/master/src/MixedModeBroadcastAD.jl#L80 https://github.com/jrevels/MixedModeBroadcastAD.jl/blob/mast... let I = @cuda_index(shape) @inbounds dest[I] = bc[I] end There are several things to point out here: 1) The macro @cuda_index doesn't really need to be a macro, but like this it's the shortest way to check the index and return when out of range (in CUDA/OpenCL you often need to schedule a loop with more iterations than you need). 2) the `dest[I] = bc[I]` does all the unzipping of arguments and dispatch to the correct fast getindex for a certain layout of dest/bc via the type system + tuples of the arguments (not using C-style varargs). This simple function makes it possible to get all the advantages of julia's lazy fused broadcasting to work. Feel free to checkout this great blog article to see what's possible with the new lazy broadcasting: https://julialang.org/blog/2018/05/extensible-broadcast-fusion https://julialang.org/blog/2018/05/extensible-broadcast-fusi... You can also look directly at the broadcasting code in Julia Base, which is already a bit bloated to deal with lots of edge cases, but is still pretty readable in parts: https://github.com/JuliaLang/julia/blob/master/base/broadcast.jl#L539 https://github.com/JuliaLang/julia/blob/master/base/broadcas...

abstractcontrol 8y ago
I like how the broadcasting comes out in the end in Julia. Besides that Julia has quite many niceties that I could only wish Spiral had and I do think that they matter significantly even if they are not vital. This is definitely praiseworthy work. Since I haven't done so and I really should have, if you or anyone else still looking at this thread are curious how GPU programming without macros looks like then take a peek at this: https://github.com/mrakgr/The-Spiral-Language/blob/3a94ca644160310cb494c92ebf8c6a5ed9836f5f/Learning/CudaModule.fs#L1289 https://github.com/mrakgr/The-Spiral-Language/blob/3a94ca644... Comparing Spiral and Julia code that has been linked so far, I feel that no one would ever guess without knowing ahead of time that Spiral is a static language and Julia dynamic based on these examples. As an example of this in action, here is how layer normalization is implemented using the 'seq_broadcast' kernel which does a sequence of reductions and broadcasts in registers. It is also used to implement softmax for example. https://github.com/mrakgr/The-Spiral-Language/blob/3a94ca644160310cb494c92ebf8c6a5ed9836f5f/Learning/LearningModule.fs#L171 https://github.com/mrakgr/The-Spiral-Language/blob/3a94ca644...
- simondanisch 8y ago
  >GPU programming without macros looks like then take a peek at this: I'm not sure if you missed my examples, or if there is some other missunderstanding going on ;) GPU code in Julia works 100% without macros, but you are free to use them, so people do ;) Also, you can pretty much write the gpu kernels like pseydo code, not sure how much simpler it can get. With your linked spiral code, I can't even really tell where the algorithm starts and where the setup code ends - which of course might easily be attributed to my unfamiliarity with Spiral ;) I wrote an article some time ago about generic programming with Julia's gpu infrastructure: https://techburst.io/writing-extendable-and-hardware-agnostic-gpu-libraries-b21c145a8dad https://techburst.io/writing-extendable-and-hardware-agnosti... If you can make concrete examples of how things can be simpler than this, I'd be delighted to hear them :) Cuda only (for no reason actually) and with lots of macros, here is another article about gpu programing in Julia: https://mikeinnes.github.io/2017/08/24/cudanative.html https://mikeinnes.github.io/2017/08/24/cudanative.html Do you have any benchmarks for the softmax kernel? If that kernel has optimal performance, it would be quite interesting. If it's sup par, it looks much longer than a simple version.

StefanKarpinski 8y ago

You keep making assertions based on completely false assumptions about Julia and its type system. It might come across better if you posed these assertions as questions instead. A few counterpoints... > One immediate benefit of a powerful type system that I'd like to point out is that Spiral allows reflection over tuples, which is a lot more elegant solution to a problem of a function having variable arguments than the C-style varargs mechanism that I see Julia using in the examples you've shown me. * Julia's type system can reflect on tuples just fine at both compile time and run time and we do so all the time. When tuple size and contents are statically knowable, there is no run time overhead. * Julia's varargs are represented as tuples, which can be and are reflected on by the compiler. Julia's varargs has nothing to do with C varargs except superficial usage similarity. > Another benefit is that Spiral has no need for separate static array machinery. Because literals can be a part of a variable's type and if not, unless explicitly prevented the literals tend to be propagated through function call boundaries (join points). * The fact that Julia's default array types doesn't include static dimensions is a design _choice_ not a type system limitation. Specializing on the exact dimensions of an array is _not_ usually what you want to do in general computational work. If it _is_ what you want to do, you can use the StaticArrays package. * Literals can be type parameters in Julia too—this is done all the time. Indeed, as I said elsewhere in this thread, type parameters can be other types, integers, floats, symbols, tuples, and recursive constructions thereof. * The StaticArrays package is implemented in pure Julia, demonstrating that the type system is perfectly capable of reflecting on tuples and having dimension sizes and tuples of them as type parameters—that's how the static array types are implemented. The compiler generates completely unrolled, fully inlined code for static array operations since the dimensions are known at compile time. > Assuming they are known at compile time the inlining of tensor dimensions in loops is actually the default behavior in Spiral. I've yet to do benchmarking to see how helpful that is, but I am sure it would help reduce register pressure. I understand that Spiral is targeted at a context (GPUs) where static arrays are a good default. The fact that Julia does not take that approach as a default is not a type system limitation (as proven by the existence of StaticArrays), it's a choice based on the fact that static arrays are not the right default for general numerical computing. There are many situations where being forced to know array dimensions at compile time is not helpful or even actively problematic. > Your examples are not quite enough to get me to apologize for my rude behavior, but are enough to get me to shift my views a little so I'll change the offending sentence so it reflects reality more accurately. I'm unclear on why any examples are necessary to apologize for rudeness. Being right doesn't justify being rude. As far as I can tell, the relationship between Spiral and Julia's approaches is roughly: * Spiral is static and primarily focused on statically known, fixed sized arrays and generating good code for GPUs. * Julia is dynamic, and primarily focused on flexibility along with C-level performance on CPUs. * Julia partially covers Spirals target territory with StaticArrays and GPU code generation capabilities. * Spiral does not, on the other hand address (or aim) to the more general array computing that Julia supports. Given the substantial overlap in capabilities, it seems like dismissing Julia out of hand for the application areas you're interested is both unwarranted and unwise. You may want to be a bit less dismissive and understand it and learn from it instead. In particular, Spiral's type system is not more powerful than Julia's. They are different since Spiral's is static and forces types to be known and checked at compile time (as far as I understand). Julia's type system relaxes this, allowing types to be unknown until much later—and as a tradeoff, checked much later. I have yet to see anything that Spiral's type system can express that cannot be expressed in Julia's. Of course, we're also cheating since Julia doesn't do type checking and therefore having a highly expressive type system doesn't really cost us much.