4 ms·

If you're interested in advanced layout transformations in Julia, I talked about this a bit here: https://www.youtube.com/watch?v=uecdcADM3hY&t=1658s https://ww

• abstractcontrol 8y ago
  Sorry about not seeing your reply yesterday, HN is not as helpful as Reddit so I have to scan for them manually which can be error prone. I just went through your talk. It was definitely interesting. Despite my confidence in the high-performance abstractions that Spiral allows, I do not have much experience in doing high performance optimizations myself so such field reports motivate my efforts. So Julia can definitely do flexible data structure layouts, but looking at your code I am starting to understand why it took an expert such as yourself to show me this example. Had you not told me that the link is to a map kernel I would have difficulty figuring it out on my own. This shifts my objections from 'Julia cannot do it', to 'macro heavy code is quite hard to grasp'. In Spiral you could have written all of this without the need for macros. Spiral does have (text) macros, but their purpose is to do language interop and not abstraction. The use of macros is a typical anti-pattern in languages whose type systems are too weak for the problem at hand - I am decently sure that with sufficient macro magic any language that has them would have been capable of performing these same optimizations. One immediate benefit of a powerful type system that I'd like to point out is that Spiral allows reflection over tuples, which is a lot more elegant solution to a problem of a function having variable arguments than the C-style varargs mechanism that I see Julia using in the examples you've shown me. Also, in Spiral reflection is done using pattern matching rather than if statements. Another benefit is that Spiral has no need for separate static array machinery. Because literals can be a part of a variable's type and if not, unless explicitly prevented the literals tend to be propagated through function call boundaries (join points). Assuming they are known at compile time the inlining of tensor dimensions in loops is actually the default behavior in Spiral. I've yet to do benchmarking to see how helpful that is, but I am sure it would help reduce register pressure. Your examples are not quite enough to get me to apologize for my rude behavior, but are enough to get me to shift my views a little so I'll change the offending sentence so it reflects reality more accurately.
  • simondanisch 8y ago
    On Julia 0.7, the broadcast kernel will look much more like this (and actually the version used in CuArrays/CLArrays currently also doesn't use those macros): https://github.com/jrevels/MixedModeBroadcastAD.jl/blob/master/src/MixedModeBroadcastAD.jl#L80 https://github.com/jrevels/MixedModeBroadcastAD.jl/blob/mast... let I = @cuda_index(shape) @inbounds dest[I] = bc[I] end There are several things to point out here: 1) The macro @cuda_index doesn't really need to be a macro, but like this it's the shortest way to check the index and return when out of range (in CUDA/OpenCL you often need to schedule a loop with more iterations than you need). 2) the `dest[I] = bc[I]` does all the unzipping of arguments and dispatch to the correct fast getindex for a certain layout of dest/bc via the type system + tuples of the arguments (not using C-style varargs). This simple function makes it possible to get all the advantages of julia's lazy fused broadcasting to work. Feel free to checkout this great blog article to see what's possible with the new lazy broadcasting: https://julialang.org/blog/2018/05/extensible-broadcast-fusion https://julialang.org/blog/2018/05/extensible-broadcast-fusi... You can also look directly at the broadcasting code in Julia Base, which is already a bit bloated to deal with lots of edge cases, but is still pretty readable in parts: https://github.com/JuliaLang/julia/blob/master/base/broadcast.jl#L539 https://github.com/JuliaLang/julia/blob/master/base/broadcas...
    • abstractcontrol 8y ago
      I like how the broadcasting comes out in the end in Julia. Besides that Julia has quite many niceties that I could only wish Spiral had and I do think that they matter significantly even if they are not vital. This is definitely praiseworthy work. Since I haven't done so and I really should have, if you or anyone else still looking at this thread are curious how GPU programming without macros looks like then take a peek at this: https://github.com/mrakgr/The-Spiral-Language/blob/3a94ca644160310cb494c92ebf8c6a5ed9836f5f/Learning/CudaModule.fs#L1289 https://github.com/mrakgr/The-Spiral-Language/blob/3a94ca644... Comparing Spiral and Julia code that has been linked so far, I feel that no one would ever guess without knowing ahead of time that Spiral is a static language and Julia dynamic based on these examples. As an example of this in action, here is how layer normalization is implemented using the 'seq_broadcast' kernel which does a sequence of reductions and broadcasts in registers. It is also used to implement softmax for example. https://github.com/mrakgr/The-Spiral-Language/blob/3a94ca644160310cb494c92ebf8c6a5ed9836f5f/Learning/LearningModule.fs#L171 https://github.com/mrakgr/The-Spiral-Language/blob/3a94ca644...