5 ms·
The important part of Julia is its programming model. The implicitly parallel fork-join model is easy to program and incredibly general. And I'm glad to see a
by dragontamer 6y ago
The important part of Julia is its programming model.
The implicitly parallel fork-join model is easy to program and incredibly general. And I'm glad to see a high-level language embrace it.
-------
I probably should note some other languages of this model: CUDA, OpenCL, ISPC, OpenMP, OpenACC. Most of these other languages are low-level, where you manage memory directly (and GPUs have very little memory per thread. So manual memory management is still hugely important)
But for speed of development, prototyping, and higher level reasons, a language like Julia that implements this "mindset" is going to be hugely important moving forward.
------
The fact that parallel fork-join scales from 2 CPU threads all the way up to thousands of GPU-threads... or CPU + SIMD (such as AVX512), is proof that this methodology is useful. I feel like people are sleeping on this model: its hugely useful for scaling on what I believe to be the computer of the future.
- marmaduke 6y agoClassing for example ISPC as low level invalidates this comment for me; ISPC among others provides a great interface to SIMD model for CPUs that simply isn’t available in Julia.
- dragontamer 6y agoISPC is higher level than its peers, but its still new/delete based manual memory allocation. So in the great scheme of programming languages, its still rather low level (since you're manually handling memory). Indeed: ISPC provides constructs to provide structure-of-arrays, and other low-level memory layout details. This is a good thing: these details have significant implications on the speed of your program. Nonetheless, any language which wrangles with manual details of memory layout, or new/delete based memory allocation, is inevitably going to be classified as low level in my books. > ISPC among others provides a great interface to SIMD model for CPUs that simply isn’t available in Julia. If Julia can compile into GPU-assembly (which is innately SIMD), I'm sure an AVX-based compile could work eventually. They may have to target AVX512 (since most GPU-assembly requires per-thread exec-masks), but the general concept is being proven as Julia can now compile down into PTX or AMDGPU assembly. Julia's compile down to GPU-assembly / SIMD code is not supported in the "general case", only in select circumstances. But that's still an incredible boon for a high-level language.
- marmaduke 6y ago> I'm sure an AVX-based compile could work eventually. Eventually is nice but it’s also easy to imagine that it never gets done. ISPC works today.
- dragontamer 6y agoThe CUDA portion of Julia works today, and you can get 10Tflop or 20Tflop GPUs under $1000 to play with it. It's a different language for a different computer. But the fundamentals are still there, laying the groundwork for the future. Sure, ispc can win on latency. But the raw compute girth of GPU SIMD cannot be underestimated. In any case, Julia has proven itself capable of compiling down to a SIMD instruction set and achieving nearly the full performance of those GPUs. Even if it's not a computer you prefer, the programming model and technology demonstrated here is clear.
- marmaduke 6y agoI agree with this and have the hardware but simply prefer to write CUDA kernels directly instead of in Julia syntax. My comment was more geared to the CPU side of things since I write code whose users don’t usually have GPUs available. There are also problem sizes which don’t fit the GPU’s more stark memory system separation: tens of CPU cores with tens of MBs of cache can move past GPUs in terms of memory bandwidth for such cases so banking on CUDA just doesn’t work for everyone.
- marmaduke 6y ago> but its still new/delete based manual memory allocation I guess we aren’t solving the same problems: memory allocation is trivial is my domain, mapping complex nested control flow in SIMD is the hard part. > Julia can now compile down into PTX or AMDGPU assembly. Sure but Julia-as-syntax is nothing special now, Numba does this for a Python as well.
- dnautics 6y agoyou can and you can't. Julia is composable -- Suppose I want to write a library to find compression polynomials for reed-solomon encoding system (see mary wootters' talks on youtube) for my storage product. I need a LU decomposition algorithm that operates on GF256 (which is just an 8-bit int). But the +/-/x/divide operations are all messed up. I'd have to rewrite LU decomposition. How confident are you that you can get even that right? I'm pretty good at implementing algorithms, but I've messed up LU decomposition before. Then suppose I rewrite the LU decomposition algorithm, using python. Now I want to accelerate the search by running the search on GPUs. I have to re-rewrite the code from scratch. Each GF256 encoding has to have rejiggered operators, and so I need to rewrite custom GPU kernels, then figure out how to resequence the operations (* looks different for each GF256 encoding), etc. This is all SUPER easy in julia.
- staticfloat 6y agoJulia's SIMD programming model is still very much a work in progress; I think we have a way to go in providing the kind of flexibility and control that languages such as ISPC, Halide, TVM, etc... provide. That being said, packages such as SIMD.jl [0], and LoopVectorization.jl [1] are making fantastic progress, to the point that LoopVectorization forms the basis of a legitimate BLAS contender, in pure Julia [2]. It's not totally there yet, but it's close enough that real work is being done in LV at OpenBLAS-like speeds. As an aside, I find it incredible that these kinds of extensions can be built in packages thanks to the fact that Julia's compiler is extensible enough to allow for direct manipulation of the LLVM intrinsics being emitted by user code. [0] https://github.com/eschnett/SIMD.jl https://github.com/eschnett/SIMD.jl [1] https://github.com/chriselrod/LoopVectorization.jl https://github.com/chriselrod/LoopVectorization.jl [2] https://github.com/MasonProtter/Gaius.jl https://github.com/MasonProtter/Gaius.jl
- marmaduke 6y agoIt’s not a jab at Julia, really rather that ISPC provides a workable model that would later on be nice to see elsewhere. > find it incredible that these kinds of extensions can be built in packages thanks to the fact that Julia's compiler is extensible Come on, jeez.. Julia’s compiler is a Lisp-based LLVM driver: of course it can do these things.
- staticfloat 6y ago> It’s not a jab at Julia I didn't take it as such; there are legitimate shortcomings to any tool, I just wanted to provide pointers to other readers that the devs are aware of it, and that there is ongoing development to address it. :) > Come on, jeez.. Julia’s compiler is a Lisp-based LLVM driver: of course it can do these things. As someone who, before Julia, was firmly entrenched in C/C++/Python land, I suppose I am discovering many of these "obvious" things for the first time. :)
- marmaduke 6y ago> I suppose I am discovering many of these "obvious" things sorry for the flippant remark then! it's great to be in discover mode, enjoy ;)
- amelius 6y agoCould you be more precise about what do you mean by fork-join model? Is this similar to how in Go starting a thread is easy and waiting for it to finish and collecting results is easy as well?
- dragontamer 6y agoI don't program in Go, so I can't speak to that. The fork-join model generally starts off with one-thread, often called "The Single" or "The Master". Its where main() begins. But somewhere along the line, you discover a portion of code that needs to be multithreaded. So you fork into many threads: maybe dozens of threads, or thousands in the case of SIMD or GPUs. This multithreaded portion is then "joined", that is, the "master" thread refuses to continue until all children are done executing. Lets say there's a raytracer you are writing. main() { // In the "Single" thread Foo(); Bar(); fork(doRayCastingInParallel(), 5-million); // Spawns 5,000,000-threads, one for each ray // Implicitly wait for all 5-million threads to finish doSomethingElse(); } Now, raycasting has a bunch of things to do, but you can break it up into the same steps for all 5-million threads. doRayCastingInParallel(int myIdx){ doTrigonometry(ray[myIdx]); findCollision(ray[myIdx]); outputColor[myIdx] = calculateColors(ray[myIdx]); } So from both the "single" thread, as well as your "child" thread, it feels like you're writing single-threaded code. But in actuality, your "child" thread is a 5-million spawned copy, executing in gross parallelism. The "Fork-Join" model assigns a different "myIdx" number to each thread. So Ray#500 knows to do things differently than Ray#68121. But otherwise, they run through the same code. --------------- The important bit here is how general this mindset is. Its commonly used for GPU-programming (CPU issues a GPU command to spawn millions of threads often one-per-pixel, or even thousands per pixel). Then the CPU waits for the GPU threads to finish, and then just carries on. But ISPC proved that you can use this fork-join model on SIMD units like AVX512 or SSE. Julia and Python are showing off how you can use it in higher level languages. Etc. etc. OpenMP further proves that fork-join works on normal CPU threads. Maybe there's some other CPU-threading languages (but OpenMP is the one I'm personally familiar with).
- pkulak 6y ago
- CyberDildonics 6y agoFork join parallelism is just one technique for using multiple cores and it is pretty ubiquitous. Most large arrays can be read by multiple threads at one time to let them all come up with part of the answer. I don't think this is specifically an advantage of julia.
- gnufx 6y agoOpenMP, in particular, is not limited to fork-join, and you don't want to be so limited for efficient, scalable numerical systems. Not that you get large-scale parallelism just with OpenMP. A recent keynote of Jack Dongarra's covers replacing the decades-old use of fork-join for (or course) dense linear algebra: https://www.iwomp2020.org/wp-content/uploads/iwomp-2020-K1-Dongarra.pdf https://www.iwomp2020.org/wp-content/uploads/iwomp-2020-K1-D... It's significantly in the past, not the future, of large-scale computation of the sort Julia particularly targets. I don't know why Julia should be stuck with that, though. Parallelism isn't new to Lisp-y systems, of course.
- dragontamer 6y agoI guess I still see Julia as a quick and dirty prototyping language akin to Python and some others. Clearly, Julia has ambitions for greater capabilities. Apparently, Julia now supports some more complicated forms of parallelism: Async, @Spawn (aka: Tasks), and others. But @Thread based for loops are just easier to grok. Overall, Julia is one of the few high-level languages that is explicitly adding in highly-threaded concepts like Fork-join, SIMD, or even GPUs to the language. Yeah, some Python libraries extend these concepts to Python. But something like @Threads for ... end really demonstrates how the fork-join parallelism is a first-class member of the Julia language.
- gnufx 6y agoAny language supporting (current) OpenMP supports fork-join, tasking, SIMD specification, and offload, though you still want distributed memory, and overlapped communication and computation. Baking particular technology into a system isn't a good idea long-term, or even medium-term, unless you wait for the cycle of re-invention, like vector computing. (I don't know if Julia really does that.)