10 ms·
Why programming languages matter [video]
- Supermancho 3y agohttps://www.youtube.com/watch?v=JqYCt9rTG8 https://www.youtube.com/watch?v=JqYCt9rTG8 is not available anymore.
- gabrielsroka 3y agohttps://youtu.be/JqYCt9rTG8g https://youtu.be/JqYCt9rTG8g Missing the last letter
- jokoon 3y agooddly, there are very few job positions to work in things related to programming languages.
- anon291 3y agoI feel like every major company is hiring for AI compiler engineers right now (based on my inbox at least). May not be directly related to general 'programming languages', but my take, as someone in the industry, is that all the PL people are working on this right now.
- synergy20 3y agosame here, some companies are hiring a compiler team for AI in fact. I was told Rice and UIUC provide the best compiler program, though not necessarily AI related, should be similar though.
- krick 3y agoWhat does it mean, specifically, "compiler team for AI"? I get it that it's some hot new trend from the last 2 posts, but I'm struggling to imagine what exactly the perfect product should look like and why everyone wants it so bad, allegedly.
- recursivecaveat 3y agoIt's just about compiling a neural net down to run as efficiently as possible. Either on GPU, CPU, or your own accelerator. Neural nets are very computational intensive, while being pretty uniform internally and as a class. Everyone making silicon and a fair few other companies as well has an AI compiler team right now. At the moment the hot product would just be LLM tokens for as few cents each as possible.
- jokoon 3y agoI believe it's either related to the new AI-specialized chips, or maybe to the factoring or neural network graphs. Any specialized domain tend to have its own domain specific language, so obviously it would be true for AI, too.
- synergy20 3y agocorrect, every AI chip maker needs their own compiler team these days
- anon291 3y agoAi problems boil down to compiling linear algebra problems onto very complicated chips. In standard processing, the code is so branchy that we often resort to heuristics in order to get 'good enough' perf. The FLOPS difference between a cpu and gpu is huge. It makes things that are intractable on cpus possible. Without gpus there is no deep learning. That being said, writing code for gpus by relying on cpu compilers will result in terrible perf. In order to take advantage of the hardware you have to take into account minute details of the architecture that most cpu compilers ignore. Cache oblivious algorithms are algorithms that know that there is a cache but don't rely on particular cache sizes. It's the way a lot of cpu code is written because it means not having to deal with particulars. On gpus, particulars matter. For example, to compile a matrix multiply on an Nvidia GPU, you cant just use vectorized multiplies and adds. No. In order to achieve max performance you need to utilize the warp level matrix multiply instruction which requires that you split an arbitrarily sized matrix into the perfect native tensor sizes and then orchestrate the memory loads (which are asynchronous on gpus, and transparently synchronous on cpus) correctly. If you don't you waste millions of dollars (literally). So whereas on a cpu you might just modify your matrix multiply loops to get contiguous memory access and add some vectorization in and cross your fingers, on a gou your compiler needs to take the trivial three nested loop algorithm, look up the cache size particulars and instruction capabilities for the particular generation of the chip, and then rewrite the loop nesting to make it optimal. All while making sure you don't introduce further memory hazards (bank conflicts), etc. So your simple three nestled loop algorithm gets turned into a nine nested loop monstrosity. The stakes are much higher here and the optimizations much different. Whereas on a cpu, we kind of give up with the branching complexity, and just do our best since we never truly know the state of the program, on gpus, the algorithms being executed are extremely amenable to static analysis so we do that, and optimize the shit out of them.
- seanmcdirmid 3y agoThe hot thing is AI now, but you can sneak PL into a wide variety of SWE jobs.
- mjfl 3y agoVery little innovation in programming languages has happened regarding new realities at the hardware level especially transition from serial to parallel execution.
- nomel 3y agoDoes anyone have any good counterpoints to this? From my, naive, perspective, this seems to be relatively true. I've always assumed that, by now, I would be able to write code, in a semi mainstream language, and it would be made somewhat parallel, by the compiler. No need for threads, or me thinking of it. There's projects like https://polly.llvm.org https://polly.llvm.org, but I guess I assumed there would be more progress through the decades.
- tylerhou 3y agoTake a look at Halide, which can autovectorize and multi-thread graphics computations (but does require a restricted language).
- nomel 3y agoI was thinking outside of "embarrassingly parallel" [1] type work. :) But, that is fair. [1] https://en.wikipedia.org/wiki/Embarrassingly_parallel#:~:text=In%20parallel%20computing%2C%20an%20embarrassingly,a%20number%20of%20parallel%20tasks https://en.wikipedia.org/wiki/Embarrassingly_parallel#:~:tex....
- 7734128 3y agoEspecially when doing things like mapping or list comprehension. I'd love to be able to do operations on collections in parallel by simply marking my functions as pure. Just a simple xs.map {x -> f(x)} combined with "f" marked as pure confirmed by the compiler to make the magic happen.
- jcranmer 3y agoProving legality of transformations in the compiler is frequently impossible. Consequently, the main mode of implementation has been to essentially think of the problems in terms of the user saying that this loop is parallel, please make it run in parallel. OpenMP or Rust's rayon crate, for example. The other similar innovation has been programming SIMD as if each lane were an independent thread, which is essentially the model of ispc or CUDA (or #pragma omp simd, natch). The other big impossible task is that most code isn't written to be able to take advantage of theoretical autoparallelization--you really want data to be in struct-of-arrays format, but most code tends to be written in array-of-struct format. This means that vectorization cost model (even if proven, whether by user assertion or sufficiently smart compiler, legal) sees it needs to do a lot of gathers and scatters and gives up on a viable path to vectorization really quickly.
- vimisntgood 3y ago[flagged]
- pdimitar 3y agoProgramming languages do matter but not as much as many people think f.ex. the HN crowd has a soft spot for LISP, and most of the don't even have proper parallel execution (and no I am not talking OS threads, having direct access to those should honestly be removed at one point). And sure, Racket and a few others are "Working on having an actor model". Wake me up when they achieve it. I am betting on somewhere in the 2030s, best case scenario. A combination of a good and restrictive language compiler (Rust, OCaml, Haskell) and an amazing runtime (Erlang) is the sweet spot that everyone should be aiming at. If anything, I am very tired of seeing yet another LISP dialect -- or any other language really, come to think of it now -- being announced. Many of us the programmers love to play and going to check on these languages is IMO taking away precious mind-share. If anything, in my eyes it's exactly because programming languages matter is the reason why we should have less of them. We should start folding some languages inside others. (Or abandon them.)
- nick_ 3y agoJust FYI: Swift has actor model. Pony is built around actor model.
- pdimitar 3y agoSure. But now we get to the really hairy problem: library coverage and community support. That's why I think most languages should start converging together already. IMO we the programmers scatter ourselves too much.