6 ms·
I think python is dominant in AI, because the python-C relationship mirrors the CPU-GPU relationship. GPUs are extremely performant, and also very hard to code
by anonylizard 3y ago
I think python is dominant in AI, because the python-C relationship mirrors the CPU-GPU relationship.
GPUs are extremely performant, and also very hard to code in, so people just use highly abstracted API calls like pytorch to command the GPU.
C is very performant, and hard to code in, so people just use python as a abstraction layer over C.
Its not clear if people need to understand GPUs that much (Unless you are deep in AI training/ops land). In time, since moore's law has ended and multithreading becomes the dominant mode of speed increases, there'll probably be brand new languages dedicated to this new paradigm of parallel programming. Mojo is a start.
- sitkack 3y agoMoore’s law is far from over and multithreading is not the answer. Your opening sentence is spot on tho.
- bmc7505 3y agoCare to elaborate?
- guyomes 3y agoNo idea about the future of the Moore's law precisely. Yet recent research results show that there is still room for faster semiconductors, as discussed on HN [0]. [0]: https://news.ycombinator.com/item?id=38201624 https://news.ycombinator.com/item?id=38201624
- bmc7505 3y agoI mean why isn’t multithreading the answer?
- samus 3y agoMultithreading has the following disadvantages: * Overhead: the overhead to start and manage multiple threads is considerable in practice. Most multithreaded algorithms are in fact slower than optimized serial implementations when n_threads=1 * Communication: threads have to communicate with each other and synchronize access to shared resources. "Embarrassingly parallel" problems don't require synchronization, but many interesting problems are not of that kind. * Amdahl's law: there is a point of diminishing returns on parallelizing an application since it quite likely contains parts that are not easily parallelized.
- bmc7505 3y agoWell, yes. But assuming the question is how to reduce overall latency, what alternative is there besides algorithmic improvements or increasing clock frequency?
- samus 3y agoRecently, SIMD has been increasingly used to make use of the increased number of compute units in a CPU. Sure, it's not as easy and straightforward to use as multithreading (I never expected to say that about multithreading), but libraries and programming languages make more and more use of them. Edit: latency is difficult. But accellerating CPUs and using GPUs for compute was never about latency. Most I/O bottlenecks are because CPUs have sped up so much and left the rest of the platform in the dust. Much of it is also due to fundamental limitations due to the speed of light. Increasing throughput is always easier that reducing latency.
- bmc7505 3y agoI guess I was thinking more of multithreading as an umbrella term for parallelism but I see your point. If you squint, SIMD and multithreading are the same problem. What I mean is, if you have a computable function you want to evaluate on a finite amount of bits, the problem of how to physically layout and schedule the computation to minimize latency is very similar to maximizing throughput. From a circuit complexity standpoint, you could evaluate a wide but shallow circuit to evaluate the function in a nanosecond, or a deep but narrow circuit that takes eons. Whether parts of that circuit are evaluated synchronously or asynchronously is immaterial, although synchronous computation does seem easier to reason about and the UX is more user-friendly from a programmability standpoint. I agree with you the fundamental limitation is the speed of light if you are width-bounded (i.e., if physical space is the dominating constraint).
- anticensor 3y agoYou forgot Wirth's law: software complexity compensates increase in speed of hardware.
- Kamq 3y ago> Moore’s law is far from over and multithreading is not the answer. Wut? We hit the power wall back in 2004. There was a little bit of optimization around the memory wall and ilp wall afterwards, but really, cores haven't gotten faster since. It's been all about being able to cram more cores in since then, which implies at least multi-threading, but multi-processing is basically required to get the most out of a cpu these days.
- ikura 3y agoMoore's law is "the observation that the number of transistors in an integrated circuit doubles about every two years". For a while clock speed was a proxy for that metric, but it's not the 'law' itself.
- Kamq 3y agoYeah, but today number of cores is the rough proxy for that metric. How do you operate in that world if "multithreading isn't the answer"?
- samus 3y agoModern CPUs contain a lot more computing units than cores. For a while, hyperthreading was thought to be a useful way to make use of them. More recently, people have turned to advanced instruction sets like SSE and AVX.
- FridgeSeal 3y agoThose things aren’t mutually exclusive. Also firstly, I suspect there’s more “low hanging fruit” in making more software make use of more cores. We’re increasingly getting better languages, tooling and libs for multi threading stuff, and it’s far more in the realm of your average developer than writing SIMD compatible code and making sure your code can pipeline properly.
- samus 3y ago
- danielmarkbruce 3y agoGPUs aren't that difficult to program. CUDA is fairly straightforward for many tasks and in many cases there is an easy 100x improvement in processing speed just sitting there to be had with <100 lines of code.
- z2h-a6n 3y agoSure, but if you've never used CUDA or any other GPU framework, how many lines of documentation do you need to read, and how many lines of code are you likely to write and rewrite and delete before you end up with those <100 lines of working code?
- danielmarkbruce 3y agoThey have nice getting started guides. Try it and see. It's... really pretty simple. There is a reason they've built a trillion $$ company - they've done a great job.
- z2h-a6n 3y agoOk, I tried it (for as long as my limited free time and interest in CUDA allows). The closest I came to a getting started guide is this [0], which by my (perhaps naive count) is 25561 lines of documentation, and I would probably need to learn more C++ to understand it in detail. I'm sure CUDA is great, and if I had more free time and/or better reasons to improve the performance of my code it would probably be great for me. My point was mainly that a few lines of code which may be trivial for one person to write may not be for someone else with different experience. Depending on what the code is being used for even a vast increase in performance may not be worth the extra time it takes to implement it. [0] https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html https://docs.nvidia.com/cuda/cuda-c-programming-guide/index....
- danielmarkbruce 3y agohttps://developer.nvidia.com/blog/even-easier-introduction-cuda/ https://developer.nvidia.com/blog/even-easier-introduction-c...
- mft_ 3y agoI've wondered for a while: is there a space for a (new?) language which invisibly maximises performance, whatever hardware it is run on? As in, every instruction, from a simple loop of calculations onward, is designed behind the scenes so that it intelligently maximises usage of every available CPU core in parallel, and also farms everything possible out to the GPU? Has this been done? Is it possible?
- teaearlgraycold 3y agoI’m not sure if that’s a good idea at the moment, but we should start with making development with vector instructions more approachable. The code should look more or less the same as working with u64s.
- howling 3y agoYou might be interested in https://github.com/HigherOrderCO/HVM https://github.com/HigherOrderCO/HVM
- runlaszlorun 3y agoHVM looks very interesting. Thx for posting.
- jfoutz 3y agoThere's definitely a space for it. It may even be possible. But if you consider the long history of lisp discussions (flamewars?) about "a sufficiently smart compiler" and comparisons to C. Or maybe Java vs C++, it seems unlikely. At least very very difficult. There are little bits of research on algorithm replacement. Like, have the compiler detect that you're trying to sort, and generate the code for quick sort or timsort. it works, kinda. There are a lot of ways to hide a sort in code, and the compiler can't readily find them all.
- Simpliplant 3y agoNot exactly it but Mojo sounds closest from available options https://www.modular.com/mojo https://www.modular.com/mojo
- 3y ago
- sgbeal 3y ago> C is very performant, and hard to code in, so people just use python as a abstraction layer over C. C is a way of life. Those of us who code exclusively, or nearly so, in C cannot stomach python's notion of "significant white-space."
- yoyohello13 3y agoI code in both (c for hobbies and python professionally) and “significant white space” is a non-issue if you spend any amount of time getting used to it. Complaining about significant white-space is like complaining that lisp has too many parentheses. It’s an aesthetic preference that just doesn’t matter in practice.
- adventured 3y agoA form of Sayre's Law is very common in tech (eg spaces vs tabs; framework vs framework; language vs language).
- adolph 3y ago> cannot stomach python's notion of "significant white-space." Why belly ache about it? Whitespace is significant to one’s fellow humans.
- boredtofears 3y agoPrecisely why it should be of no significance to the machine.
- drdrey 3y agoSource code is not for the machine to read, it’s for your fellow humans
- boredtofears 3y agoThat doesn't mean that source code has no meaning to the interpreter.
- mhh__ 3y agoYou can pretty easily get C performance (I'd argue that C's lack of abstraction makes slow but simple code more appealing) with pythonic expressiveness pretty easily with a more modern language.