11 ms·
Linus: Parallel computing is a huge waste of everybody's time
- 0xFFC 12y agoholy crap , seems science going to wrong direction !
- noobermin 12y agoReading his comment, he seems to be referring to normal users, not things like scientific cluster jobs. Still, I'm kind of upset with his comment.
- gamesbrainiac 12y agoIf you have a saved copy, could you just paste it as a edit to your comment? I'd definitely like to read it.
- vardump 12y agoWhy are you upset with it? Without having seen his comment, so far parallel computing has brought little to consumer end users. Out of that little, by far the most of the benefit is about graphics processing in GPUs.
- r00fus 12y agoHe also excluded ASICs and the like and focused on GP CPUs. Not sure why that'd upset you - what would you use a desktop (assuming x86 or x64) CPU to do that actually requires explicit parallelism?
- rasz_pl 12y agoWe dont need parallelism, we need performance. We have been sitting on 4GHz for the last 4 years now? Things like games require more and more cpu cycles, and the only way to add more is additional cores. Linus post reads like he thinks there is a choice between fast fat cores versus lot of thin ones. There is no choice, there are no 6-8GHz cpus around the corner. We are where we are and there is only one way to "progress".
- szatkus 12y agoI'm not sure what you mean. Intel's Core progressed about 30-50% per clock performance during last 4 years (Nehalem -> Haswell).
- qznc 12y agoI think robots are an interesting use case: Image/speech recognition, motion/task planning, etc. Lots of things to do in parallel in (soft) real time on a mobile device. How to do it is still very unclear and for experimentation you want general purpose processors.
- r00fus 12y agoFPGAs are programmable and faster that GP processors...
- fsloth 12y agoThe way I see it, from a practical point of view, usually the program does not need to execute faster than the user can issues modifications. Usually, the user does only a few tens of operations per minute maximum. There is no need for the processes to execute faster than that. Most stuff people generally do with computers are nowadays blazingly fast in single core. Of course, we can add architectural baggage to slow this to a crawl. Or ignore memory optimization, and waste most of the cycles for cache flushes, reads, writes, allocations, etc. Which do not magically go away once there are more cores... There is a time and place for multicore but that is not everyhwhere like Linus wrote.
- dagw 12y agoEven as someone who does large parallel cluster calculations I kind of agree with him. A CPU with 64 tiny cores and basically no cache will never be as useful to the vast majority of people as a CPU with 4 large cores and lots of cache, even if the 64 cores are in theory able to produce more flops. Also a lot of scientific computation is split up into lots of smaller jobs with each job running on 1-4 cores, rather than a a few jobs running on 64 cores. So even that won't benefit that much from a heavy focus parallelism. And the single jobs that are best parallelized across lots and lots of cores are almost always best handled by something like a GPU and CUDA, and so falls outside the domain of the Linux kernel.
- dagw 12y agoIt's interesting to note that "science" and scientific computing has been back and forth along this road several times already. Back when I started out it was much more common to have large single image machines with 64-512 cores each. Then clusters (image a Beowulf cluster of... and all that) became a thing and all of a sudden people where buying machines 1-4 cores. Then we started getting more and more cores, but even then rarely more than 24-48 cores. Now we're seeing a turn back down again with more and more people buying machines with 4-16 cores and bunch of coprocessor cards.
- 0xFFC 12y agoactually my comment was sarcastic ! I don't know why people down voted my comment
- gamesbrainiac 12y agoI can't seem to get access to the site, "Database Error". Did we just cause the site to crash?
- thaumaturgy 12y agoNaw. Wordpress on a budget VPS caused it to crash.
- raverbashing 12y agoMaybe if his database had some kind of parallelism...
- corford 12y agoOr caching...
- raverbashing 12y agoOr load balancing
- astrodust 12y agoParallel programming is a waste of time, obviously.
- deleted 12y ago[deleted]
- szatkus 12y agoAnd I was wondering in the morning, who broke RWT :)
- zurn 12y agoSite seems to be down, saved copy: http://pastebin.com/Z08gtRFj http://pastebin.com/Z08gtRFj
- VMG 12y agoHeavily editorialized title. Actual quote: > Parallel stupid small cores without caches are horrible unless you have a very specific load that is hugely regular (ie graphics). And > The only place where parallelism matters is in graphics or on the server side, where we already largely have it. Pushing it anywhere else is just pointless. Edit: "editorialized" meaning "choosing the most extreme quote of a rant"
- plinkplonk 12y agoCan a mod edit the title to something less inflammatory?
- MrBuddyCasino 12y agoI intuitively tried to come up with more examples to counter that claim, but all I can think of on the client side is: - 3D Graphics - 2D Graphics (Photoshop, video editing and encoding) - Web Browsers This is where we really use parallel computing. Maybe compilers, file compression and PAR2 on top of that. What else? So theres some truth to that. For the other argument, that a small number of complex OoO cores is better than a huge number cores with slow single-thread performance for client PCs should be obvious, imho.
- illumen 12y agoHow many processes do most users have on their desktops? Hundreds. I have 397 on my OSX laptop, and many of them are multithreaded. Firefox alone is using 61 threads. Dedicated compression/decompression threads, threads for encryption, threads for sound, threads for... well lots of things. Remove the caches and you get more consistent performance. Dedicate tasks to their own core, and they can use that cache for themselves. They don't have to swap it out every time. - Science...
- dfox 12y agoAnother thing is how many of these threads are not blocked waiting for something to happen.
- 12y ago
- moconnor 12y agoThis reads as a fairly direct attack on the idea of that many core architectures such as the Intel Xeon Phi (which does exactly that: replaces 8 powerful Xeon cores with 60+ slower ones per socket) will ever become the norm on the client side. It's an interesting argument but rests upon all new algorithms (he brings up machine vision as an example) having dedicated hardware. Ultimately, sure, but there's still a hell of a gap between viable algorithm and dedicated mobile-ready hardware. If the pace of invention slows I'd agree with Linus. I think the pace of invention will continue to accelerate and parallel processing on the client will be a valuable resource to have.
- taspeotis 12y ago> This reads as a fairly direct attack on the idea of that many core architectures such as the Intel Xeon Phi Whoever's pushing that should read the overview page for Intel Xeon Phi [1]: > While a majority of applications (80 to 90 percent) will continue to achieve maximum performance on Intel Xeon processors, certain highly parallel applications will benefit dramatically by using Intel Xeon Phi coprocessors. To take full advantage of Intel Xeon Phi coprocessors, an application must scale well to over 100 software threads and either make extensive use of vectors or efficiently use more local memory bandwidth than is available on an Intel Xeon processor. Examples of segments with highly parallel applications include: animation, energy, finance, life sciences, manufacturing, medical, public sector, weather, and more. Following that is a picture that says "pick the right tool for the job". [1] http://www.intel.com.au/content/www/au/en/processors/xeon/xeon-phi-coprocessor-overview.html http://www.intel.com.au/content/www/au/en/processors/xeon/xe...
- raverbashing 12y agoSo, it's a rebirth of Larabee? (Essentially a Intel "GPGPU")
- dfox 12y agoYes, it's essentially Larabee without framebuffer.
- 12y ago
- jetm9 12y agoi reallt cant get where his comments are directed. is 6-core or 8-core marketing ARMs? very very high core count niche processors? then he says on the server side we already have it. what is that we have? i dont know optimum number of cores in mobile cpus. when phone is in your pocket and not used very low power core can get phone going even with bg notifications.
- albertzeyer 12y agoI found the last paragraph the most interesting one: "[Parallelism] does not necessarily make sense elsewhere. Even in completely new areas that we don't do today because you cant' afford it. If you want to do low-power ubiquotous computer vision etc, I can pretty much guarantee that you're not going to do it with code on a GP CPU. You're likely not even going to do it on a GPU because even that is too expensive (power wise), but with specialized hardware, probably based on some neural network model." Esp the last bit. I wonder what he means by "specialized hardware, probably based on some neural network model". These ones? http://www.research.ibm.com/cognitive-computing/neurosynaptic-chips.shtml http://www.research.ibm.com/cognitive-computing/neurosynapti... I'm doing Machine Learning / Deep Neural Network research, and of course, parallelism is very important for us. Our chair mostly trains on GPUs at the moment, but the big companies use heavily more parallelism and much bigger sizes, e.g. look at DistBelief from Google.
- jwr 12y agoI read the headline, thought to myself "that can't be right, Linus is too smart for that" — then read the article. And the headline is wrong. So are most of the comments. He explicitly says "Parallel code makes sense in the few cases I mentioned, where we already largely have it covered, because in the server space, people have been parallel for a long time" — so most of that rant is about desktop and mobile computers, often called "client-side". Where indeed four cores seem to be more than enough for most needs. On the "server" side, though (I hate the outdated name), the story is quite different. I can easily make use of many cores, especially with tools like Clojure, that make writing correct concurrent programs much easier.
- pavlov 12y agoWhy do you feel the distinction between client and server is outdated? It seems pretty clear-cut to me... A "client" computer is one that's designed around single-user interaction sessions. It has human interface devices (display, input) on fast interfaces with prioritized OS-level support (interrupts, graphics acceleration, etc). A "server" computer is one where interaction happens over a network connection, and multiple sessions are typically taking place simultaneously.
- ghshephard 12y agoThe difference between client and server can be a bit grey, but I agree, the distinctions are important. A client is about mobility (small size), low power use (low heat, small battery), and very low latency, focus on human-computer interaction experience. A server has fewer power concerns, and is all about throughput. Heat, size, are not as relevant (but not irrelevant). iPhone and iPad are obviously clients. A Mac Pro is a Server. A macbook air is probably a client (though, ironically, it is far more powerful than the $10K servers I had in 1999). A MacBook Pro falls somewhat in the middle, but leans towards client. An iMac is also in the middle, but leans towards server. I think the angst (which I have to some degree as well) about the distinction between Client vs Server, is because it's not completely clear how to position the Mac Pro and iMac. In some benchmarks, the iMac meets, or exceeds the MacPro. But, in terms of sheer throughput for bulk tasks (Video Rendering), you can get a higher top-line performance from the MacPro. And, for day-day interaction, the 5K Display of the iMac (human-computer interaction), beats out the MacPro. [edit: 99.99% of the time, downvotes are immaterial to me, but I'm genuinely intrigued in what the contrary opinion is here - I expect to learn something, please share!]
- noonespecial 12y agoSite seems down. Umm... should have run some parallel instances and loadbalanced?
- BuckRogers 12y ago"Four cores ought to be enough for anybody!"
- rpwverheij 12y agomy thought exactly. Unless we see some big breakthrough in Ghz for single cores, more cores will come, and be properly used, in time.
- willvarfar 12y ago> more cores will come, and be properly used This is usually not possible; see Amdahl's law :(
- marcosdumay 12y agoAmdahl's law is almost useless on practice. If a resource come, people will consume it. If they can't make their code run faster, they'll use it to run more code at the same time. Yet, all that seems to miss Linus point.
- Dewie 12y agoMaybe he means more like "four cores is as good as it gets" (for certain domains). We can always want things to get harder, faster, stronger, but delivering on that is another matter.
- Htsthbjig 12y agoI understand Linux position. We have used parallel computing quite extensively for: 1-3D, 2D vector graphics. 2-Audio processing 3-Image processing 4-All of the above(video). We could parallelize something to be more than 100 times more efficient(ops per Watt) than on the CPU(). But proper parallelization comes at a cost: Efficient memory management is hell. I mean, people are afraid of c manual memory management, that is nothing compared with the complexity of parallel memory management. You need semaphores or mutexes to access common memory, but the most important thing is that you need to make independent as much memory as you can from each other, replace sequential steps, etc... So if the reason for making the kernel parallel is a 10% THEORETICAL increase, forget about it, 10% is nothing with the complexity you have to add. () Power consumption normally increases with the square of frequency, so using more cores instead of higher frequency you get very efficient. The brain itself runs very slow but with a high amount of cores(neuron clusters).
- qznc 12y agoIn the case of software which is widely used a complexity increase matters little. Imagine what companies like Google or Amazon save with a 1% improvement in their data centers. They could employ 10 more kernel developers to deal with the complexity in return.
- BuckRogers 12y agoHmm site is down, must have needed moar coars.
- sz4kerto 12y agoNo, it's not a huge waste. Maybe it is in the short run (<5 years), but there isn't any other way forward (quantum computing is not something we can count on for a while). Yes, with the current architectures and state of compilers it it kind of hopeless, but it doesn't mean we should give up on it.
- rtpg 12y agoWe can scale horizontally (more machines) as well as vertically (faster machines). If reasoning about more machines is easier than imagining faster machines (implicitly here: ones with more cores), then going the other route could be better
- sz4kerto 12y agoThe problem is exactly that we can't scale vertically. Look at single threaded performance charts -- they're already far below Moore's Law. STP/watt is increasing quickyl, but peak performance per core isn't really. (It is, but the second derivative of performance vs time is already negative.)
- ww520 12y agoThe point is the spending of huge effort in turning general processing programs into parallel programs is a waste of time. It's better to write simple single threaded single purpose program to run in a dedicated core. Since we will get more cores, offload single purpose program to a dedicated core is pretty cheap. Parallelism usually occurs in very coarse grain level. Forcing fine grain parallelism at language level is not very productive. The sad state of parallel support in languages and compilers is probably because we are pursuing the wrong goal.
- jokoon 12y agoI think what people seems to not understand about neuron networks is the memory locality. Neurons pass messages directly between each other, so they always retain their own memory state, which is very different from a CPU or a GPU. CPU and GPU have a shared memory architecture, and thus memory access is generally slower overall, because of how they are designed for programming simplicity. Don't ever forget that computer speed is never about calculation, but always about memory access. A computer speed is always limited by its L1 and L2 cache size. If you want to draw one image, GPUs are fine, but they are not really able to simulate neuron networks because they don't have as much parallelism or memory locality a neuron network needs. GPU are more parallized than a CPU, yes, thanks to OpenCL. But they're still specialized towards image blitting, not towards massively parallelism. Neuron networks are not very hard to understand, but if you think about simulating one, you quickly understand that most hardware is just not adequate and unable to run a neuron network properly, it will be just too slow. Computers were never designed and intended to simulate something like a brain. Also don't forget most algorithm we use today are not parallelizable, most are sequential. We could rethink many algorithms and adapt them, but parallelism is often a huge constraint. Sequentiality is a specialized case of computability if you think about it.
- calibwam 12y agoAbsolutes are dumb. Practically, a CPU is throttled on memory access, but it is not hard to imagine scenarios where the computation is the bottle neck. For instance, if most of the program is run in tight loops, where cache misses are few, and the whole loop can fit in L1, almost the whole overhead would be in computation.
- vidarh 12y agoHave you looked at the Parallela / Epiphany from Adapteva? Their current top model is 64 core, but their roadmap is targetting 4K cores or more per chip. They have local (on core) memory, and a routing mesh to let all cores access each others memory - either for communication or for extra storage, and optionally access the host systems RAM. The chips also expose multiple 10Gbps links that can be used to connect the chips themselves into a bigger mesh. You "pay" for accessing remote memory with additional cycles latency based on distance to the node you want to address, so it massively favours focusing on memory locality. I have two 16 core Parallelas sitting around, but have had less time than I'd hoped to play with them.
- Beltiras 12y agoHe simply has to be talking about the kernel, not parallelism in general.
- deleted 12y ago[deleted]
- calibwam 12y agoI don't like predicting the future. > So give up on parallelism already. It's not going to happen. > End users are fine with roughly on the order of four cores, > and you can't fit any more anyway without using too much > energy to be practical in that space. End users was fine with a single core Pentium 4 on their workstation. We progressed. How would even Linus know that we won't find a way to make parallel work en masse?
- dagw 12y agoEnd users was fine with a single core Pentium 4 on their workstation. Not really. 2-4 cores have been available on workstations for decades, so no one is arguing the 1 core is all you need. Even Linus is saying that having 4 cores is probably a good thing in many cases. The argument is not 1 vs 4, but more 4 vs 64, especially if you assume a fixed power budget.
- calibwam 12y agoI'm not saying they would be fine now, but once (10 years ago), that was what you had. And now we have 4-8 cores on our laptops, and 2-4 on our phones. Why shouldn't we have 64 cores in the future if we solve the programming problems, and can make them energy efficient? Just because 4 cores are fine now, doesn't mean we shouldn't try to increase that.
- dagw 12y agobut once (10 years ago), that was what you had. No it wasn't. Multi-processor Intel based workstation have been available since the very early 90s. People have realized for a very long time that having 2-4 cores is useful. I'm still not convinced that, given that I have X Watts to spend, that I'm not better off with 4 CPUs using X/4 Watts each rather than 64 cores using X/64 Watts each. But I'm willing to be proven wrong.
- calibwam 12y agoReally? Do you mean people chained together multiple processors, or that Intel produced something? For I can't find it and would be interested in reading about those old systems.
- vidarh 12y agoConsider that we have parallel stupid small cores, and have had them for years: Most harddrives have full CPU's on the controllers these days, and there are full CPU cores all over the place outside of our "view". E.g. consider things like the Transcend Wifi SD cards with ARM SoC's on board. Many hard drive controllers have full CPUs. You find embedded CPUs "hidden" in all kinds of PC hardware these days. The PC architecture started out being hampered by the CPU + dumb peripherals architecture, but this was a big departure from the norm during that era. In the "home" market, most machines either had CPUs too weak to drive the peripherals (my first "parallel stupid small cores" system was a Commodore 64 + a 1541 disk drive - you could, and people did, and wrote book about, download your own code to the 1541 and do computations on it) or explicitly used CPUs or co-processors all over the place to offload things, like the Amiga. My A2000 had of course the 68000 (and later a 68020 accelerator), but also had a 6502 compatible core running the keyboard, a Z80 on the SCSI controller card, on top of the Amiga's custom chips which had the copper, blitter and sound chips that all had limited programmability. The irony is that it perhaps made a lot of us overly gung ho on the 680x0 line (though it does have a beautiful instruction set) because our machines felt so much faster than comparably clocked PCs - hence many of use were ok with not upgrading CPU models and clock rates as fast as in the PC world. It was first when PC's started sprouting co-processors (graphics cards and sound cards with advanced capabilities first), that the Amiga truly lost its edge (at the same time Motorola failed to deliver fast enough versions of their newest 680x0 models, faster CPU's would have been insufficient) - until then the co-processors and philosophy offloading everything possible had compensated for the by then anaemic average CPU speeds. Though the 3rd party expansions gave one more fascinating multi-processing step: Systems that would let you run 680x0 and PPC code on the same machine (PPC cpu on the expansion card, 680x0 on the motherboard). PCs have been steadily sprouting more small cores. They're just not as visible. On the server side it is extreme: I have single CPU servers at work where the main CPU may have 6-8 x86 cores, but where there may be 30+ ARM or MIPS cores when you tally up the harddrives (dual or tri-core in many cases), RAID controllers, IPMI cards, some networking hardware etc.. But these cores are getting so cheap and so small that we should expect to continue to see offloading of more functionality. On the Amiga, this was what was our day to day reality. SCSI was always favoured, for example, because it was lighter on the CPU than IDE was (hence universal disdain for the guy that forced engineering to put IDE in the last Amiga models - the A4000 and 1200 - which helped cripple them at a time when they were lagging in overall CPU performance; the man in charge at Commodore the time was the guy that had been responsible for the PCjr disaster..), because it offloaded more logic (and hence was more expensive, hence the cut). Rather than be exposed to the parallelism, expect that we'll see more higher level functionality being subsumed into peripherals so that the main CPU gets to focus on running your code. E.g. there are TCP offload functionality on some high end network cards. Just like in the architectures born in the 60's and 70's out of necessity because of wimpy CPUs... The "one CPU to rule them all" PC is an aberration, born out of a time when CPUs where getting to a performance level where it was possible, and where the norm of single-tasking OSs for personal computers meant it for a short few years seemingly made little difference if the main CPU spent all its time serving disk interrupts while loading stuff. The norm in computing have been multiple parallel cores. Even multiple parallel CPUs. Often on multiple buses. And over the last few years we've gone full circle.
- qwerta 12y agoI make 'parallel' stuff for living and I sort of agree. Often optimizing your code is better choice than going parallel. Compact code can fit into single CPU cache which brings huge performance boost. On other side parallel programming does not have to be hard. Fork-Join framework in Java and parallel collections in Java 8, are trivial to use and scale vertically pretty well. And finally I think there is not actual demand for parallel programming. 99% computers have 4 cpus or less. GPUs are useless for most tasks. I have prototype of my program which scaled well to 20 cores, but nobody is interested.
- edem 12y agoIt says "Error establishing a database connection".
- touristtam 12y agosame here.
- kbart 12y agoTotally agree with Linus. These >10 cores smarphones etc. look just like marketing trick, not that user would feel any meaningful difference. Same goes for PCs -- usually you won't feel any difference between running, let's say, 4 and 8 core CPU (of the same architecture), except synthetic benchmark tests that have not much to do with actual performance. Of course, there are some corner cases where it makes sense (scientific calculations, heavy graphics, simulations, compiling etc.), but a common user does not benefit much.
- sklogic 12y agoCompiling is not such a corner case. Even non-developers, "common users" are often enjoying the source-based Linux distributions, things like Mac Ports and alike.
- kbart 12y agoEven so, I doubt it's an every day task for them. Also, most popular open source projects provide ready binaries of the stable versions for such users. Still, I hardly imagine my grandma compiling some open source projects for her use..
- sklogic 12y agoAlmost every time I install or update something from mac ports, it's ending up compiling, binaries are very rarely cached. I doubt my setup is unusual (the latest OS X and the latest XCode). And I suspect that, for example, JS engines in the browsers are going to be more and more parallel. And that kind of compilation is something that every user does on a daily basis. More cores => smoother web browsing experience. Not that I approve this whole thing that's going on with the JS craze, but it's the fact everyone have to cope with, unfortunately.
- deleted 12y ago[deleted]
- dagw 12y ago
- koliber 12y agoI cannot access the article because of an error "Error establishing a database connection". Perhaps if there were parallel computing resources set up to handle increased load, I could read Linus's argument against parallel computing.
- eleitl 12y agoOf course Linux wouldn't run on a parallel system, so "four cores ought to be enough for everybody" Linus is correct, as far as Linux goes.
- im2w1l 12y ago>Where the hell do you envision that those magical parallel algorithms would be used? Parallel sort. Parallel n-nary search. Parallel linear search. A stupid core for the zero-page thread. Hardware blas. Time stepping multi object systems.
- eccstartup 12y agoWhat Linus means is that "You guys are so stupid to write parallel code." Learn physics first!