4 ms·
Can't really argue about 1. About 2, in-order single instruction execution hasn't been an assumption for a very long time; c and c++ optimizers (and programmer
by gpderetta 11y ago
Can't really argue about 1.
About 2, in-order single instruction execution hasn't been an assumption for a very long time; c and c++ optimizers (and programmers) have been able to take advantage of these CPU features for a while. There are language extensions (Cilk++, OpenMP), to take advantage of extra cores for fine grained parallelism.
Regarding GPUs, arguably C and C++ have the most mature and transparent offloading support all around (OpenACC, again OpenMP, whatever MS offloading extensions are called) and the most popular GPU programming language (CUDA) is a C++ dialect.
Regarding the flat memory model, for large scale programming the only sane model is a flat, cache coherent one; those architectures that don't provide that, either evolve to provide it or die (cf. CELL) supplanted by those that do (yes, that doesn't mean that all memory is the same, but that is true with your standard CPU anyway).
I don't have an opinion on FPGAs, I expect that, if they ever go mainstream, initially people will just assemble predefined blocks via high level languages, but who knows what the future reserves us.
- pjc50 11y agoParallelism: yes, although OpenMP isn't nearly as accessible as e.g. Swift async closures. C++ only got proper language-native threading in C++11. for large scale programming the only sane model is a flat, cache coherent one Do google view their datacenters as a single flat cache-coherent memory space? No, they built mapreduce instead. That's the point of view I'm coming from: distributed systems engineering working downwards. Rather than a single large program operating on a single memory space, a set of fragments whose programmers are aware that there is latency when communicating between nodes. DRAM is just another "node" that you have to send messages to and wait for a response.
- gpderetta 11y ago"Parallelism: yes, although OpenMP isn't nearly as accessible as e.g. Swift async closures." I'm not familiar with them, do you have a pointer? Cilk does have powerful semantics and a very light weight syntax. "C++ only got proper language-native threading in C++11." Sure, but OpenMP and Cilk are significantly older. "Do google view their datacenters as a single flat cache-coherent memory space?" No, but I'm pretty sure they whish they could. Many HPC clusters do present a single memory image across thousands of machines. "they built mapreduce instead." Mapreduce (and its extensions) is not a general programming model though.
- nickpsecurity 11y agoExactly. We had older stuff that could handle this. There was a ton of innovation in MPP's and clusters that could be applied to today's problems with ASIC's or FPGA's with interesting results. I intend to do just that at some point. Far as nanosecond comms, they might want to consider using the old Active Messages or Fast Messages schemes. Their latency was tiny even on Ethernet. SGI also put FPGA's on NUMA interconnect back in the day. What Intel is doing is an increment on that rather than revolutionary or anything. One could use NUMAscale's chips to connect these things together. There's also academic tools that could be polished for producing Verilog/VHDL automatically from higher-level descriptions. High-level synthesis it's called. Works well enough for simple constructs like he says he uses. Don't have to go straight to RTL level haha.
- neutronicus 11y agoAll the HPC I've done is explicitly message-passing. There's certainly no abstraction layer that allows, me to treat it is a single memory space.
- gpderetta 11y agoDistributed Shared Memory [1] is a thing, although I guess that today's ultra large clusters make it impractical. [1] https://en.wikipedia.org/wiki/Distributed_shared_memory https://en.wikipedia.org/wiki/Distributed_shared_memory
- vmarsy 11y ago> I'm pretty sure they whish they could [view their datacenters as a single flat cache-coherent memory space]. Many HPC clusters do present a single memory image across thousands of machines. No not really, at some point when you're dealing with PetaBytes of RAM and millions of cores, the Law of Physics kicks in, your RAM is spreading across a large physical area no matter how clever you are. If you want a flat memory space you have to guarantee an access to any memory address in less than X cycles otherwise you have a NUMA architecture[1] While this is true that HPC clusters present a single memory image per cluster node (where one node = 8-32 processors (maybe 64)), the other nodes's memory has to be access with Message passing or other mechanisms. You need a different programming model, MapReduce is too specific, that's why Google is trying things like their "DataFlow" platform. [1]https://en.wikipedia.org/wiki/Non-uniform_memory_access#NUMA_vs._cluster_computing https://en.wikipedia.org/wiki/Non-uniform_memory_access#NUMA...
- maxlybbert 11y agoWith OpenMP your parallel loops look like normal loops (they have a #pragma to mark them as parallelizable). With asynchronous closures, your parallel code looks like callbacks. Obviously, which you prefer is personal preference, but I've never heard anybody say OpenMP wasn't accessible.
- lallysingh 11y agoComputers don't really have the flat memory model today. They're networked cores with coherent distributed memory wrapped in a simple API that has very high performance variability. Any kind of performant multithreaded work today has to treat their motherboard as such unless they want significant performance hits.
- lightcatcher 11y ago> Regarding the flat memory model, for large scale programming the only sane model is a flat, cache coherent one; those architectures that don't provide that, either evolve to provide it or die (cf. CELL) supplanted by those that do (yes, that doesn't mean that all memory is the same, but that is true with your standard CPU anyway). CUDA on Nvidia GPUs gives a non-coherent last level cache. Most programs distributed over multiple nodes have no shared address space abstraction, but instead do explicit message passing. I agree that coherent caches make programming easier to think about, but I don't think I would go as far to say they are the "only sane model" for "large scale programming" given that MPI and CUDA both are popular.
- gpderetta 11y agoSure, my point is that the general trend is for the hardware to get smarter and hide the ugliness of the system to the programmer. Every time it has been attempted to push the complexity to the programming language and compiler, it has ended in tragedy.