3 ms·
"Parallelism: yes, although OpenMP isn't nearly as accessible as e.g. Swift async closures." I'm not familiar with them, do you have a pointer? Cilk does have
by gpderetta 11y ago
"Parallelism: yes, although OpenMP isn't nearly as accessible as e.g. Swift async closures."
I'm not familiar with them, do you have a pointer? Cilk does have powerful semantics and a very light weight syntax.
"C++ only got proper language-native threading in C++11."
Sure, but OpenMP and Cilk are significantly older.
"Do google view their datacenters as a single flat cache-coherent memory space?"
No, but I'm pretty sure they whish they could. Many HPC clusters do present a single memory image across thousands of machines.
"they built mapreduce instead."
Mapreduce (and its extensions) is not a general programming model though.
- nickpsecurity 11y agoExactly. We had older stuff that could handle this. There was a ton of innovation in MPP's and clusters that could be applied to today's problems with ASIC's or FPGA's with interesting results. I intend to do just that at some point. Far as nanosecond comms, they might want to consider using the old Active Messages or Fast Messages schemes. Their latency was tiny even on Ethernet. SGI also put FPGA's on NUMA interconnect back in the day. What Intel is doing is an increment on that rather than revolutionary or anything. One could use NUMAscale's chips to connect these things together. There's also academic tools that could be polished for producing Verilog/VHDL automatically from higher-level descriptions. High-level synthesis it's called. Works well enough for simple constructs like he says he uses. Don't have to go straight to RTL level haha.
- neutronicus 11y agoAll the HPC I've done is explicitly message-passing. There's certainly no abstraction layer that allows, me to treat it is a single memory space.
- gpderetta 11y agoDistributed Shared Memory [1] is a thing, although I guess that today's ultra large clusters make it impractical. [1] https://en.wikipedia.org/wiki/Distributed_shared_memory https://en.wikipedia.org/wiki/Distributed_shared_memory
- vmarsy 11y ago> I'm pretty sure they whish they could [view their datacenters as a single flat cache-coherent memory space]. Many HPC clusters do present a single memory image across thousands of machines. No not really, at some point when you're dealing with PetaBytes of RAM and millions of cores, the Law of Physics kicks in, your RAM is spreading across a large physical area no matter how clever you are. If you want a flat memory space you have to guarantee an access to any memory address in less than X cycles otherwise you have a NUMA architecture[1] While this is true that HPC clusters present a single memory image per cluster node (where one node = 8-32 processors (maybe 64)), the other nodes's memory has to be access with Message passing or other mechanisms. You need a different programming model, MapReduce is too specific, that's why Google is trying things like their "DataFlow" platform. [1]https://en.wikipedia.org/wiki/Non-uniform_memory_access#NUMA_vs._cluster_computing https://en.wikipedia.org/wiki/Non-uniform_memory_access#NUMA...
- gpderetta 11y ago"If you want a flat memory space you have to guarantee an access to any memory address in less than X cycles otherwise you have a NUMA architecture[1]" There is nothing wrong with NUMA (well, ccNUMA, but today that's a given). Even a simple modern two socket server is a NUMA machine. Anyways, as I've commented elsewhere, I'm not arguing that shared memory is practical today on a large HPC cluser.
- lomnakkus 11y ago> Anyways, as I've commented elsewhere, I'm not arguing that shared memory is practical today on a large HPC cluser. I think the point that was being made was that it'll never be practical purely for physical reasons. Any physical separation means that light takes a certain amount of time to travel and no known law of physics will let you circumvent that... A distance of a foot will always incur a latency of ~1ns (at best), so our models must account for latency. (At some point -- it's not obvious that we've reached the end of how compact a computer can be, but there is a limit where you just end up with a tiny black hole instead of computer.)
- 11y ago