3 ms·

"circumvent these limits in highly parallel systems with precise I/O and execution scheduling, and relying much less on caching, but those people don’t work on

by scrubs 3y ago

"circumvent these limits in highly parallel systems with precise I/O and execution scheduling, and relying much less on caching, but those people don’t work on databases." Can I bug you for a library, book, algo, gpu i/o card or whatever good stuff you may be referring to there?

jandrewrogers 3y ago
Tragically, supercomputing is mostly dead as a hardcore research area AFAICT, and they rarely published in no small part because so much of the research was funded by US intelligence agencies. It was pure serendipity that I ended up in that world; I solved a theoretical CS problem that was particularly interesting to people working in that space and was given free supercomputing time to do further research (which was awesome). I learned a lot about how to think about software at extreme scales from the graybeards in HPC. I was one of the few young people in the space even back then. To answer your question, almost all of the valuable theory comes from research in silicon architectures designed to hide latency, with graph analysis as the canonical problem case. It is the question of how to get the most throughput out of silicon that is frequently waiting for a cache line to fill or a lock to release. In silicon, this typically refers to barrel processors specifically and the most refined form of that was arguably the Cray XMT. In modern silicon, you don’t have to squint too hard to see that GPUs are a specialized type of barrel processor. The less obvious aspect is that you can do very effective latency-hiding in software, but how is poorly documented. Nonetheless, if you study how latency-hiding silicon works, it isn’t that hard to figure out how to design a software architecture that has similar properties albeit with fewer guarantees than silicon offers. Latency hiding silicon works by having a huge number of operations in flight at any one time, surveying the landscape, and choosing the operation that is mostly likely to make forward progress. Software has the advantage that it can look much further into the future of operations than silicon can and in many ways is actually more efficient even though you have less control of the fine details.

by scrubs 3y ago

jandrewrogers 3y ago

Tragically, supercomputing is mostly dead as a hardcore research area AFAICT, and they rarely published in no small part because so much of the research was funded by US intelligence agencies. It was pure serendipity that I ended up in that world; I solved a theoretical CS problem that was particularly interesting to people working in that space and was given free supercomputing time to do further research (which was awesome). I learned a lot about how to think about software at extreme scales from the graybeards in HPC. I was one of the few young people in the space even back then. To answer your question, almost all of the valuable theory comes from research in silicon architectures designed to hide latency, with graph analysis as the canonical problem case. It is the question of how to get the most throughput out of silicon that is frequently waiting for a cache line to fill or a lock to release. In silicon, this typically refers to barrel processors specifically and the most refined form of that was arguably the Cray XMT. In modern silicon, you don’t have to squint too hard to see that GPUs are a specialized type of barrel processor. The less obvious aspect is that you can do very effective latency-hiding in software, but how is poorly documented. Nonetheless, if you study how latency-hiding silicon works, it isn’t that hard to figure out how to design a software architecture that has similar properties albeit with fewer guarantees than silicon offers. Latency hiding silicon works by having a huge number of operations in flight at any one time, surveying the landscape, and choosing the operation that is mostly likely to make forward progress. Software has the advantage that it can look much further into the future of operations than silicon can and in many ways is actually more efficient even though you have less control of the fine details.