3 ms·

I'm impressed - this is the first system I've read about from a startup that actually sounds as if it is architected to be fast. Targeting I/O and memory bottle

I'm impressed - this is the first system I've read about from a startup that actually sounds as if it is architected to be fast. Targeting I/O and memory bottlenecks as the only acceptable limits is exactly what you should do. Nice to see HPC learning being used outside the field. Many people horrifically underestimate or simply overlook the cost of fetching data from outside your NUMA node.

jandrewrogers 11y ago
I spent a few years designing new parallel algorithms on a number of supercomputing architectures that I enjoyed immensely, mixed in with my usual database work. I learned a tremendous amount about software design and optimization I probably would have not learned otherwise. There is a lot of knowledge in HPC that tends not to travel beyond that domain. It gave me the "assume you have a million cores..." mindset when thinking about algorithms and data structures. In fact, debugging an HPC code on a Cray machine late one night, staring at large dumps of binary coded hyper-rectangles, was where I first noticed the deep bit-wise patterns in some types of spatial representations that SpaceCurve now exploits. It allows computation on hyper-rectangle primitives to be nearly as fast as integer primitives on ordinary silicon, which was not the case with earlier versions of my work.
- nickpsecurity 11y ago
  That's a trip. I agree with you about HPC having much wisdom that isn't exploited outside of it. Lots of what I heard from "cloud," etc had already been done in MPP's, etc. They still haven't caught up to the field totally and I still plan on resurrecting MPP eventually. A prior discussion here suggests what holds it back is cultures. The HPC people keep to themselves and don't try to make anything useful outside the field. The cloud projects act like they're the only thing going on. And so on. Bridging the gap may never happen but I always encourage upstarts to dig into the HPC literature on their topics as they might find gold. One of few cases an ACM/IEEE fee is worthwhile. ;) Personal example was trying to find a patent-resistant, NUMA architecture to use for security-enhanced SPARC CPU's. Stumbled upon the MIT Alewife machine that did NUMA up to 512 nodes using (a) SPARC processor modified for quick context-switching & message passing; (b) a single, custom chip to handle NUMA aspects. Making cheap, COTS hardware NUMA w/ open design would take one ASIC development using that architecture. Just cuz I studied supercomputing once and remembered to look it up. :)
- corysama 11y ago
  When the article says "The only primitive types are hyper-rectangles of infinite volume" what I imagine is the intersection volume of 1 axis-aligned splitting plane per dimension. Is that close enough or off-base?

jandrewrogers 11y ago

I spent a few years designing new parallel algorithms on a number of supercomputing architectures that I enjoyed immensely, mixed in with my usual database work. I learned a tremendous amount about software design and optimization I probably would have not learned otherwise. There is a lot of knowledge in HPC that tends not to travel beyond that domain. It gave me the "assume you have a million cores..." mindset when thinking about algorithms and data structures. In fact, debugging an HPC code on a Cray machine late one night, staring at large dumps of binary coded hyper-rectangles, was where I first noticed the deep bit-wise patterns in some types of spatial representations that SpaceCurve now exploits. It allows computation on hyper-rectangle primitives to be nearly as fast as integer primitives on ordinary silicon, which was not the case with earlier versions of my work.

nickpsecurity 11y ago
That's a trip. I agree with you about HPC having much wisdom that isn't exploited outside of it. Lots of what I heard from "cloud," etc had already been done in MPP's, etc. They still haven't caught up to the field totally and I still plan on resurrecting MPP eventually. A prior discussion here suggests what holds it back is cultures. The HPC people keep to themselves and don't try to make anything useful outside the field. The cloud projects act like they're the only thing going on. And so on. Bridging the gap may never happen but I always encourage upstarts to dig into the HPC literature on their topics as they might find gold. One of few cases an ACM/IEEE fee is worthwhile. ;) Personal example was trying to find a patent-resistant, NUMA architecture to use for security-enhanced SPARC CPU's. Stumbled upon the MIT Alewife machine that did NUMA up to 512 nodes using (a) SPARC processor modified for quick context-switching & message passing; (b) a single, custom chip to handle NUMA aspects. Making cheap, COTS hardware NUMA w/ open design would take one ASIC development using that architecture. Just cuz I studied supercomputing once and remembered to look it up. :)
corysama 11y ago
When the article says "The only primitive types are hyper-rectangles of infinite volume" what I imagine is the intersection volume of 1 axis-aligned splitting plane per dimension. Is that close enough or off-base?