3 ms·

Why is it that they aren't published? Competitive advantage?

• jandrewrogers 14y ago
  Yes. The R&D was almost entirely funded by multiple private research organizations over several years at relatively high cost (and not always the ones you would think). As a practical matter, many algorithm patents have not been enforceable for years even though they are well-supported in international law. Consequently, almost everything related to massively parallel graph technologies is closely held as a trade secret. Academia is pretty far behind the state-of-the-art in this area. I've done work in other algorithm areas where the situation is similar. It often costs millions of dollars to develop major computer science advances but the best way to recover that investment is to leverage it without publishing it. Reverse engineering a thoroughly obfuscated technology takes a lot longer than when someone publishes the blueprints. I will add that a lot of these algorithms are not at all obvious until you understand how they work. While knowing something is possible helps, it still requires a fair amount of theoretical computer science cleverness.
  • jasonwatkinspdx 14y ago
    I understand you may be under NDA, as well as the general pressure to not reveal knowledge that is the basis of your career, but would you care to point out any open literature you think is worthwhile to study?
    • jandrewrogers 14y ago
      For massively distributed and parallel algorithms, a core part of the computer science is based around space decomposition data structures. If your algorithm is not based on a space decomposition structure, it won't linearly parallelize across a vast number of cores. When I say some algorithm won't distribute or parallelize, this is what I am usually (but not always) looking at. Space decomposition structures take a metric space and decompose it using an invariant space partitioning function. The partition points are fixed regardless of data distribution, the only decision process is when to apply the partitioning function. The two well-known examples in literature are quad-trees and distributed hash tables. These are both very simple, narrow examples of space decomposition structures in a universe of possible metric spaces and possible partitioning functions. Many parallel data problems can't be made to fit into the simple space decomposition structures computer scientists are familiar with. When massively parallel machinery is the foundational assumption, every computational structure must be represented in terms of space decomposition primitives. Right now, most computer scientists treat space decomposition structures as dumb data buckets and are largely oblivious to their expressiveness. Most published parallel algorithm computer science is based on computational structures that will only sort of parallelize on sufficiently small systems.
      • curiouskat 14y ago
        Are you referring to space-filling curves?
        • jandrewrogers 14y ago
          Not really, though there is a natural relationship between partitioning functions and space-filling curves.
      • jasonwatkinspdx 14y ago
        > "Right now, most computer scientists treat space decomposition structures as dumb data buckets and are largely oblivious to their expressiveness." This is very interesting. Is there an example of the expressiveness of spatial decomposition you could share?