2 ms·

Was skimming at your Space Curve writeup [1] and your mention of discreet internal components caught my eye. Are you open to expanding a bit on this statement:

Was skimming at your Space Curve writeup [1] and your mention of discreet internal components caught my eye. Are you open to expanding a bit on this statement: "Discrete topology internals largely obviate secondary indexing" [1]: https://www.jandrewrogers.com/2015/10/08/spacecurve/ https://www.jandrewrogers.com/2015/10/08/spacecurve/

eternalban 7y ago
Ah, my coin dropped here. Most likely you mean the content addressable segmented architecture.
jandrewrogers 7y ago
Secondary indexing is a hack to address the reality that most indexing algorithms can only represent a single type of relationship efficiently and typically only in a single dimension. This is not a law of the universe, it is just how our algorithms tend to work. If you could eliminate secondary indexing without sacrificing selectivity, it would be a massive win for performance and scalability. However, this would require a single indexing algorithm for complex data models that preserved an arbitrary mix of relational, time-series, spatial, graph, etc relationships for searches and joins. To make this work in a practical database engine, you can't index the data model per se but you can make it work by indexing a moduli space into which arbitrary data models can be mapped. These tend to naturally expose the topological structure of the underlying data model for computational purposes even though you are not computing on the data model per se. Designing very general moduli spaces for database purposes is non-trivial and, to make matters worse, they are pathologically incompatible with typical surrounding database infrastructure once you figure out how to construct them. But you can use the exposed topology to execute complex searches and joins on the underlying data model. None of my database engines use secondary indexing at all, hence the excellent scaling and write performance, even for complex mixed-mode data models. A decade ago the representations were pretty brittle and limited because I didn't know how to express many things, but these days I know how to elegantly express just about every common data model.
- eternalban 7y ago
  Thanks! > a moduli space into which arbitrary data models can be mapped Very interesting. Somehow reminds me of using latices for deterministic concurrency. Is this a topic that is discussed in public literature or an innovation of yours? Love to learn more about this.
  - jandrewrogers 7y ago
    The work is mostly mine but I've had collaborators for some of the research over the years. I accidentally invented it many years ago when I discovered an unorthodox but elegant solution to an algorithm problem which had stymied researchers for decades (and which I needed to solve for a specific application). Some months later, a research group I was working with were convinced that the unusual algorithm construction might be applicable to an unrelated algorithm problem they had been working on. Six weeks later I had solved their problem too by extending the concepts developed for my original algorithm. By that point I realized that while those two very different algorithms were cool, the half-baked computer science I had developed to construct them was even cooler. I spent the next decade fleshing out, generalizing, and extending the scope of the computer science while figuring out how to efficiently express that theory in practical software (which is not trivial). While I wrote quite a few papers on this computer science many years ago when it was new, the distribution was required to be non-public. I sporadically teach bits of it but writing up hundreds of pages of old research in my spare time is a lot less fun than working on my backlog of interesting computer science projects.

eternalban 7y ago

Ah, my coin dropped here. Most likely you mean the content addressable segmented architecture.

jandrewrogers 7y ago

Secondary indexing is a hack to address the reality that most indexing algorithms can only represent a single type of relationship efficiently and typically only in a single dimension. This is not a law of the universe, it is just how our algorithms tend to work. If you could eliminate secondary indexing without sacrificing selectivity, it would be a massive win for performance and scalability. However, this would require a single indexing algorithm for complex data models that preserved an arbitrary mix of relational, time-series, spatial, graph, etc relationships for searches and joins. To make this work in a practical database engine, you can't index the data model per se but you can make it work by indexing a moduli space into which arbitrary data models can be mapped. These tend to naturally expose the topological structure of the underlying data model for computational purposes even though you are not computing on the data model per se. Designing very general moduli spaces for database purposes is non-trivial and, to make matters worse, they are pathologically incompatible with typical surrounding database infrastructure once you figure out how to construct them. But you can use the exposed topology to execute complex searches and joins on the underlying data model. None of my database engines use secondary indexing at all, hence the excellent scaling and write performance, even for complex mixed-mode data models. A decade ago the representations were pretty brittle and limited because I didn't know how to express many things, but these days I know how to elegantly express just about every common data model.

eternalban 7y ago
Thanks! > a moduli space into which arbitrary data models can be mapped Very interesting. Somehow reminds me of using latices for deterministic concurrency. Is this a topic that is discussed in public literature or an innovation of yours? Love to learn more about this.
- jandrewrogers 7y ago
  The work is mostly mine but I've had collaborators for some of the research over the years. I accidentally invented it many years ago when I discovered an unorthodox but elegant solution to an algorithm problem which had stymied researchers for decades (and which I needed to solve for a specific application). Some months later, a research group I was working with were convinced that the unusual algorithm construction might be applicable to an unrelated algorithm problem they had been working on. Six weeks later I had solved their problem too by extending the concepts developed for my original algorithm. By that point I realized that while those two very different algorithms were cool, the half-baked computer science I had developed to construct them was even cooler. I spent the next decade fleshing out, generalizing, and extending the scope of the computer science while figuring out how to efficiently express that theory in practical software (which is not trivial). While I wrote quite a few papers on this computer science many years ago when it was new, the distribution was required to be non-public. I sporadically teach bits of it but writing up hundreds of pages of old research in my spare time is a lot less fun than working on my backlog of interesting computer science projects.