4 ms·

Can you give names of closed-source database engines that have these kinds of performance improvements? I mean, the mainstream RDBMS like Oracle, DB2 etc don't

Can you give names of closed-source database engines that have these kinds of performance improvements? I mean, the mainstream RDBMS like Oracle, DB2 etc don't seem to be ahead of the open source databases; they are all stagnant too!

einpoklum 7y ago
Actian Vector, formerly Vectorwise: https://docs.actian.com/vector/5.0/index.html https://docs.actian.com/vector/5.0/index.html From the "vectorized columnar execution, operator at a time" school. HyperDB: https://hyper-db.de/ https://hyper-db.de/ That's from the "row-oriented execution, query plan JIT compilation" school.
- dgacmu 7y ago
  The hyper-db folks do great work. Indeed, Huanchen collaborated with Viktor Leis (one of the senior folks on HyPer) on the SuRF "Succinct Range Filter" (think Bloom filter but for range queries) work: http://www.cs.cmu.edu/~huanche1/publications/surf_paper.pdf http://www.cs.cmu.edu/~huanche1/publications/surf_paper.pdf and github: https://github.com/efficient/SuRF https://github.com/efficient/SuRF and both Viktor and Thomas Neumann's work in this area is fantastic and everyone should go read it. :) The code for several of Huanchen's projects is available in our group's github repo: https://github.com/efficient/ https://github.com/efficient/ The order-preserving compression work outlined in his thesis will be appearing soon at SIGMOD, but there's no need to wait for it, since it's roughly the same in both. We'll get the source to that released by the time of SIGMOD.
jandrewrogers 7y ago
Databases are severely constrained by the architecture choices from when they were designed, you can't back port modern database architecture and computer science to e.g. an Oracle or DB2. To integrate new computer science advances you often need to write a new kernel from first principles. I sunset the designs I license to companies every several years, starting over from a clean sheet. Most new high-performance database engines are intended to give the developing company a new qualitative uplift in capability, scale, or operational efficiency. No one sells public licenses these days. You've heard of the organizations that are buying building these semi-bespoke database engines but they are intended for internal use only. The reason no one sells these capabilities as a product anymore is pragmatic: it is extremely expensive to design a database engine for general public consumption and the economics are difficult to justify as an investment. But many large companies are willing to pay many millions of dollars for a narrowly focused database capabilities, and the reduced scope makes the development cost more palatable.
- eternalban 7y ago
  Was skimming at your Space Curve writeup [1] and your mention of discreet internal components caught my eye. Are you open to expanding a bit on this statement: "Discrete topology internals largely obviate secondary indexing" [1]: https://www.jandrewrogers.com/2015/10/08/spacecurve/ https://www.jandrewrogers.com/2015/10/08/spacecurve/
  - eternalban 7y ago
    Ah, my coin dropped here. Most likely you mean the content addressable segmented architecture.
  - jandrewrogers 7y ago
    Secondary indexing is a hack to address the reality that most indexing algorithms can only represent a single type of relationship efficiently and typically only in a single dimension. This is not a law of the universe, it is just how our algorithms tend to work. If you could eliminate secondary indexing without sacrificing selectivity, it would be a massive win for performance and scalability. However, this would require a single indexing algorithm for complex data models that preserved an arbitrary mix of relational, time-series, spatial, graph, etc relationships for searches and joins. To make this work in a practical database engine, you can't index the data model per se but you can make it work by indexing a moduli space into which arbitrary data models can be mapped. These tend to naturally expose the topological structure of the underlying data model for computational purposes even though you are not computing on the data model per se. Designing very general moduli spaces for database purposes is non-trivial and, to make matters worse, they are pathologically incompatible with typical surrounding database infrastructure once you figure out how to construct them. But you can use the exposed topology to execute complex searches and joins on the underlying data model. None of my database engines use secondary indexing at all, hence the excellent scaling and write performance, even for complex mixed-mode data models. A decade ago the representations were pretty brittle and limited because I didn't know how to express many things, but these days I know how to elegantly express just about every common data model.
    - 7y ago

einpoklum 7y ago

Actian Vector, formerly Vectorwise: https://docs.actian.com/vector/5.0/index.html https://docs.actian.com/vector/5.0/index.html From the "vectorized columnar execution, operator at a time" school. HyperDB: https://hyper-db.de/ https://hyper-db.de/ That's from the "row-oriented execution, query plan JIT compilation" school.

dgacmu 7y ago
The hyper-db folks do great work. Indeed, Huanchen collaborated with Viktor Leis (one of the senior folks on HyPer) on the SuRF "Succinct Range Filter" (think Bloom filter but for range queries) work: http://www.cs.cmu.edu/~huanche1/publications/surf_paper.pdf http://www.cs.cmu.edu/~huanche1/publications/surf_paper.pdf and github: https://github.com/efficient/SuRF https://github.com/efficient/SuRF and both Viktor and Thomas Neumann's work in this area is fantastic and everyone should go read it. :) The code for several of Huanchen's projects is available in our group's github repo: https://github.com/efficient/ https://github.com/efficient/ The order-preserving compression work outlined in his thesis will be appearing soon at SIGMOD, but there's no need to wait for it, since it's roughly the same in both. We'll get the source to that released by the time of SIGMOD.

jandrewrogers 7y ago

Databases are severely constrained by the architecture choices from when they were designed, you can't back port modern database architecture and computer science to e.g. an Oracle or DB2. To integrate new computer science advances you often need to write a new kernel from first principles. I sunset the designs I license to companies every several years, starting over from a clean sheet. Most new high-performance database engines are intended to give the developing company a new qualitative uplift in capability, scale, or operational efficiency. No one sells public licenses these days. You've heard of the organizations that are buying building these semi-bespoke database engines but they are intended for internal use only. The reason no one sells these capabilities as a product anymore is pragmatic: it is extremely expensive to design a database engine for general public consumption and the economics are difficult to justify as an investment. But many large companies are willing to pay many millions of dollars for a narrowly focused database capabilities, and the reduced scope makes the development cost more palatable.

eternalban 7y ago
Was skimming at your Space Curve writeup [1] and your mention of discreet internal components caught my eye. Are you open to expanding a bit on this statement: "Discrete topology internals largely obviate secondary indexing" [1]: https://www.jandrewrogers.com/2015/10/08/spacecurve/ https://www.jandrewrogers.com/2015/10/08/spacecurve/
- eternalban 7y ago
  Ah, my coin dropped here. Most likely you mean the content addressable segmented architecture.
- jandrewrogers 7y ago
  Secondary indexing is a hack to address the reality that most indexing algorithms can only represent a single type of relationship efficiently and typically only in a single dimension. This is not a law of the universe, it is just how our algorithms tend to work. If you could eliminate secondary indexing without sacrificing selectivity, it would be a massive win for performance and scalability. However, this would require a single indexing algorithm for complex data models that preserved an arbitrary mix of relational, time-series, spatial, graph, etc relationships for searches and joins. To make this work in a practical database engine, you can't index the data model per se but you can make it work by indexing a moduli space into which arbitrary data models can be mapped. These tend to naturally expose the topological structure of the underlying data model for computational purposes even though you are not computing on the data model per se. Designing very general moduli spaces for database purposes is non-trivial and, to make matters worse, they are pathologically incompatible with typical surrounding database infrastructure once you figure out how to construct them. But you can use the exposed topology to execute complex searches and joins on the underlying data model. None of my database engines use secondary indexing at all, hence the excellent scaling and write performance, even for complex mixed-mode data models. A decade ago the representations were pretty brittle and limited because I didn't know how to express many things, but these days I know how to elegantly express just about every common data model.
  - 7y ago