4 ms·

Nikita, I am a huge fan of SingleStore - amazing what you’ve accomplished there and now what you’re building at Neon. Do you plans to build a columnstore index

by nojvek 4y ago

Nikita, I am a huge fan of SingleStore - amazing what you’ve accomplished there and now what you’re building at Neon. Do you plans to build a columnstore index on top of postgres that supports insert/update/delete? Love how MSSQL has a columnstore index for a subset of columns on a row store table. Always wondered why nobody has built something like that for postgres yet. Citus is nice but it’s append only, which is a huge restriction.

nikita 4y ago
We are not building it in house right now - we are focusing on OLTP. But have plenty ideas of how to do it in the future. In the first iteration we are thinking about an integration with Singlestore, Snowflake, and BigQuery. Singlestore will be best due to low latency updates, but the other two are very popular so integration with Snowflake and Big Query just makes sense. In the future we can integrate a columnstore right into the engine to make a smooth on system experience. There are some awesome open source implementations: arrow and duckdb. Updatability is tricky but doable as proven by Singlestore and SQL Server (I'm ex SQL Server and a huge fan of this feature). Not this year.
- nojvek 4y ago
  Makes sense. I agree. DuckDB is pretty phenomenal. I enjoyed reading its source code and playing around with it. Another open source nice C++ codebase is Typesense for Hyperfast text search (algolia competitor). It’s been on my mind for many months how to build indexing like this as postgres extensions. I love how versatile postgres is with so many indexing datastructures.
  - nikita 4y ago
    Again it is a big commitment to do a good job on implementing columnstores. If you don't do it all the way it not very usable and you are confusing your users by giving them too many options. The performance expectation is now set by great columstore implementations and you just can't afford a half ass job here. Here is an example from Google AlloyDB: https://twitter.com/mim_djo/status/1527900193626025984 https://twitter.com/mim_djo/status/1527900193626025984. My understanding that DuckDB is even faster on the TPCH benchmark. TPCDS is much harder and I doubt AllowDB can even run it at any reasonable scale.

by nojvek 4y ago

nikita 4y ago

We are not building it in house right now - we are focusing on OLTP. But have plenty ideas of how to do it in the future. In the first iteration we are thinking about an integration with Singlestore, Snowflake, and BigQuery. Singlestore will be best due to low latency updates, but the other two are very popular so integration with Snowflake and Big Query just makes sense. In the future we can integrate a columnstore right into the engine to make a smooth on system experience. There are some awesome open source implementations: arrow and duckdb. Updatability is tricky but doable as proven by Singlestore and SQL Server (I'm ex SQL Server and a huge fan of this feature). Not this year.

nojvek 4y ago
Makes sense. I agree. DuckDB is pretty phenomenal. I enjoyed reading its source code and playing around with it. Another open source nice C++ codebase is Typesense for Hyperfast text search (algolia competitor). It’s been on my mind for many months how to build indexing like this as postgres extensions. I love how versatile postgres is with so many indexing datastructures.
- nikita 4y ago
  Again it is a big commitment to do a good job on implementing columnstores. If you don't do it all the way it not very usable and you are confusing your users by giving them too many options. The performance expectation is now set by great columstore implementations and you just can't afford a half ass job here. Here is an example from Google AlloyDB: https://twitter.com/mim_djo/status/1527900193626025984 https://twitter.com/mim_djo/status/1527900193626025984. My understanding that DuckDB is even faster on the TPCH benchmark. TPCDS is much harder and I doubt AllowDB can even run it at any reasonable scale.