4 ms·
What is the situation with and roadmap for using JSON columns in DuckDB? A friend has a use case involving searching structured data (like everyone). I think i
by twic 3y ago
What is the situation with and roadmap for using JSON columns in DuckDB?
A friend has a use case involving searching structured data (like everyone). I think it's basically reports on different companies, so have fields like name, location, market cap, etc. They come as JSON documents, with very similar structures, which change slowly over time as fields are added and removed. He needs to give users an interface to these, where they can filter on various fields ("show me all small hairdressers in Minnesota" etc).
He currently dumps the JSON documents into ElasticSearch, and queries that. It works, but my gut feeling is that ElasticSearch is wildly less efficient than the theoretical optimum for this. Could DuckDB be an alternative?
The easiest thing would be if he could put whole JSON documents into a JSON column. Less easy would be breaking them up into real columns; i know that DuckDB is able to do this automatically, but it feels like a brittle thing to depend on.
I did some playing around with searching data from JSON, comparing a single JSON column to automatic 'proper' columns, and the proper columns were massively faster. Is there scope for making queries on JSON much faster, using indexes, or some setting i missed, or something?
Is DuckDB even likely to be a good solution for this? It's intended for analytics, whereas this is more of a retrieval / search / filtering use.
- deleted 3y ago[deleted]
- tabtab 3y agoJSON columns are a kludge. What's really needed is Dynamic Relational (see link). JSON columns are second-class citizens in such RDBMS. Dynamic Relational would make all columns just as powerful. A startup could use something like the Zend business model to make an open-source version and a for-fee enterprise version for scaling. https://www.reddit.com/r/CRUDology/comments/12ari2l/dynamic_relational_getting_dynamism_without/ https://www.reddit.com/r/CRUDology/comments/12ari2l/dynamic_...
- pzmarzly 3y agoOld Reddit link, so you don't need an app on mobile: https://old.reddit.com/r/CRUDology/comments/12ari2l/dynamic_relational_getting_dynamism_without/ https://old.reddit.com/r/CRUDology/comments/12ari2l/dynamic_...