3 ms·
I like using tools like Apache Drill for the schemaless use case. I apply that in use cases where you’re dropping files with maybe similar schemes into a folde
by sixdimensional 8y ago
I like using tools like Apache Drill for the schemaless use case. I apply that in use cases where you’re dropping files with maybe similar schemes into a folder and doing exploratory analysis on the way to a structured solution, or for quick throwaways, or even for data integration ELT type solutions, as you mentioned. This use case works well with data vault-like architecture (or “data lakes”).
Actually, anything that can read and perform a SQL query against a standards based file in a file system (e.g. CSV, JSON, Parquet, etc. on HDFS or regular FS). Apache Hive, PrestoDB, Spark, or even some RDBMS which support Foreign Data Wrappers can do this, although some of those require a schema to be specified for querying even if the underlying files can be of different schemas. Drill doesn’t have that limitation and can literally query without metadata. Schema / data types matter so true schemaless can be difficult unless you manually map data types, but it has a place.