6 ms·

I'm curious if there are any projects out there that would search "schemaless" structured data, but still retain some of the structure. Essentially something l

I'm curious if there are any projects out there that would search "schemaless" structured data, but still retain some of the structure. Essentially something like the result of indexing a json row of field/value pairs for a bunch of csvs (with different fields) that would lead to being able to do faceted search on an individual field across rows/datasets, or being able to find the bits related to a needle deep in the dataset haystack.

karterk 5y ago
Typesense (from the latest version) has automatic schema detection: https://typesense.org/docs/0.20.0/api/collections.html#with-auto-schema-detection https://typesense.org/docs/0.20.0/api/collections.html#with-... It is quite powerful and you can even choose to "stringify" or co-erce types automatically to deal with dirty data.
- wiredfool 5y ago
  I'm not sure I see how that would provide faceting, or even how multiple rows would be coerced into something that could match either the dataset or the row. How it would work if there were incompatible fields in the schemas across different datasets? (unless it was basically a stringify operation across the whole thing. )
  - jabo 5y ago
    When you set the data type to "auto", the data type is inferred from the first document that you index. When subsequent documents are indexed, the fields are first attempted to be coerced to the previously determined data types. For eg: If record1 has {field1: 32, field2: true, field3: '22'} and record2 has {field1: "32", field2: 'true', field3: 22} When record1 is indexed, the data type for field1 is set to int, field2 is set to bool and field3 is set to string. Then when record2 shows up, field1 is coerced to an int, field2 is coerced to a bool and field3 is coerced to a string. If a coercion is not possible, you can configure it to be ignored or error out. re: facets, you can use a regex field name and set for example all fields that end with `.*_facet` to be a facet.

karterk 5y ago

Typesense (from the latest version) has automatic schema detection: https://typesense.org/docs/0.20.0/api/collections.html#with-auto-schema-detection https://typesense.org/docs/0.20.0/api/collections.html#with-... It is quite powerful and you can even choose to "stringify" or co-erce types automatically to deal with dirty data.

wiredfool 5y ago
I'm not sure I see how that would provide faceting, or even how multiple rows would be coerced into something that could match either the dataset or the row. How it would work if there were incompatible fields in the schemas across different datasets? (unless it was basically a stringify operation across the whole thing. )
- jabo 5y ago
  When you set the data type to "auto", the data type is inferred from the first document that you index. When subsequent documents are indexed, the fields are first attempted to be coerced to the previously determined data types. For eg: If record1 has {field1: 32, field2: true, field3: '22'} and record2 has {field1: "32", field2: 'true', field3: 22} When record1 is indexed, the data type for field1 is set to int, field2 is set to bool and field3 is set to string. Then when record2 shows up, field1 is coerced to an int, field2 is coerced to a bool and field3 is coerced to a string. If a coercion is not possible, you can configure it to be ignored or error out. re: facets, you can use a regex field name and set for example all fields that end with `.*_facet` to be a facet.