4 ms·

DataChain: DBT for Unstructured Data

163 points by shcheklein 2y ago

whalesalad 2y ago
> It is made to organize your unstructured data into datasets and wrangle it at scale on your local machine. How does one wrangle terabytes of data on a local machine?
- shcheklein 2y ago
  The idea is that it doesn't store binary files locally, just pointers in the DB + meta data (SQLite if you run locally, open source). So, it's versioning, structuring of datasets, etc by "references" if you wish. (that's is different from let's say DVC - that does copy files into a local cache, always)
  - aduffy 2y ago
    So in the case from the README, where you're trying to curate a sample of your data, the only thing that you're reading is the metadata, UNTIL you run `export_files` and that actually copies the binary data to your local machine?
    - dmpetrov 2y ago
      Exactly! DataChain does lazy compute. It will read metadata/json while applying filtering and only download a sample of data files (jpg) based on the filter. This way, you might end up downloading just 1% of your data, as defined by the metadata filter.
mpeg 2y ago
It took me a minute to grok what this was for, but I think I like it It doesn't really replace any of the tooling we use to wrangle data at scale (like prefect or dagster or temporal) but as a local library it seems to be excellent, I think what confused me most was the comparison to dbt. I like the from_* utils and the magic of the Column class operator overloading and how chains can be used as datasets. Love how easy checkpointing is too. Will give it a go
- dmpetrov 2y ago
  Yes, it's not meant to replace data engineering tools like Prefect or Temporal. Instead, it serves as a transformation engine and ad-hoc analytics for images/video/text data. It's pretty much DBT use case for text and images in S3/GCS, though every analogy has its limits. Try it out - looking forward to your feedback!
dmpetrov 2y ago
Yay! Excited to see DataChain on the front page :) Maintainer and author here. Happy to answer any questions. We built DataChain because our DVC couldn't fully handle data transformations and versioning directly in S3/GCS/Azure without data copying. Analogy with "DBT for unstractured data" applies very well to DataChain since it transforms data (using Python, not SQL) inside in storages (S3, not DB). Happy to talk more!
jerednel 2y ago
Cool! Does this assume the unstructured data already has a corresponding metadata file? My most common use cases involve getting PDFs or HTML files and I have to parse the metadata to store along with the embedding. Would I have to run a process to extract file metadata into JSONs for every embedding/chunk? Would keys created based off document be title+chunk_no? Very interested in this because documents from clients are subject to random changes and I don’t have very robust systems in place.
- dmpetrov 2y ago
  DataChain has no assumptions about metadata format. However, some formats are supported out of the box: WebDataset, json-pair, openimage, etc. Extract metadata as usual, then return the result as JSON or a Pydantic object. DataChian will automatically serialize it to internal dataset structure (SQLite), which can be exported to CSV/Parquet. In case of PDF/HTML, you will likely produce multiple documents per file which is also supported - just `yield return my_result` multiple times from map(). Check out video: https://www.youtube.com/watch?v=yjzcPCSYKEo https://www.youtube.com/watch?v=yjzcPCSYKEo Blog post: https://datachain.ai/blog/datachain-unstructured-pdf-processing https://datachain.ai/blog/datachain-unstructured-pdf-process...
  - spott 2y ago
    > DataChain has no assumptions about metadata format. Could your metadata come from something like a Postgres sql statement? Or an iceberg view?
    - dmpetrov 2y ago
      Absolutely, that's a common scenario! Just connect from your Python code (like the lambda in the example) to DB and extract the necessary data.
  - nbbaier 2y ago
    > However, some formats are supported out of the box: WebDataset, json-pair, openimage, etc. Forgive my ignorance, but what is "json-pair"?
datascientist 2y ago
How does this relate to https://github.com/lancedb/lance https://github.com/lancedb/lance
- dmpetrov 2y ago
  Lance is just a data format. Lance DB might be more comparable to DataChain. DataChain focuses on data transformation and versioning, whereas LanceDB appears to be more about retrieving and serving data. Both designed for multimodal use cases. From technical side: Lance has it's own data format and DB engine while DataChain utilizes existing DB engines (SQLite in open-source and ClickHouse/BigQuery in SaaS). In SaaS, DataChain has analytics features including data lineage tracking and visualization for PDFs, videos, and annotated images (e.g., bounding boxes, poses). I'm curious to understand the unique value of LanceDB's SaaS — insight would be helpful! You could think of it as OLTP (Lance) versus OLAP (DataChain) for multimodal data, though this analogy may not be perfect.
  - m0sth8 2y ago
    How about daft https://github.com/Eventual-Inc/Daft https://github.com/Eventual-Inc/Daft - also looks like a new multimodal dataframe framework
    - dmpetrov 2y ago
      Good question! I’m not so familiar with it. It looks like Daft is closer to Lance with it’s own data format and engine. But I’d appreciate more insights from users or the creators.
    - jaychia 2y ago
      One of the maintainers of Daft here. Just dug through the datachain codebase to understand a little more. I think while both projects have a Dataframe interface, they're very different projects! Datachain seems to operate more on the orchestration layer, running Python libraries such as PIL and requests (for making API calls) and relying on an external database engine (SQLite or BigQuery/Clickhouse) for the actual compute. Daft is an actual data engine. Essentially, it's "multimodal BigQuery/Clickhouse". We've built out a lot of our own data system functionality such as custom Rust-defined multimodal data structures, kernels to work on multimodal types, a query optimizer, distributed joins etc. In non-technical terms, I think this means that Datachain really is more of a "DBT" which orchestrates compute over an existing engine, whereas Daft is the actual compute/data engine that runs the workload. A project such as Datachain could actually run on top of Daft, which can handle the compute and I/O operations necessary to execute the requested workload.
tatigabru 2y ago
Wow! Great instrument, so excited to see it here
- dmpetrov 2y ago
  Please share your feedback!
SOLAR_FIELDS 2y ago
> Datachain does not abstract or hide the AI models and API calls, but helps to integrate them into the postmodern data stack. I’m not sure if this term postmodern data stack was invented for the purposes of this copy. Probably not. But terms like this don’t really engender a lot of faith that this isn’t yet another piece of the now decades long hype cycle data engineering products face
- dmpetrov 2y ago
  Good point - it does sound a bit like marketing bs. We’ll rephrase it.

163 points by shcheklein 2y ago

> It is made to organize your unstructured data into datasets and wrangle it at scale on your local machine. How does one wrangle terabytes of data on a local machine?

shcheklein 2y ago
The idea is that it doesn't store binary files locally, just pointers in the DB + meta data (SQLite if you run locally, open source). So, it's versioning, structuring of datasets, etc by "references" if you wish. (that's is different from let's say DVC - that does copy files into a local cache, always)
- aduffy 2y ago
  So in the case from the README, where you're trying to curate a sample of your data, the only thing that you're reading is the metadata, UNTIL you run `export_files` and that actually copies the binary data to your local machine?
  - dmpetrov 2y ago
    Exactly! DataChain does lazy compute. It will read metadata/json while applying filtering and only download a sample of data files (jpg) based on the filter. This way, you might end up downloading just 1% of your data, as defined by the metadata filter.

mpeg 2y ago

It took me a minute to grok what this was for, but I think I like it It doesn't really replace any of the tooling we use to wrangle data at scale (like prefect or dagster or temporal) but as a local library it seems to be excellent, I think what confused me most was the comparison to dbt. I like the from_* utils and the magic of the Column class operator overloading and how chains can be used as datasets. Love how easy checkpointing is too. Will give it a go

dmpetrov 2y ago
Yes, it's not meant to replace data engineering tools like Prefect or Temporal. Instead, it serves as a transformation engine and ad-hoc analytics for images/video/text data. It's pretty much DBT use case for text and images in S3/GCS, though every analogy has its limits. Try it out - looking forward to your feedback!

dmpetrov 2y ago

Yay! Excited to see DataChain on the front page :) Maintainer and author here. Happy to answer any questions. We built DataChain because our DVC couldn't fully handle data transformations and versioning directly in S3/GCS/Azure without data copying. Analogy with "DBT for unstractured data" applies very well to DataChain since it transforms data (using Python, not SQL) inside in storages (S3, not DB). Happy to talk more!

jerednel 2y ago

Cool! Does this assume the unstructured data already has a corresponding metadata file? My most common use cases involve getting PDFs or HTML files and I have to parse the metadata to store along with the embedding. Would I have to run a process to extract file metadata into JSONs for every embedding/chunk? Would keys created based off document be title+chunk_no? Very interested in this because documents from clients are subject to random changes and I don’t have very robust systems in place.

dmpetrov 2y ago
DataChain has no assumptions about metadata format. However, some formats are supported out of the box: WebDataset, json-pair, openimage, etc. Extract metadata as usual, then return the result as JSON or a Pydantic object. DataChian will automatically serialize it to internal dataset structure (SQLite), which can be exported to CSV/Parquet. In case of PDF/HTML, you will likely produce multiple documents per file which is also supported - just `yield return my_result` multiple times from map(). Check out video: https://www.youtube.com/watch?v=yjzcPCSYKEo https://www.youtube.com/watch?v=yjzcPCSYKEo Blog post: https://datachain.ai/blog/datachain-unstructured-pdf-processing https://datachain.ai/blog/datachain-unstructured-pdf-process...
- spott 2y ago
  > DataChain has no assumptions about metadata format. Could your metadata come from something like a Postgres sql statement? Or an iceberg view?
  - dmpetrov 2y ago
    Absolutely, that's a common scenario! Just connect from your Python code (like the lambda in the example) to DB and extract the necessary data.
- nbbaier 2y ago
  > However, some formats are supported out of the box: WebDataset, json-pair, openimage, etc. Forgive my ignorance, but what is "json-pair"?

datascientist 2y ago

How does this relate to https://github.com/lancedb/lance https://github.com/lancedb/lance

dmpetrov 2y ago
Lance is just a data format. Lance DB might be more comparable to DataChain. DataChain focuses on data transformation and versioning, whereas LanceDB appears to be more about retrieving and serving data. Both designed for multimodal use cases. From technical side: Lance has it's own data format and DB engine while DataChain utilizes existing DB engines (SQLite in open-source and ClickHouse/BigQuery in SaaS). In SaaS, DataChain has analytics features including data lineage tracking and visualization for PDFs, videos, and annotated images (e.g., bounding boxes, poses). I'm curious to understand the unique value of LanceDB's SaaS — insight would be helpful! You could think of it as OLTP (Lance) versus OLAP (DataChain) for multimodal data, though this analogy may not be perfect.
- m0sth8 2y ago
  How about daft https://github.com/Eventual-Inc/Daft https://github.com/Eventual-Inc/Daft - also looks like a new multimodal dataframe framework
  - dmpetrov 2y ago
    Good question! I’m not so familiar with it. It looks like Daft is closer to Lance with it’s own data format and engine. But I’d appreciate more insights from users or the creators.
  - jaychia 2y ago
    One of the maintainers of Daft here. Just dug through the datachain codebase to understand a little more. I think while both projects have a Dataframe interface, they're very different projects! Datachain seems to operate more on the orchestration layer, running Python libraries such as PIL and requests (for making API calls) and relying on an external database engine (SQLite or BigQuery/Clickhouse) for the actual compute. Daft is an actual data engine. Essentially, it's "multimodal BigQuery/Clickhouse". We've built out a lot of our own data system functionality such as custom Rust-defined multimodal data structures, kernels to work on multimodal types, a query optimizer, distributed joins etc. In non-technical terms, I think this means that Datachain really is more of a "DBT" which orchestrates compute over an existing engine, whereas Daft is the actual compute/data engine that runs the workload. A project such as Datachain could actually run on top of Daft, which can handle the compute and I/O operations necessary to execute the requested workload.

tatigabru 2y ago

Wow! Great instrument, so excited to see it here

dmpetrov 2y ago
Please share your feedback!

SOLAR_FIELDS 2y ago

> Datachain does not abstract or hide the AI models and API calls, but helps to integrate them into the postmodern data stack. I’m not sure if this term postmodern data stack was invented for the purposes of this copy. Probably not. But terms like this don’t really engender a lot of faith that this isn’t yet another piece of the now decades long hype cycle data engineering products face

dmpetrov 2y ago
Good point - it does sound a bit like marketing bs. We’ll rephrase it.