30 ms·
Hey! Looks great! Is there an example of the Github integration - how does it looks like? I'm one of the developers and maintainer of the DVC project and we re
by ishcheklein 6y ago
Hey! Looks great! Is there an example of the Github integration - how does it looks like?
I'm one of the developers and maintainer of the DVC project and we recently released CML.dev- which integrates with Github and can be used to run some checks on data as well. But in our case it's about analyzing files more or less. I'm curious how does that integration look like in your case.
- hichkaker 6y agoThanks! We are going to add an example to our website and perhaps publish it as a Github App. The flow is: For every whitelisted (say, SQL) file changed by a pull request, our bot compares the production data (produced by the master branch) vs. the data output of the new branch and pastes the diff as a comment to the PR. Some YAML configuration is needed to, for example, adjust it to your repo file structure.
- ishcheklein 6y agoThanks! And how and where does setup happens which database to use to run the query for the specific SQL file? Also if it's part of some pipeline will it have to run the whole pipeline from the very beginning?
- hichkaker 6y agoSo there are two main workflows: 1) Run diff as part of CI (on code change), that can be done via Github workflow (or other CI). The Github workflow takes in a YAML config file, and also is customizable so you can fit it according your needs. For example, one of our customers names SQL files after tables they populate, so when the bot sees a changed SQL file, it knows what table to compare it to. 2) If you are looking to run a diff within the data pipeline itself (e.g. in Airflow), you can wrap Datafold API in an Airflow operator to integrate it as a task in your DAG. I hope that I covered your questions but please let me know if you are interested in specific use cases!