4 ms·

Show HN: SQLMesh – The Future of DataOps

29 points by captaintobs 4y ago

Hey Show HN! I’m Toby and over the last few months, I’ve been working with a team of engineers from Airbnb, Apple, Google, and Netflix, to simplify developing data pipelines with SQLMesh (https://github.com/TobikoData/sqlmesh https://github.com/TobikoData/sqlmesh). We’re tired of fragile pipelines, untested SQL queries, and expensive staging environments for data. Software engineers have reaped the benefits of DevOps through unit tests, continuous integration, and continuous deployment for years. We felt like it was time for data teams to have the same confidence and efficiency in development as their peers. It’s time for DataOps! SQLMesh can be used through a CLI/notebook or in our open source web based IDE (in preview). SQLMesh builds efficient dev / staging environments through “Virtual Data Marts” using views, which allows you to seamlessly rollback or roll forward your changes! With a simple pointer swap you can promote your “staging” data into production. This means you get unlimited copy-on-write environments that make data exploration and preview of changes cheap, easy, safe. Some other key features are: Automatic DAG generation by semantically parsing and understanding SQL or Python scripts CI-Runnable Unit and Integration tests with optional conversion to DuckDB Change detection and reconciliation through column level lineage Native Airflow Integration Import an existing DBT project and run it on SQLMesh’s runtime (in preview) We’re just getting started on our journey to change the way data pipelines are built and deployed. We’re huge proponents of open source and hope that we can grow together with your feedback and contributions. Try out SQLMesh by following the quick start guide (https://sqlmesh.readthedocs.io/en/stable/quick_start/ https://sqlmesh.readthedocs.io/en/stable/quick_start/). We’d love to chat and hear about your experiences and ideas in our Slack community (https://join.slack.com/t/tobiko-data/shared_invite/zt-1ma66d79v-a4dbf4DUpLAQJ8ptQrJygg https://join.slack.com/t/tobiko-data/shared_invite/zt-1ma66d...).

mikabasketball 4y ago
Why should I switch to SQLMesh if I am already on DBT? DBT works fine for my use case today.
- captaintobs 4y ago
  Even if you don't have much data, SQLMesh will save you time over dbt. Iteration is an important part to developing data pipelines, and fully refreshing your warehouse every time is wasteful and can slow you down. Trying out SQLMesh is easy, you can import your existing project and leverage SQLMesh's runtime. At the very least, you can add unit tests to your project. You can see a more detailed comparison here https://sqlmesh.readthedocs.io/en/stable/comparisons/ https://sqlmesh.readthedocs.io/en/stable/comparisons/.

29 points by captaintobs 4y ago

mikabasketball 4y ago

Why should I switch to SQLMesh if I am already on DBT? DBT works fine for my use case today.

captaintobs 4y ago
Even if you don't have much data, SQLMesh will save you time over dbt. Iteration is an important part to developing data pipelines, and fully refreshing your warehouse every time is wasteful and can slow you down. Trying out SQLMesh is easy, you can import your existing project and leverage SQLMesh's runtime. At the very least, you can add unit tests to your project. You can see a more detailed comparison here https://sqlmesh.readthedocs.io/en/stable/comparisons/ https://sqlmesh.readthedocs.io/en/stable/comparisons/.