4 ms·

SQL is simpler to understand for majority of users. It's easier to get started without learning a tool chain, programming best practices, etc. that could presen

SQL is simpler to understand for majority of users. It's easier to get started without learning a tool chain, programming best practices, etc. that could present a challenge to new users. SQL is also great at representing relationships between datasets and developing business logic transformations tends to be simpler and easier to understand. Oftentimes, when using another programming language you end up with many modules, imported libraries, code hacks and optimizations. It all can very quickly make it difficult to read. One example is dbt. They started as writing simple SQL models. With the introduction of Jinja, majority of models look nothing like SQL anymore. You need to visualize your model to understand relationships. It took the beauty of SQL and mucked it. At Upsolver we built a streaming+batch ETL tool that lets you build data pipelines in SQL. We did it because it's easier for non-data engineers to get started, easy to version and maintain as code and easy to automate (not that you can't do this with other languages). The same goes with kSQLDB, Materialize and even Spark and Flink use SQL as a way to simplify onboarding for non-developers.

billythemaniam 4y ago
> One example is dbt. They started as writing simple SQL models. With the introduction of Jinja, majority of models look nothing like SQL anymore. You need to visualize your model to understand relationships. It took the beauty of SQL and mucked it. I know you are pitching your startup, but that's just completely untrue.
- srcreigh 4y ago
  The dbt site describes this as a simple example: select * from {{ref('really_big_table')}} {% if incremental and target.schema == 'prod' %} where timestamp >= (select max(timestamp) from {{this}}) {% else %} where timestamp >= dateadd(day, -3, current_date) {% endif %}
  - billythemaniam 4y ago
    As someone who has written 100s (maybe 1000s at this point) of DBT models, the amount of Jinja you need is at most 1-2% of the codebase.

billythemaniam 4y ago

> One example is dbt. They started as writing simple SQL models. With the introduction of Jinja, majority of models look nothing like SQL anymore. You need to visualize your model to understand relationships. It took the beauty of SQL and mucked it. I know you are pitching your startup, but that's just completely untrue.

srcreigh 4y ago
The dbt site describes this as a simple example: select * from {{ref('really_big_table')}} {% if incremental and target.schema == 'prod' %} where timestamp >= (select max(timestamp) from {{this}}) {% else %} where timestamp >= dateadd(day, -3, current_date) {% endif %}
- billythemaniam 4y ago
  As someone who has written 100s (maybe 1000s at this point) of DBT models, the amount of Jinja you need is at most 1-2% of the codebase.