3 ms·
JavaScript Tooling for Data Pipelines
JS fullstack devs out there, what tooling are you using for creating data pipelines or for orchestrating data jobs? do you have to wander into SQL/Python land for this? or do you hack something together your self? As someone new to the JS community I have struggled to find a tool or platform for this. In Python land you have tings like Airflow and Dagster for example
- cotera-grant 3y agoWe're a "data team as a service" type company and we actually just open sourced our TypeScript library for doing data pipelines. We're still pretty early in the open sourcing / docs process but we've used it in production for over a year now and are really happy with the stuff we've built on top of it. A big bonus for us as fellow full stack JS devs is that the full compiler and type checker and SQL gen runs in the browser which we use to build cool stuff https://getnasty.dev/ https://getnasty.dev/ https://news.ycombinator.com/item?id=39641713 https://news.ycombinator.com/item?id=39641713
- floyergilmour 3y agoSuper cool!!! Is it almost like Drizzle but for BQ/Snowflake? I don't understand from the website if NASTY also is meant to just run typescript code to do (lets say webscraping, update some postgres DB, check for missing billing stripe billing etc etc)