4 ms·

Hi Blake, nice product, and great to see your templates on github, it really helped to understand what they do. I've been using Prefect for a while as an ELT c

by sails 6y ago

Hi Blake, nice product, and great to see your templates on github, it really helped to understand what they do. I've been using Prefect for a while as an ELT coordinator, and pretty happy with it. It is in use as a "smart-cron" rather than sophisticated DAG creator, but so far I've been pretty happy. From what I've read on your website and watching the video, Shipyard (love the theme, but I'm all over nautical themes) seems like a "Zapier for data/analytics engineering teams". How close is that to your idea? From the demo, I imagine using this in the following scenario: 1. Don't want to self-host any infrastructure 2. Need to write a few scripts to glue some things together (client wants dataset in email weekly / lead scoring hack / data quality check) 3. Would benefit from some pre-built source/sink/chat connectors and recipes. I'd guess you aren't positioning this as a bulk ELT tool, but perhaps more for a scenario where execution order is priority. My only thought for an addition would be to enable self-hosting code for my python/bash vessels, to ensure I have better control over deployments.

blakeburch 6y ago
Hey Matt - You hit the nail on the head with "Zapier for data/analytics engineering teams" and your example scenario. Our goal is to make it as simple as possible to get any quick data scripts up and running. However, we are positioning this as a tool where you can accomplish bulk ETL. The infrastructure is designed to scale for long-running jobs with heavy data usage. It's easier to demo and understand the "quick solutions", but we have teams running Fleets that download and process 100s of GBs of data every day. Agreed on the need to allow self-hosting the code. It's currently on our roadmap to build in native Github connections so that teams can host the code and version manage it there, connecting Shipyard as part of their CI/CD flow. Having to copy/paste the code, or upload it, is a means to an end right now, but not much different than the flow for something like AWS Lambda. As for Prefect, we've heard a lot of great things about their setup. However, there's a few areas that we find it lacking: - Python Only. We want to enable workflows that can connect solutions built in multiple different languages (which we support through Bash for now). - Infrastructure management is your responsibility, so you still might hit snags and resource constraints. - Their workflow as code setup requires you to change how you write your scripts to make them work in Prefect... and it's my understanding that all the steps of a workflow have to live in a single script. We're opting for a more modular approach, where code could easily be run on your local machine, or your own infrastructure, without needing to rewrite it. I'd love to get more feedback about your current setup and experience. Feel free to hit me up at blake[at]shipyardapp.com

by sails 6y ago

blakeburch 6y ago

Hey Matt - You hit the nail on the head with "Zapier for data/analytics engineering teams" and your example scenario. Our goal is to make it as simple as possible to get any quick data scripts up and running. However, we are positioning this as a tool where you can accomplish bulk ETL. The infrastructure is designed to scale for long-running jobs with heavy data usage. It's easier to demo and understand the "quick solutions", but we have teams running Fleets that download and process 100s of GBs of data every day. Agreed on the need to allow self-hosting the code. It's currently on our roadmap to build in native Github connections so that teams can host the code and version manage it there, connecting Shipyard as part of their CI/CD flow. Having to copy/paste the code, or upload it, is a means to an end right now, but not much different than the flow for something like AWS Lambda. As for Prefect, we've heard a lot of great things about their setup. However, there's a few areas that we find it lacking: - Python Only. We want to enable workflows that can connect solutions built in multiple different languages (which we support through Bash for now). - Infrastructure management is your responsibility, so you still might hit snags and resource constraints. - Their workflow as code setup requires you to change how you write your scripts to make them work in Prefect... and it's my understanding that all the steps of a workflow have to live in a single script. We're opting for a more modular approach, where code could easily be run on your local machine, or your own infrastructure, without needing to rewrite it. I'd love to get more feedback about your current setup and experience. Feel free to hit me up at blake[at]shipyardapp.com