3 ms·
I recently had the experience of setting up some Prefect pipelines, which I can compare to this article. Note that while I'm not new to data engineering, I'm ne
by singhrac 4y ago
I recently had the experience of setting up some Prefect pipelines, which I can compare to this article. Note that while I'm not new to data engineering, I'm new to open source frameworks, and have some insight into Airflow (studied architecture in depth, written a lot of code in it).
Prefect is generally very easy to use. Essentially, you: (a) write a Python-based flow, which defines some job to run (with subtasks), (b) turn on an orchestrator on a server somewhere, (c) turn on an agent on a server somewhere (to run the flow when instructed by the orchestrator), and (d) connect to the orchestrator, build & apply a deployment, and run it.
I find the docs a little half baked right now. One example is that cron jobs, which one would think are essential to something like Prefect, basically can't be done (as of a month ago) without touching the Prefect UI. This is extremely odd.
I also found it fairly confusing which components were supposed to be checked into source control, and which weren't. I blame this on Python deployment generally being very odd and confusing, but Prefect docs don't make it any more clear. Prefect assumes that there's an S3-like storage that both the submitting computer (my laptop) and the orchestrator (the server) can access.
Overall I find it quite handy, and probably won't switch. It feels more lightweight than say using full Docker containers, which we probably don't need right now. The UI is nicer than Airflow's, and the orchestrator & agent are much easier on resources. It feels more reproducible. I haven't tried Prefect Cloud, and we're unlikely to (security & cost are the main reasons).
- nojito 4y agoThe single best reason to use Prefect is in it's integration with Dask/Ray. This makes scaling Prefect so much simpler than Airflow.
- lmeyerov 4y agoWe've been looking at a few for what I thought "basic" AI data prep tasks like scraping with backfill & periodic refreshes (basically a bunch of REST queries with care not to overwhelm the targets; 2000's era backpressure good citizen stuff), and found that all the scheduling primitives ended up being half-baked from a data orchestration perspective. That's even before we get into periodic rescoring for the actual AI parts. So these ended up feeling like generic manual task/orchestration tools, which there are simpler + more powerful technologies. We did the same exercise 2-3 years ago, and surprisingly, not much change in core scheduler interfaces here from this perspective. Curious if there are positive experiences with any tools here from a data orchestration perspective, esp OSS?
- swyx 4y agoi mean, the leading new OSS solutions are Prefect and Dagster, its not like there are a million of these out there, would just try them out and see what you think, each have their fans
- lmeyerov 4y agoWe have tried + investigated a variety, trying to avoid unnecessary shade for indiv co's. Hundreds of millions maybe even billions have gone into funding this space now, so folks get twitchy :)
- rpeden 4y agoDisclaimer: I work for Prefect. It looks like we added cron and other schedule types to the deployment CLI just under a month ago[1]. Over the last couple of releases, we've also made it easier to pull deployments from GitHub or bake your flow code into Docker images instead of needing S3-like storage. As with any product, there's always more to do, so I appreciate you sharing your thoughts. More than anywhere else I've worked, community feedback is a huge driver of product enhancements and feature development. Feel free to join our Slack community[2] if you'd like to share more feedback or ask questions. [1] https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md#210 https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES... [2] https://www.prefect.io/slack/ https://www.prefect.io/slack/