4 ms·
We've been looking at a few for what I thought "basic" AI data prep tasks like scraping with backfill & periodic refreshes (basically a bunch of REST queries wi
by lmeyerov 4y ago
We've been looking at a few for what I thought "basic" AI data prep tasks like scraping with backfill & periodic refreshes (basically a bunch of REST queries with care not to overwhelm the targets; 2000's era backpressure good citizen stuff), and found that all the scheduling primitives ended up being half-baked from a data orchestration perspective. That's even before we get into periodic rescoring for the actual AI parts. So these ended up feeling like generic manual task/orchestration tools, which there are simpler + more powerful technologies. We did the same exercise 2-3 years ago, and surprisingly, not much change in core scheduler interfaces here from this perspective.
Curious if there are positive experiences with any tools here from a data orchestration perspective, esp OSS?
- swyx 4y agoi mean, the leading new OSS solutions are Prefect and Dagster, its not like there are a million of these out there, would just try them out and see what you think, each have their fans
- lmeyerov 4y agoWe have tried + investigated a variety, trying to avoid unnecessary shade for indiv co's. Hundreds of millions maybe even billions have gone into funding this space now, so folks get twitchy :)