3 ms·
Disco is interesting, though it seems like the multi-language support is a bit lacking. Someone wrote a Go worker, but it's unfinished and abandoned. A pipelin
by lobster_johnson 10y ago
Disco is interesting, though it seems like the multi-language support is a bit lacking. Someone wrote a Go worker, but it's unfinished and abandoned.
A pipeline processing system needs durable queueing, though. Maybe you could use the Disco distributed file system or maybe plug in something like Kafka, though I'm not sure how.
- toast0 10y ago> A pipeline processing system needs durable queueing, though. Eh, it might be nice to have, and a lot of systems have it, but it's not necessarily a need. Depends on your use case, and your environment -- most days, none of my servers have any problems; on the days where there are problems, I could manually restart my pipeline jobs. (I would need durable storage of the inputs to the pipeline.)
- lobster_johnson 10y agoI was referring to the inputs, mostly. My app also has a lot of intermediate data that wants to stick around and be durable across batches, things like scraping results that needs to avoid incurring duplicate work when nothing has changed. For that, some kind of data-locality-awareness would also be nice.