4 ms·

I've been using Airflow to coordinate scrapers that hit a number of various sites as part of a global market awareness system I've been building over the last y

I've been using Airflow to coordinate scrapers that hit a number of various sites as part of a global market awareness system I've been building over the last year or so. I have given up on BeautifulSoup and Scrapy since so many modern websites use obfuscated JS to hide the underlying data they are serving up, so I feel like its better to just act like a user and slowly walk through whatever site actions need to be done to get to the data you want to ingest. Needless to say, as many have touched on in this post's comments, scraping reliably, and selectively retrying based on the many tens if not hundreds of different potential errors that can occur (either server side / API limitations, or client side based on the interaction that your browser, i.e. shit crashing, etc) is really almost an optimization problem of its own. Definitely a boon to have scraping as an option, but as always, licensure of data especially if you want to resell it becomes a major concern that you should be thinking about up-front even if you kinda just want to hack things together in the beginning.

guskel 5y ago
Does Airflow support streaming the outputs to downstream tasks? I tried to do something like this with Prefect but with Prefect you have to wait for the upstream task to finish before a downstream task can begin working.
- shmoogy 5y ago
  My understanding is each task is supposed to idempotent, so I don't believe this is a valid use for Airflow.
  - arthurcolle 5y ago
    Yeah your understanding is correct but if you relax the idempotency constraint you can achieve a lot more with just a little bit extra logic in your interface layer with other internal services or potentially other mechanisms to ensure consistency. YMMV
- arthurcolle 5y ago
  Hit me up at my email in my profile if you want to chat about this stuff, I have a lot of thoughts on this but it's probably off topic for this post and I usually am just hacking stuff together to get my systems up and running. What you're talking about is very sensible and I was equally surprised that Airflow didn't support long running tasks but you can layer over the workflow orchestration system a kind of ad-hoc higher order system that enables what you speak of. It kind of feels ugly but can get a lot done. There are definitely ways to accomplish what you are saying using a combination of DockerOperators + ephemeral WebSocket servers running within containers as semi-long running tasks, and basically just have a dumb/heavy Redis container that persists to run streaming between the coordination architecture across these data flow jobs. "Work in progress" lol! EDIT: updating Airflow from 1.10.10 to 2.1.2 recently was a huge pain in the ass for what it's worth, good luck to all our fellow protagonists that are dealing with multi tens of thousands of task DAG setups... big ooooff

guskel 5y ago

Does Airflow support streaming the outputs to downstream tasks? I tried to do something like this with Prefect but with Prefect you have to wait for the upstream task to finish before a downstream task can begin working.

shmoogy 5y ago
My understanding is each task is supposed to idempotent, so I don't believe this is a valid use for Airflow.
- arthurcolle 5y ago
  Yeah your understanding is correct but if you relax the idempotency constraint you can achieve a lot more with just a little bit extra logic in your interface layer with other internal services or potentially other mechanisms to ensure consistency. YMMV
arthurcolle 5y ago
Hit me up at my email in my profile if you want to chat about this stuff, I have a lot of thoughts on this but it's probably off topic for this post and I usually am just hacking stuff together to get my systems up and running. What you're talking about is very sensible and I was equally surprised that Airflow didn't support long running tasks but you can layer over the workflow orchestration system a kind of ad-hoc higher order system that enables what you speak of. It kind of feels ugly but can get a lot done. There are definitely ways to accomplish what you are saying using a combination of DockerOperators + ephemeral WebSocket servers running within containers as semi-long running tasks, and basically just have a dumb/heavy Redis container that persists to run streaming between the coordination architecture across these data flow jobs. "Work in progress" lol! EDIT: updating Airflow from 1.10.10 to 2.1.2 recently was a huge pain in the ass for what it's worth, good luck to all our fellow protagonists that are dealing with multi tens of thousands of task DAG setups... big ooooff