6 ms·

How to Manage Apache Airflow with Systemd on Debian or Ubuntu

66 points by njanakiev 7y ago

einpoklum 7y ago
How to manage it? In my opinion: Just switch to Devuan[1] - making it systemd-less - and use sysvinit scripts (these days there's support for parallelism and dependencies with insserv and startpar). [1] : https://www.devuan.org/
- fignews 7y ago
  Your reply has nothing to do with the article. Perhaps after reading the post your systemd hate will subdue some.
  - 1996 7y ago
    For managing a complex set of daemons, systemd is the thing. I only wish that rules did integrate with system monitoring to have conditions (ex: something like StopWhenLoadThreshold_40=xx, StartWhenDiskFreeThreshold=99% )
villasv 7y ago
I have found that using KillMode=mixed is very useful when running Airflow with Celery workers. It allows the system to shudown gracefully coupled with TimeoutStopSec, so the workers will stop receiving new jobs but will finish their current jobs before exiting which is nice for auto scaling or spot instances on AWS (coupled with EC2 lifecycle hooks).
- 1996 7y ago
  Interesting. Why do you care about graceful shutdown? Why not kill ASAP and restart? Do you have problems with load booming up sometimes?
  - akramar 7y ago
    In some cases we need to do an update-deploy-restart while a DAG is still running (not even the one being updated). Then several minutes or hours later the child processes raise a segfault and the jobs they were working on fail, requiring restarting any of those jobs. I imagine a graceful shutdown would allow the job to finish up and the DAG to continue with the remaining jobs.
tyingq 7y ago
ETL is a funny space. At least in the "Enterprise" world, it's dominated by Ab Initio, which is crazy expensive. They seem to be coasting too, for quite some time. Their website is probably the most terrible site I've ever seen for an expensive piece of software. You can't even tell what it is, how to buy it, or even how to contact them. https://www.abinitio.com/en/ https://www.abinitio.com/en/
- thenaturalist 7y ago
  What's your source for the fact that Ab Initio is dominating? Other than that, there are a several other tools in the Enterprise Analytics space that fall into a similar pattern like Alteryx or Collibra. But from their perspective it makes perfect sense, I guess. When your sales is done by relationship building and there isn't much competition once you're in, there isn't really a need to boast a fancy website or make an effort. If anyone has a good resource on how enterprise IT procurement is done or the dynamics around it, I'd love to read up on that.
  - tyingq 7y ago
    Just anectdotal experience. By "dominating", I mean in non-tech US Fortune 500 companies. They have very few employees, and the software is very expensive. "When your sales is done by relationship building and there isn't much competition once you're in, there isn't really a need to boast a fancy website or make an effort." Sure, but try and find a phone number or email address on their site. They've taken coasting to a whole new level.
  - kfk 7y ago
    Well I have a lot of experience with IT procurement. IT is more about cost and risk mitigation than technology. The last thing IT wants is to try new software that might blow up in the hands of their unprepared outsourced helpdesk support people. So the absolute first thing you want to show is robust and cheap support or even better a way to make sure your product works with their helpdesk setup. Another thing is cost and how your product helps them run things with less money. They could not care less about the quality of the product because they will not be the ones using it. In general you sell by telling a story that fits with whatever story they are telling the business. So if IT is telling business cloud is the next thing and you sell on premise you won’t go anywhere.
EddieCPU 7y ago
I would have thought a scripting language would have been a better choice than unit files. The script engine would take care of verifying the script and starting the processes in the correct order and allowing/restricting access to other components. Thereby negating the need for such directives as After= or PrivateTmp=.
- onefuncman 7y ago
  Unit files are one of the things systemd got right. A scripting language isn't well suited for process management, which is a very well specified task. Now, if you mentioned https://ammonite.io/ https://ammonite.io/ you might have got my attention...
Peteris 7y ago
Side note: the best way to build Airflow pipelines is through Kedro https://github.com/quantumblacklabs/kedro-airflow https://github.com/quantumblacklabs/kedro-airflow.
Lucasoato 7y ago
(Sorry if it's a little off topic) We were close to adopt Airflow in our company but we were let down by a detail: the scheduler isn't natively in a high availability mode. There was an article from Clairvoyant about how to make it HA but it didn't look safe at all. That was a serious issue for us, and at the end we went for NiFi. Have anyone had this problem?
- robbyt 7y ago
  Use K8s to schedule the work.
  - Filligree 7y ago
    And make sure you hire two engineers to keep it working.
- smooc 7y ago
  Airflow maintainer here If you have idempotent tasks, which is a best practice, it is possible to use Airflow in HA even in active/active. It might occasionally schedule a task twice, which should be caught but in any case mitigated by idempotency. If you are looking for more Enterprise support you can reach out to Astronomer (disclaimer: I'm a advisor to them) or use a Google cloud hosted version (Cloud Composer). Both are great products.
kfk 7y ago
Me and my team tried Airflow but found it didn’t fit well with our analytics work flow. For instance you must rewrite your jupyter notebook into an Airflow dag, doing basically the same work 2 times. We use dask and will soon deploy dask.distributed. I have yet to figure out where Airflow actually fits in the BI/data science architecture.
- onefuncman 7y ago
  I recall Jupyter being a parallel evolution alongside Airflow+Spark+Zeppelin (or similar mashup) and I think Jupyter has become "better".
  - smooc 7y ago
    Airflow maintainer here Jupyter doesn't do scheduling and integrates pretty well with Airflow.
- jmngomes 7y ago
  Not sure what you mean by "BI/data science architecture" but Airflow is essentially a scheduler and orchestrator for data processing jobs. These activities are usually managed by cron and more often by advanced scheduler tools (depending on the vendor), so it's quite a core part of any architecture that needs to e.g. load/reload/refresh data periodically. If the requirement is simply to connect notebooks to a data lake, then the only scheduling required is to load the data lake, and something like Airflow may be overkill for this, depending on what/how the data is processed and loaded.
  - kfk 7y ago
    I mean the same thing you mean. My issue with airflow is that it’s complicated and doesn’t adapt well to cloud computing. Dask runs on aws emr and eks, Kubernetes, etc.. Unfortunately orchestration is a lot more complicated than it looks. Parallel executions, retries, logs, status tracking, email notifications. Airflow doesn’t really tackle all orchestration work.
    - 7y ago
carlosf 7y ago
Good article! I actually like systemd, but nowadays I generally run stuff as containers, so there is less and less opportunity to use it. Here is my current setup for Airflow: - 1 container for the webserver - 1 container for the scheduler - 1 managed database (I use Postgres, it's a fairly small instance.) - 1 S3 bucket to deploy DAGs. I mount it on my containers using s3fs-fuse. - You can monitor the scheduler using a PID file, whereas the webserver can be monitored probing your admin URL. - Most configuration can be done using environment variables, which is perfect for containers. - I also configure DAG logs to be shipped to a S3 bucket.
jpollock 7y ago
Why are all these tools DAGs? I'm guessing I misunderstand what is meant by a "Workflow"? My assumption is that these are managing a state machine, where workflow is stand-in for "Business Process"? If it's doing that sort of job, I'd expect timers and loops? However, it seems these are aimed at data conversion pipelines?
- bathtub365 7y ago
  It has a cron scheduler
- jpau 7y ago
  Heyo! Data guy here. Airflow and its DAG-managing peers are important for us. Data transformations are one thing. For us, it’s the most important thing. Our data warehouse runs as a massive DAG of nightly batched transformations over app-generated data. We also use DAG-managing tools to call external APIs and get new data (eg for weather and geocoding) and batched ML training/inference pipelines too. Why something like Airflow? Dependencies are easier to manage reliably. If you have hundreds or thousands of nodes in your DAG, then it is a lifesaver to be able to easily 1) run many threads of independent nodes; 2) re-run on failures; and 3) find nodes impacted by failure.
  - jpollock 7y ago
    Sorry, I most definitely didn't want to make light of the problem! Pulling data from all the various teams' locally created data stores and external systems to push to analytics is definitely a large problem. I was trying to figure out if these are aimed at data transformation pipelines, or state management systems - I've got state management problems, not data transformation problems. Slightly different problems, but both fit with "Workflow".
- closeparen 7y ago
  No. The tasks inside a workflow, concretely, would be things like Spark job execution, SQL query execution, download a CSV from the internet to HDFS and load it as a Hive table, etc. Think fancy cron that deals correctly with failures in multistage processes. The number of pipelines and executions is a function of the complexity of your application, and invariant of the number of records being processed by the batch jobs within those workflows.

66 points by njanakiev 7y ago

How to manage it? In my opinion: Just switch to Devuan[1] - making it systemd-less - and use sysvinit scripts (these days there's support for parallelism and dependencies with insserv and startpar). [1] : https://www.devuan.org/

fignews 7y ago
Your reply has nothing to do with the article. Perhaps after reading the post your systemd hate will subdue some.
- 1996 7y ago
  For managing a complex set of daemons, systemd is the thing. I only wish that rules did integrate with system monitoring to have conditions (ex: something like StopWhenLoadThreshold_40=xx, StartWhenDiskFreeThreshold=99% )

villasv 7y ago

I have found that using KillMode=mixed is very useful when running Airflow with Celery workers. It allows the system to shudown gracefully coupled with TimeoutStopSec, so the workers will stop receiving new jobs but will finish their current jobs before exiting which is nice for auto scaling or spot instances on AWS (coupled with EC2 lifecycle hooks).

1996 7y ago
Interesting. Why do you care about graceful shutdown? Why not kill ASAP and restart? Do you have problems with load booming up sometimes?
- akramar 7y ago
  In some cases we need to do an update-deploy-restart while a DAG is still running (not even the one being updated). Then several minutes or hours later the child processes raise a segfault and the jobs they were working on fail, requiring restarting any of those jobs. I imagine a graceful shutdown would allow the job to finish up and the DAG to continue with the remaining jobs.

tyingq 7y ago

ETL is a funny space. At least in the "Enterprise" world, it's dominated by Ab Initio, which is crazy expensive. They seem to be coasting too, for quite some time. Their website is probably the most terrible site I've ever seen for an expensive piece of software. You can't even tell what it is, how to buy it, or even how to contact them. https://www.abinitio.com/en/ https://www.abinitio.com/en/

thenaturalist 7y ago
What's your source for the fact that Ab Initio is dominating? Other than that, there are a several other tools in the Enterprise Analytics space that fall into a similar pattern like Alteryx or Collibra. But from their perspective it makes perfect sense, I guess. When your sales is done by relationship building and there isn't much competition once you're in, there isn't really a need to boast a fancy website or make an effort. If anyone has a good resource on how enterprise IT procurement is done or the dynamics around it, I'd love to read up on that.
- tyingq 7y ago
  Just anectdotal experience. By "dominating", I mean in non-tech US Fortune 500 companies. They have very few employees, and the software is very expensive. "When your sales is done by relationship building and there isn't much competition once you're in, there isn't really a need to boast a fancy website or make an effort." Sure, but try and find a phone number or email address on their site. They've taken coasting to a whole new level.
- kfk 7y ago
  Well I have a lot of experience with IT procurement. IT is more about cost and risk mitigation than technology. The last thing IT wants is to try new software that might blow up in the hands of their unprepared outsourced helpdesk support people. So the absolute first thing you want to show is robust and cheap support or even better a way to make sure your product works with their helpdesk setup. Another thing is cost and how your product helps them run things with less money. They could not care less about the quality of the product because they will not be the ones using it. In general you sell by telling a story that fits with whatever story they are telling the business. So if IT is telling business cloud is the next thing and you sell on premise you won’t go anywhere.

EddieCPU 7y ago

I would have thought a scripting language would have been a better choice than unit files. The script engine would take care of verifying the script and starting the processes in the correct order and allowing/restricting access to other components. Thereby negating the need for such directives as After= or PrivateTmp=.

onefuncman 7y ago
Unit files are one of the things systemd got right. A scripting language isn't well suited for process management, which is a very well specified task. Now, if you mentioned https://ammonite.io/ https://ammonite.io/ you might have got my attention...

Peteris 7y ago

Side note: the best way to build Airflow pipelines is through Kedro https://github.com/quantumblacklabs/kedro-airflow https://github.com/quantumblacklabs/kedro-airflow.

Lucasoato 7y ago

(Sorry if it's a little off topic) We were close to adopt Airflow in our company but we were let down by a detail: the scheduler isn't natively in a high availability mode. There was an article from Clairvoyant about how to make it HA but it didn't look safe at all. That was a serious issue for us, and at the end we went for NiFi. Have anyone had this problem?

robbyt 7y ago
Use K8s to schedule the work.
- Filligree 7y ago
  And make sure you hire two engineers to keep it working.
smooc 7y ago
Airflow maintainer here If you have idempotent tasks, which is a best practice, it is possible to use Airflow in HA even in active/active. It might occasionally schedule a task twice, which should be caught but in any case mitigated by idempotency. If you are looking for more Enterprise support you can reach out to Astronomer (disclaimer: I'm a advisor to them) or use a Google cloud hosted version (Cloud Composer). Both are great products.

kfk 7y ago

Me and my team tried Airflow but found it didn’t fit well with our analytics work flow. For instance you must rewrite your jupyter notebook into an Airflow dag, doing basically the same work 2 times. We use dask and will soon deploy dask.distributed. I have yet to figure out where Airflow actually fits in the BI/data science architecture.

onefuncman 7y ago
I recall Jupyter being a parallel evolution alongside Airflow+Spark+Zeppelin (or similar mashup) and I think Jupyter has become "better".
- smooc 7y ago
  Airflow maintainer here Jupyter doesn't do scheduling and integrates pretty well with Airflow.
jmngomes 7y ago
Not sure what you mean by "BI/data science architecture" but Airflow is essentially a scheduler and orchestrator for data processing jobs. These activities are usually managed by cron and more often by advanced scheduler tools (depending on the vendor), so it's quite a core part of any architecture that needs to e.g. load/reload/refresh data periodically. If the requirement is simply to connect notebooks to a data lake, then the only scheduling required is to load the data lake, and something like Airflow may be overkill for this, depending on what/how the data is processed and loaded.
- kfk 7y ago
  I mean the same thing you mean. My issue with airflow is that it’s complicated and doesn’t adapt well to cloud computing. Dask runs on aws emr and eks, Kubernetes, etc.. Unfortunately orchestration is a lot more complicated than it looks. Parallel executions, retries, logs, status tracking, email notifications. Airflow doesn’t really tackle all orchestration work.
  - 7y ago

carlosf 7y ago

Good article! I actually like systemd, but nowadays I generally run stuff as containers, so there is less and less opportunity to use it. Here is my current setup for Airflow: - 1 container for the webserver - 1 container for the scheduler - 1 managed database (I use Postgres, it's a fairly small instance.) - 1 S3 bucket to deploy DAGs. I mount it on my containers using s3fs-fuse. - You can monitor the scheduler using a PID file, whereas the webserver can be monitored probing your admin URL. - Most configuration can be done using environment variables, which is perfect for containers. - I also configure DAG logs to be shipped to a S3 bucket.

jpollock 7y ago

Why are all these tools DAGs? I'm guessing I misunderstand what is meant by a "Workflow"? My assumption is that these are managing a state machine, where workflow is stand-in for "Business Process"? If it's doing that sort of job, I'd expect timers and loops? However, it seems these are aimed at data conversion pipelines?

bathtub365 7y ago
It has a cron scheduler
jpau 7y ago
Heyo! Data guy here. Airflow and its DAG-managing peers are important for us. Data transformations are one thing. For us, it’s the most important thing. Our data warehouse runs as a massive DAG of nightly batched transformations over app-generated data. We also use DAG-managing tools to call external APIs and get new data (eg for weather and geocoding) and batched ML training/inference pipelines too. Why something like Airflow? Dependencies are easier to manage reliably. If you have hundreds or thousands of nodes in your DAG, then it is a lifesaver to be able to easily 1) run many threads of independent nodes; 2) re-run on failures; and 3) find nodes impacted by failure.
- jpollock 7y ago
  Sorry, I most definitely didn't want to make light of the problem! Pulling data from all the various teams' locally created data stores and external systems to push to analytics is definitely a large problem. I was trying to figure out if these are aimed at data transformation pipelines, or state management systems - I've got state management problems, not data transformation problems. Slightly different problems, but both fit with "Workflow".
closeparen 7y ago
No. The tasks inside a workflow, concretely, would be things like Spark job execution, SQL query execution, download a CSV from the internet to HDFS and load it as a Hive table, etc. Think fancy cron that deals correctly with failures in multistage processes. The number of pipelines and executions is a function of the complexity of your application, and invariant of the number of records being processed by the batch jobs within those workflows.