7 ms·

LLM Workflows then Agents: Getting Started with Apache Airflow

ringtailedlemur 2y ago
[dead]
datadrivenangel 2y ago
I'm looking into using LLM calls inside SQL Triggers to make agents / 'agentic' workflows. Having LLM powered workflows can get you powerful results and are basically the equivalent of 'spinning up' an agent.
- jackthetab 2y ago
  Know of any online examples of the same?
- fancy_pantser 2y ago
  Been having a great time with postgresml for this exact kind of thing. If you don't need a complex DAG but have a simple pipeline or work queue that can be easily represented in postgres anyway, it's very straightforward to work with and nicely encapsulates all of your processing (traditional data munging and LLM calls) together with a modest extension of a familiar system.
- jumski 2y ago
  I'm not sure if this would scratch your itch, but I'm building a Postgres-native workflow engine that separates orchestration from execution. I want to be able to start my flows from within db triggers. It just exposes set of functions to propagate the DAG through states and queues tasks for a task worker to perform the actual work and acknowledge completion or failure back to the sql orchestrator. I was working on it for last few months and it will be ready in upcoming weeks, first version is dedicated to work on Supabase but I plan to make it agnostic. If you want to learn more, check out the SQL Core readme which explains the whole concept (https://github.com/pgflow-dev/pgflow/tree/main/pkgs/core#readme https://github.com/pgflow-dev/pgflow/tree/main/pkgs/core#rea...) or my Twitter for updates and some demos (https://x.com/pgflow_dev https://x.com/pgflow_dev).
itsallrelative 2y ago
Truthfully have been a little skeptical of how many workloads will actually need “agents” vs doing something totally deterministic with a little LLM augmentation. Seems like I’m not the only one that thinks the latter works a lot of the time!
- petesergeant 2y ago
  Yes! I just wrote an article on this: https://sgnt.ai/p/hell-out-of-llms/ https://sgnt.ai/p/hell-out-of-llms/
mushufasa 2y ago
this is really cool! That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton Most of these frameworks now have docs / plugins / sister libraries geared around AI agents It would be really helpful to read a good technical blog doing a landscape of design patterns in these different approaches, and thoughts on how to fit things together well into a pipeline given various quirks of LLMs (e.g. nondeterminism). This page is a good start, even if it is written as an airflow-specific how-to!
- Hasz 2y ago
  Dated doesn’t mean bad (usually the opposite in my experience!) What issues do you have with Airflow?
  - gre 2y ago
    Here's my problems with MWAA (amazon hosted airflow.) I have about 100 dags which maxes out the scheduler thread. Airflow parses all the files every minute so it's always parsing around 94% cpu. I could run a second scheduler thread if I coordinate with my SRE team and get the terraform deployed...it's really tedious. Related possibly, my dags get kill -9 for no apparent reason. The RAM usage is not that high, maybe 2gb out of 8gb system RAM in use. No reason is given in the logs. I am trying to switch to dagster, not because it's awesome, but because it hasn't crashed randomly on me.
    - mblast311 2y ago
      MWAA is hot garbage. I had similar issues and switched to running it on EKS instead.
    - alittletooraph2 2y ago
      This feels like an MWAA issue but I understand how that often gets conflated with it being an Airflow issue.
deleted 2y ago
[deleted]
falcor84 2y ago
This is about workflows that use AI, but it lead me to actually think of the inverse - has anyone experimented with AI agents defining and iterating upon long-running workflows?
ldjkfkdsjnv 2y ago
Extremely bearish on existing tools solving agentic workflows well. If anyone, it will be temporal. Airflow and the like simply were not designed for high dynamic execution, and so have all sorts of annoyances that will make them lose.
- alittletooraph2 2y ago
  Temporal’s great! That being said, there is something about being able to orchestrate LLMs and agents using what many already use to orchestrate their data workflows because there’s already proven out reliability, scalability, observability, etc. I’m sure there are boundary conditions for really advanced agentic workflows though…
  - acchow 2y ago
    Temporal is for a static graph with idempotent nodes. Powerful LLM workflows don’t fit this model.
    - ldjkfkdsjnv 2y ago
      Temporal is absolutely not for a static graph, idempotent nodes yes. Please explain your argument more
      - tomwheeler 2y ago
        > Temporal is absolutely not for a static graph I'd clarify this to say "Temporal is absolutely not limited to a static graph." It can certainly handle a static graph, but it can also handle a dynamic one. Here is an example in Go (https://github.com/temporalio/samples-go/tree/main/choice-multi https://github.com/temporalio/samples-go/tree/main/choice-mu...), there are similar ones for other languages. I think the confusion might stem from the determinism requirement in Temporal (and other replay-based Durable Execution platforms). It's not the Workflow Definition (i.e., the code) that must be deterministic, it's the Workflow Execution (i.e., a specific running instance of that code) that must be deterministic. Each running instance is allowed to take a different path through that code, so long as it does so consistently when executed with the same input.
drdaeman 2y ago
I'm sorry, I don't really know Airflow, but what's the point of `@task.agent`, as compared to plain old `return my_agent.run_sync(...)`? To me it feels like a more restrictive[1], and possibly less intuitive[2] API. [1]: Limited to what decorator arguments can do. I suspect it could become an issue with `@task.branch` if some post-processing would be needed to adjust for smaller models' finickinesses. [2]: As the final step is described at the top of the function.
- jlaneve 2y ago
  Disclaimer: author of the SDK here. It is _potentially_ more restrictive than writing pure Python functions, but the plus side is that we can interject certain Airflow-specific features into how the agent runs. And this isn't mean for someone who knows agents inside & out / wants the low-level customizability. The best example of this today is log groups: Airflow lets you log things out as part of a "group" which has some UI abstractions to make it easier. This SDK takes the raw agent tool calls and turns them each into a log group, so you can see a) at a high level what the agent is doing, and b) drill down into a specific tool call to understand what's happening within the tool call. To your point about the `@task.llm_branch`, the SDK & Pydantic AI (which the SDK uses under the hood) will re-prompt the LLM up to a certain number of attempts if it receives output that isn't the name of a downstream task. So there shouldn't be much finickiness.
nikolayasdf123 2y ago
nice. airflow is a good fit for this
curtisszmania 2y ago
[dead]
greatgib 2y ago
Decorators in the usage example looks useless, and more to show off than being a real convenience. In real life program, I don't think that you will have hundreds of calls to LLM or agent in your app so much that you have any code gains to decorator but at the opposite the decorator will make it very hard to have parametric values or values not hard coded but from config that you don't set up upfront at application startup like globals. That is a bad practice...
- jlaneve 2y ago
  Disclaimer: author of the SDK here. Airflow actually uses decorators to indicate something is an explicit task in a data pipeline vs just a utility function, so this follows that pattern! It also uses an "operator" under the hood (Airflow's term for a pre-built, parameterized task) which can be subclassed and customized if you want to do any customization.
jumski 2y ago
Nice to see some workflow engine action on Hacker News! :-) I'm currently building pgflow, which is a simple, postgres-first engine that uses task queues to perform real work. Have explicit DAG approach, strong typesafety, nice DSL in TypeScript and a dedicated task queue worker that allows it to run solely on Supabase without any external tools. I'm super close to the alpha release, if you guys want more info, check out the readme for SQL core (https://github.com/pgflow-dev/pgflow/tree/main/pkgs/core#readme https://github.com/pgflow-dev/pgflow/tree/main/pkgs/core#rea...) or my Twitter (https://x.com/pgflow_dev https://x.com/pgflow_dev). Hope that grabs someone attention :-) Cheers
- CjHuber 2y ago
  Exactly what I was looking for without even knowing it :) EDIT: well I knew I need smt like this, but I thought I'd had to build a very rudimentary version myself. Thank you for saving me tons of time in my project
  - jumski 2y ago
    Thanks! That's the reason I'm building it - I needed something like this but there was nothing abailable. I'm very close to releasing an alpha, will post here when ready!

[dead]

I'm looking into using LLM calls inside SQL Triggers to make agents / 'agentic' workflows. Having LLM powered workflows can get you powerful results and are basically the equivalent of 'spinning up' an agent.

jackthetab 2y ago
Know of any online examples of the same?
fancy_pantser 2y ago
Been having a great time with postgresml for this exact kind of thing. If you don't need a complex DAG but have a simple pipeline or work queue that can be easily represented in postgres anyway, it's very straightforward to work with and nicely encapsulates all of your processing (traditional data munging and LLM calls) together with a modest extension of a familiar system.
jumski 2y ago
I'm not sure if this would scratch your itch, but I'm building a Postgres-native workflow engine that separates orchestration from execution. I want to be able to start my flows from within db triggers. It just exposes set of functions to propagate the DAG through states and queues tasks for a task worker to perform the actual work and acknowledge completion or failure back to the sql orchestrator. I was working on it for last few months and it will be ready in upcoming weeks, first version is dedicated to work on Supabase but I plan to make it agnostic. If you want to learn more, check out the SQL Core readme which explains the whole concept (https://github.com/pgflow-dev/pgflow/tree/main/pkgs/core#readme https://github.com/pgflow-dev/pgflow/tree/main/pkgs/core#rea...) or my Twitter for updates and some demos (https://x.com/pgflow_dev https://x.com/pgflow_dev).

itsallrelative 2y ago

Truthfully have been a little skeptical of how many workloads will actually need “agents” vs doing something totally deterministic with a little LLM augmentation. Seems like I’m not the only one that thinks the latter works a lot of the time!

petesergeant 2y ago
Yes! I just wrote an article on this: https://sgnt.ai/p/hell-out-of-llms/ https://sgnt.ai/p/hell-out-of-llms/

mushufasa 2y ago

this is really cool! That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton Most of these frameworks now have docs / plugins / sister libraries geared around AI agents It would be really helpful to read a good technical blog doing a landscape of design patterns in these different approaches, and thoughts on how to fit things together well into a pipeline given various quirks of LLMs (e.g. nondeterminism). This page is a good start, even if it is written as an airflow-specific how-to!

Hasz 2y ago
Dated doesn’t mean bad (usually the opposite in my experience!) What issues do you have with Airflow?
- gre 2y ago
  Here's my problems with MWAA (amazon hosted airflow.) I have about 100 dags which maxes out the scheduler thread. Airflow parses all the files every minute so it's always parsing around 94% cpu. I could run a second scheduler thread if I coordinate with my SRE team and get the terraform deployed...it's really tedious. Related possibly, my dags get kill -9 for no apparent reason. The RAM usage is not that high, maybe 2gb out of 8gb system RAM in use. No reason is given in the logs. I am trying to switch to dagster, not because it's awesome, but because it hasn't crashed randomly on me.
  - mblast311 2y ago
    MWAA is hot garbage. I had similar issues and switched to running it on EKS instead.
  - alittletooraph2 2y ago
    This feels like an MWAA issue but I understand how that often gets conflated with it being an Airflow issue.

deleted 2y ago

[deleted]

falcor84 2y ago

This is about workflows that use AI, but it lead me to actually think of the inverse - has anyone experimented with AI agents defining and iterating upon long-running workflows?

ldjkfkdsjnv 2y ago

Extremely bearish on existing tools solving agentic workflows well. If anyone, it will be temporal. Airflow and the like simply were not designed for high dynamic execution, and so have all sorts of annoyances that will make them lose.

alittletooraph2 2y ago
Temporal’s great! That being said, there is something about being able to orchestrate LLMs and agents using what many already use to orchestrate their data workflows because there’s already proven out reliability, scalability, observability, etc. I’m sure there are boundary conditions for really advanced agentic workflows though…
- acchow 2y ago
  Temporal is for a static graph with idempotent nodes. Powerful LLM workflows don’t fit this model.
  - ldjkfkdsjnv 2y ago
    Temporal is absolutely not for a static graph, idempotent nodes yes. Please explain your argument more
    - tomwheeler 2y ago
      > Temporal is absolutely not for a static graph I'd clarify this to say "Temporal is absolutely not limited to a static graph." It can certainly handle a static graph, but it can also handle a dynamic one. Here is an example in Go (https://github.com/temporalio/samples-go/tree/main/choice-multi https://github.com/temporalio/samples-go/tree/main/choice-mu...), there are similar ones for other languages. I think the confusion might stem from the determinism requirement in Temporal (and other replay-based Durable Execution platforms). It's not the Workflow Definition (i.e., the code) that must be deterministic, it's the Workflow Execution (i.e., a specific running instance of that code) that must be deterministic. Each running instance is allowed to take a different path through that code, so long as it does so consistently when executed with the same input.

drdaeman 2y ago

I'm sorry, I don't really know Airflow, but what's the point of `@task.agent`, as compared to plain old `return my_agent.run_sync(...)`? To me it feels like a more restrictive[1], and possibly less intuitive[2] API. [1]: Limited to what decorator arguments can do. I suspect it could become an issue with `@task.branch` if some post-processing would be needed to adjust for smaller models' finickinesses. [2]: As the final step is described at the top of the function.

jlaneve 2y ago
Disclaimer: author of the SDK here. It is _potentially_ more restrictive than writing pure Python functions, but the plus side is that we can interject certain Airflow-specific features into how the agent runs. And this isn't mean for someone who knows agents inside & out / wants the low-level customizability. The best example of this today is log groups: Airflow lets you log things out as part of a "group" which has some UI abstractions to make it easier. This SDK takes the raw agent tool calls and turns them each into a log group, so you can see a) at a high level what the agent is doing, and b) drill down into a specific tool call to understand what's happening within the tool call. To your point about the `@task.llm_branch`, the SDK & Pydantic AI (which the SDK uses under the hood) will re-prompt the LLM up to a certain number of attempts if it receives output that isn't the name of a downstream task. So there shouldn't be much finickiness.

nikolayasdf123 2y ago

nice. airflow is a good fit for this

curtisszmania 2y ago

[dead]

greatgib 2y ago

Decorators in the usage example looks useless, and more to show off than being a real convenience. In real life program, I don't think that you will have hundreds of calls to LLM or agent in your app so much that you have any code gains to decorator but at the opposite the decorator will make it very hard to have parametric values or values not hard coded but from config that you don't set up upfront at application startup like globals. That is a bad practice...

jlaneve 2y ago
Disclaimer: author of the SDK here. Airflow actually uses decorators to indicate something is an explicit task in a data pipeline vs just a utility function, so this follows that pattern! It also uses an "operator" under the hood (Airflow's term for a pre-built, parameterized task) which can be subclassed and customized if you want to do any customization.

jumski 2y ago

Nice to see some workflow engine action on Hacker News! :-) I'm currently building pgflow, which is a simple, postgres-first engine that uses task queues to perform real work. Have explicit DAG approach, strong typesafety, nice DSL in TypeScript and a dedicated task queue worker that allows it to run solely on Supabase without any external tools. I'm super close to the alpha release, if you guys want more info, check out the readme for SQL core (https://github.com/pgflow-dev/pgflow/tree/main/pkgs/core#readme https://github.com/pgflow-dev/pgflow/tree/main/pkgs/core#rea...) or my Twitter (https://x.com/pgflow_dev https://x.com/pgflow_dev). Hope that grabs someone attention :-) Cheers

CjHuber 2y ago
Exactly what I was looking for without even knowing it :) EDIT: well I knew I need smt like this, but I thought I'd had to build a very rudimentary version myself. Thank you for saving me tons of time in my project
- jumski 2y ago
  Thanks! That's the reason I'm building it - I needed something like this but there was nothing abailable. I'm very close to releasing an alpha, will post here when ready!