6 ms·

Unless you have extremely complex dependency graphs, I really don't think airflow is worth it. It's very easy to end up essentially writing an "orchestrator" us

Unless you have extremely complex dependency graphs, I really don't think airflow is worth it. It's very easy to end up essentially writing an "orchestrator" using airflow, it allows for very flexible low level operations. The added complexity has minimal benefit, and like something like apache spark, what looks simple becomes hard to reason about in real world scenarios. You need to understand how it works under the hood, and get the best practices right. As mentioned elsewhere, AWS step functions are really the best in orchestration.

fmakunbound 4y ago
Definitely this. Our teams have many 1 or 2 step DAGs that are idempotent. They could have been lambdas and they're already pulling from SQS already. It could be just my misfortune, but in AWS, MWAA is kind of janky. It's difficult to track down problems in the logs (task failures look fine there) and the Airflow UI is randomly unavailable ("document returns empty", "connection reset" kind of things).
- mywittyname 4y ago
  Lambdas have resource constraints that Airflow DAGs don't. Most notably, Airflow DAGs can run for any arbitrary length of time. And the local storage attached to the Airflow cluster is actual disk space, and not just a fake in-memory disk, making it possible process files larger than the amount of memory allocated to the DAG. There's certainly some functionality overlap, but I don't see Lambda and Airflow as competitors. Each has capabilities that the other doesn't.
  - xtracto 4y ago
    I remember reading that you can attach EFS to lambdas. That would solve some of the storage issues.
arinlen 4y ago
> As mentioned elsewhere, AWS step functions are really the best in orchestration. AWS Step Functions is a proprietary service provided exclusively by AWS, which reacts to events from AWS services and calls AWS Lambdas. Unless you're already neck-deep in AWS, and are already comfortable paying through the nose for trivial things you can run yourself for free, it's hardly appropriate to even bring up AWS Step Functions as a valid alternative. For instance, Shopify's articles explicitly mention they are running their services in Google Cloud. Would it be appropriate to tell them to just migrate their whole services to AWS just because you like AWS Step Functions?
- literallyWTF 4y ago
  This is another symptom of a person who doesn’t know what they’re talking about really. It’s like those stackoverflow answers that tell the user to stop using PHP and rewrite it in Python or something.
- parsnips 4y ago
  https://github.com/checkr/states-language-cadence https://github.com/checkr/states-language-cadence allows you to define workflows in states language over cadence.
- riku_iki 4y ago
  > already comfortable paying through the nose for trivial things you can run yourself for free But fault tolerant workflow engine is not trivial thing, it may cost you many engineer hours to build, monitor and maintain it, so outsourcing it to someone else is totally viable solution.
  - arinlen 4y ago
    > But fault tolerant workflow engine is not trivial thing, The complexity and risk of migrating cloud providers eclipses whatever problem you assign to "fault tolerant workflow engines". Any mention of AWS Step Functions makes absolutely no sense at all and reads at best like a non-sequitur.
    - thorum 4y ago
      I read it as a comment on the UX / developer experience, which can superior with Step Functions vs competition regardless of whether Step Functions is an appropriate (or even physically possible) option for non-AWS projects.
- Jugurtha 4y ago
  That was one the reasons we do "bring your own compute" with https://iko.ai https://iko.ai so people who already have a billing account on AWS, GCP, Azure, DigitalOcean, can just get the config for their Kubernetes clusters and link them to iko.ai and their machine learning workloads will run on whichever cluster they select. If you get a good deal from one cloud provider, you can get started quickly. It's useful even for individuals such as students who get free credits from these providers: create a cluster and you're up and running in no time. Our rationale was that we didn't wanted to be tied to one cloud provider.
nojito 4y ago
I’ll never understand why individuals always default to cloud offerings when they are extremely expensive compared to a dedicated tool.
- literallyWTF 4y ago
  Because they don’t know what they’re doing and aren’t the ones paying the bill. “Oh I have to learn how to use and setup this tool? I think I’ll just pay the equivalent salaries and be locked in…”
  - saimiam 4y ago
    I'm very much paying my own cloud bills but there is no chance I would be able to orchestrate some of the workflows I want to orchestrate if it were not for Step Functions. For a one person shop like me, AWS is a force multiplier. With it, I can do (say) 30% of what a dedicated engineer in a specific role could do. Without it, I'd be doing 0%. I really like this tradeoff for my particular situation.
    - deleted 4y ago
      [deleted]
  - waynesonfire 4y ago
    You're going to pay the bill regardless, whether the employee is hired by your team or hired by the cloud vendor. I don't know how management works through this math, maybe managing people gets exhausting and they just want to out-source it so leadership doesn't have to deal with it and then they can just focus on the core product. And the above "I don't want to deal with it" reason isn't spoken of, the more more commonly touted benefit is cloud's "flexibility". Sure, but this is actually _really_ expensive. Every cloud migration effort I've experienced is only just worthwhile to begin to talk about because the costs are based on long-term contracts of cloud resources, not the per-hour fees. Nice flexibility. With that said, the cloud may be a good place for prototyping where the infrastructure isn't the core value add and it's uncertain. A start-up is a prototype and so here we are. But, for an established company to migrate to the cloud and fire the staff that's maintaining the on premise resources.. I'm skeptical. More than likely, this leads to maintaining both cloud and on premise resources, not firing anyone, and thus, actually increasing costs for an uncomfortably long time. And for the folks on the ground, who don't pay the bills, the increase of accidental complexity is rather painful.
pyrophane 4y ago
> As mentioned elsewhere, AWS step functions are really the best in orchestration. Why? Where else is this mentioned?
thefourthchime 4y ago
> AWS step functions are really the best in orchestration. At our company, AWS Step is a disaster. You're effectively writing code in JSON/YAML. Anything beyond very simple steps becomes 2 pages of YAML that's very hard to read or write. There is no way to debug, polling is mostly unusable. Changes need to be deployed with CF which can take forever, or worse hang. It's the most one of the most annoying technologies I've used in my 20+ years of engineering.
- ledauphin 4y ago
  yeah, to me the idea that anyone would want to write orchestration in something that isn't a programming language where you can write tests in the same language is just so backwards.
- quantisan 4y ago
  I feel you. That's why we wrote a little library on top of SFN so that we can program SFN with Clojure instead of YAML https://github.com/Motiva-AI/stepwise https://github.com/Motiva-AI/stepwise. Application code sits with SFN definition and SFN Tasks are automatically integrated as polling Activities from Clojure code. Thoughtworks made a case for this distinction in https://martinfowler.com/articles/cant-buy-integration.html#UseCommercialIntegrationToolsToSimplifyImplementationConcerns https://martinfowler.com/articles/cant-buy-integration.html#...
tootie 4y ago
I read some of these of massively complex data architecture posts and I almost always come away asking "What the hell is this for?" I know Shopify is a huge business but I see this kind of engineering complexity and all I think is it has to cost tens of millions to build and operate and how could they possibly be getting ROI. There are ten boxes on that diagram and none of them have a user interface for anyone except other developers.
- generalpf 4y ago
  A lot of times this is used for data warehousing so product managers and otherwise can query the database of one app joined with another, especially in an environment with microservices. You might join a table containing orders with another table that was from a totally different DB, like payments, to find out which kinds of items are best to offer BNPL or something. The author also mentions that it’s used for machine learning models which will ultimately feed back into Shopify’s front end, for instance.
  - tootie 4y ago
    I know what a data warehouse is for but this whole situation has doesn't even cover the reporting system which itself is a level removed from any actual product decisions and whether those decisions result in incremental revenue to justify this cost. My company is a fraction the size of Shopify but we have a robust data pipeline and reporting on millions of users run by two people and off the shelf software.
nooorofe 4y ago
very basic use cases: * run job for given reporting_period * backfill data for reporting_period between N and M * lookup failed jobs Nothing in Databricks supports that.
- maslam 4y ago
  Databricks PM here. Backfills are on our roadmap We are previewing looking up failed jobs soon. Email me at bilal dot aslam at Databricks dot com if you want more info

fmakunbound 4y ago

Definitely this. Our teams have many 1 or 2 step DAGs that are idempotent. They could have been lambdas and they're already pulling from SQS already. It could be just my misfortune, but in AWS, MWAA is kind of janky. It's difficult to track down problems in the logs (task failures look fine there) and the Airflow UI is randomly unavailable ("document returns empty", "connection reset" kind of things).

mywittyname 4y ago
Lambdas have resource constraints that Airflow DAGs don't. Most notably, Airflow DAGs can run for any arbitrary length of time. And the local storage attached to the Airflow cluster is actual disk space, and not just a fake in-memory disk, making it possible process files larger than the amount of memory allocated to the DAG. There's certainly some functionality overlap, but I don't see Lambda and Airflow as competitors. Each has capabilities that the other doesn't.
- xtracto 4y ago
  I remember reading that you can attach EFS to lambdas. That would solve some of the storage issues.

arinlen 4y ago

> As mentioned elsewhere, AWS step functions are really the best in orchestration. AWS Step Functions is a proprietary service provided exclusively by AWS, which reacts to events from AWS services and calls AWS Lambdas. Unless you're already neck-deep in AWS, and are already comfortable paying through the nose for trivial things you can run yourself for free, it's hardly appropriate to even bring up AWS Step Functions as a valid alternative. For instance, Shopify's articles explicitly mention they are running their services in Google Cloud. Would it be appropriate to tell them to just migrate their whole services to AWS just because you like AWS Step Functions?

literallyWTF 4y ago
This is another symptom of a person who doesn’t know what they’re talking about really. It’s like those stackoverflow answers that tell the user to stop using PHP and rewrite it in Python or something.
parsnips 4y ago
https://github.com/checkr/states-language-cadence https://github.com/checkr/states-language-cadence allows you to define workflows in states language over cadence.
riku_iki 4y ago
> already comfortable paying through the nose for trivial things you can run yourself for free But fault tolerant workflow engine is not trivial thing, it may cost you many engineer hours to build, monitor and maintain it, so outsourcing it to someone else is totally viable solution.
- arinlen 4y ago
  > But fault tolerant workflow engine is not trivial thing, The complexity and risk of migrating cloud providers eclipses whatever problem you assign to "fault tolerant workflow engines". Any mention of AWS Step Functions makes absolutely no sense at all and reads at best like a non-sequitur.
  - thorum 4y ago
    I read it as a comment on the UX / developer experience, which can superior with Step Functions vs competition regardless of whether Step Functions is an appropriate (or even physically possible) option for non-AWS projects.
Jugurtha 4y ago
That was one the reasons we do "bring your own compute" with https://iko.ai https://iko.ai so people who already have a billing account on AWS, GCP, Azure, DigitalOcean, can just get the config for their Kubernetes clusters and link them to iko.ai and their machine learning workloads will run on whichever cluster they select. If you get a good deal from one cloud provider, you can get started quickly. It's useful even for individuals such as students who get free credits from these providers: create a cluster and you're up and running in no time. Our rationale was that we didn't wanted to be tied to one cloud provider.

nojito 4y ago

I’ll never understand why individuals always default to cloud offerings when they are extremely expensive compared to a dedicated tool.

literallyWTF 4y ago
Because they don’t know what they’re doing and aren’t the ones paying the bill. “Oh I have to learn how to use and setup this tool? I think I’ll just pay the equivalent salaries and be locked in…”
- saimiam 4y ago
  I'm very much paying my own cloud bills but there is no chance I would be able to orchestrate some of the workflows I want to orchestrate if it were not for Step Functions. For a one person shop like me, AWS is a force multiplier. With it, I can do (say) 30% of what a dedicated engineer in a specific role could do. Without it, I'd be doing 0%. I really like this tradeoff for my particular situation.
  - deleted 4y ago
    [deleted]
- waynesonfire 4y ago
  You're going to pay the bill regardless, whether the employee is hired by your team or hired by the cloud vendor. I don't know how management works through this math, maybe managing people gets exhausting and they just want to out-source it so leadership doesn't have to deal with it and then they can just focus on the core product. And the above "I don't want to deal with it" reason isn't spoken of, the more more commonly touted benefit is cloud's "flexibility". Sure, but this is actually _really_ expensive. Every cloud migration effort I've experienced is only just worthwhile to begin to talk about because the costs are based on long-term contracts of cloud resources, not the per-hour fees. Nice flexibility. With that said, the cloud may be a good place for prototyping where the infrastructure isn't the core value add and it's uncertain. A start-up is a prototype and so here we are. But, for an established company to migrate to the cloud and fire the staff that's maintaining the on premise resources.. I'm skeptical. More than likely, this leads to maintaining both cloud and on premise resources, not firing anyone, and thus, actually increasing costs for an uncomfortably long time. And for the folks on the ground, who don't pay the bills, the increase of accidental complexity is rather painful.

pyrophane 4y ago

> As mentioned elsewhere, AWS step functions are really the best in orchestration. Why? Where else is this mentioned?

thefourthchime 4y ago

> AWS step functions are really the best in orchestration. At our company, AWS Step is a disaster. You're effectively writing code in JSON/YAML. Anything beyond very simple steps becomes 2 pages of YAML that's very hard to read or write. There is no way to debug, polling is mostly unusable. Changes need to be deployed with CF which can take forever, or worse hang. It's the most one of the most annoying technologies I've used in my 20+ years of engineering.

ledauphin 4y ago
yeah, to me the idea that anyone would want to write orchestration in something that isn't a programming language where you can write tests in the same language is just so backwards.
quantisan 4y ago
I feel you. That's why we wrote a little library on top of SFN so that we can program SFN with Clojure instead of YAML https://github.com/Motiva-AI/stepwise https://github.com/Motiva-AI/stepwise. Application code sits with SFN definition and SFN Tasks are automatically integrated as polling Activities from Clojure code. Thoughtworks made a case for this distinction in https://martinfowler.com/articles/cant-buy-integration.html#UseCommercialIntegrationToolsToSimplifyImplementationConcerns https://martinfowler.com/articles/cant-buy-integration.html#...

tootie 4y ago

I read some of these of massively complex data architecture posts and I almost always come away asking "What the hell is this for?" I know Shopify is a huge business but I see this kind of engineering complexity and all I think is it has to cost tens of millions to build and operate and how could they possibly be getting ROI. There are ten boxes on that diagram and none of them have a user interface for anyone except other developers.

generalpf 4y ago
A lot of times this is used for data warehousing so product managers and otherwise can query the database of one app joined with another, especially in an environment with microservices. You might join a table containing orders with another table that was from a totally different DB, like payments, to find out which kinds of items are best to offer BNPL or something. The author also mentions that it’s used for machine learning models which will ultimately feed back into Shopify’s front end, for instance.
- tootie 4y ago
  I know what a data warehouse is for but this whole situation has doesn't even cover the reporting system which itself is a level removed from any actual product decisions and whether those decisions result in incremental revenue to justify this cost. My company is a fraction the size of Shopify but we have a robust data pipeline and reporting on millions of users run by two people and off the shelf software.

nooorofe 4y ago

very basic use cases: * run job for given reporting_period * backfill data for reporting_period between N and M * lookup failed jobs Nothing in Databricks supports that.

maslam 4y ago
Databricks PM here. Backfills are on our roadmap We are previewing looking up failed jobs soon. Email me at bilal dot aslam at Databricks dot com if you want more info