4 ms·

For example: Managed Airflow Scheduler on AWS with "large" size costs $0.99/hour, or $8,672/year per instance. That's ~ $17,500 considering Airflow for at leas

by glogla 6y ago

For example: Managed Airflow Scheduler on AWS with "large" size costs $0.99/hour, or $8,672/year per instance. That's ~ $17,500 considering Airflow for at least non-prod and prod instances. Building it on your own on same size EC2 instance would cost $3,363/year for the EC2. Times two for two environments, let's say $6,700. $4,000 if you prepay the instance. That looks way cheaper, but then you have to do the engineering and the operational support yourself. If you consider just the engineering and assume engineer costs $50/hour and estimate this to initial three weeks of work and then 2.5 days / month for support (upgrades, tuning, ...) that's extra $4,000 upfront and $1,000/month. So on AWS you're at $17,500/year and on-prem you're at best $20,000 first year and $16,000 next years. So the AWS only comes a bit more expensive - but the math is tricky on several parts: - maybe you need 4 environments deployed instead of 2, which is more for AWS but not much more for engineering? - maybe there's less sustaining cost because you're ok with upgrading Airflow only once a quarter? - you probably already pay the engineers, so it's not an extra money cost, it's extra cost of them not working on other stuff - different boxes and budgets - maybe you're in part of a world where good devops engineer doesn't cost $50/hour but $15 hour - I'm ignoring cost of operational support, which can be a lot for on-prem if you need 24/7 - maybe you need 12+ Airflow instances thanks to your fragmented / federated IT and can share the engineering cost - etc, etc. So I think what OP was saying is that if AWS priced Managed Airflow at $0.5 per hour, it would be no brainer to use instead of build your own. The way it is, some customers will surely for their own Airflow instead, because the math favors it. Does that make sense?

bird_monster 6y ago
> That looks way cheaper, but then you have to do the engineering and the operational support yourself. In my experience, this is the piece that engineers rarely realize and that is actually one of the biggest factors in evaluating cloud providers vs. home-rolled. Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_. Valuable such that even if Airflow is cash-expensive, if using it allows your engineers to focus on building whatever makes _your business successful_, it is usually a much better idea to just use Airflow and keep moving. Clients usually will not care about whether you implemented your own version of an AWS product (unless that's your company's specific business). Clients will care about the features you ship. If you spent a ton of time re-inventing Airflow to save some cost, but then go bankrupt before you ever ship, rolling your own Airflow implementation clearly didn't save you anything.
- glogla 6y ago
  I agree. The only caveat is that this goes for founders or engineers who are financially tied with the company success. If the engineer just collects paycheck, they might prioritize fun - and I feel that might be behind a lot of the "reinventing the wheel" efforts you see in the industry. Or maybe I'm just cynical.
- tpxl 6y ago
  We used to have on-prem redis and a devops engineer to manage it, then we moved to redis in the cloud and had a devops engineer to manage it. Saying that in the cloud you don't need engineers to manage "operational support" is the biggest lie the cloud managed to sell.
  - glogla 6y ago
    If you just run redis on virtual server in cloud, you're not replacing the devops engineer who manages redis. You're replacing: - the network engineers who manage switches, firewalls, routers, nats and connectivity - the people who manage on-prem hardware, install new servers, replace failing servers, and switch broken disks for working ones and install base os using some ilom - the whole process for ordering that hardware and getting it delivered on time - including knowing when it's going to be necessary - if your on-prem had virtual servers, the people who manage and operate vmware - if your on-prem had SAN, the people who manage and operate that SAN (and buy new disks and take care of capacity planning) Some of those things you still have to do - for example configure firewall, or say "I want to take a snapshot of this virtual disk drive" but instead of doing a difficult technical thing, you can do it in web ui - but you still need to do what to do. And of course, if you never had SAN and virtual servers and two data center with your team managing the interconnect between private networks, there's a lot of stuff that the Cloud could give you that you probably don't need. Now if you move to managed redis, you're also replacing the person who installs and patches the linux the redis runs on, and the one who installs, backups and configures the redis. And you get the redis on button click, so if you suddenly need three more, you're also replacing the person who automates building redises. You are right, that it is not that the operational support is just gone. Some of it is gone. Some of it is replace by doing cloud stuff (like thinking of buying reserved instances instead of actual physical servers). Some of it is just more efficient. Now if any of this doesn't fit you're use case because you have too small or too large scale, then the Cloud is of course a bad idea.
hyperdimension 6y ago
> Managed Airflow Scheduler Management decided get rid of all the on-prem, capex-heavy HVAC systems (that are depreciating as-we-speak!) for budget-friendly cloud thermostats? Maybe they can send you the climate-controlled blown air on your office's Amazon Day. ;)

by glogla 6y ago

bird_monster 6y ago

> That looks way cheaper, but then you have to do the engineering and the operational support yourself. In my experience, this is the piece that engineers rarely realize and that is actually one of the biggest factors in evaluating cloud providers vs. home-rolled. Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_. Valuable such that even if Airflow is cash-expensive, if using it allows your engineers to focus on building whatever makes _your business successful_, it is usually a much better idea to just use Airflow and keep moving. Clients usually will not care about whether you implemented your own version of an AWS product (unless that's your company's specific business). Clients will care about the features you ship. If you spent a ton of time re-inventing Airflow to save some cost, but then go bankrupt before you ever ship, rolling your own Airflow implementation clearly didn't save you anything.

glogla 6y ago
I agree. The only caveat is that this goes for founders or engineers who are financially tied with the company success. If the engineer just collects paycheck, they might prioritize fun - and I feel that might be behind a lot of the "reinventing the wheel" efforts you see in the industry. Or maybe I'm just cynical.
tpxl 6y ago
We used to have on-prem redis and a devops engineer to manage it, then we moved to redis in the cloud and had a devops engineer to manage it. Saying that in the cloud you don't need engineers to manage "operational support" is the biggest lie the cloud managed to sell.
- glogla 6y ago
  If you just run redis on virtual server in cloud, you're not replacing the devops engineer who manages redis. You're replacing: - the network engineers who manage switches, firewalls, routers, nats and connectivity - the people who manage on-prem hardware, install new servers, replace failing servers, and switch broken disks for working ones and install base os using some ilom - the whole process for ordering that hardware and getting it delivered on time - including knowing when it's going to be necessary - if your on-prem had virtual servers, the people who manage and operate vmware - if your on-prem had SAN, the people who manage and operate that SAN (and buy new disks and take care of capacity planning) Some of those things you still have to do - for example configure firewall, or say "I want to take a snapshot of this virtual disk drive" but instead of doing a difficult technical thing, you can do it in web ui - but you still need to do what to do. And of course, if you never had SAN and virtual servers and two data center with your team managing the interconnect between private networks, there's a lot of stuff that the Cloud could give you that you probably don't need. Now if you move to managed redis, you're also replacing the person who installs and patches the linux the redis runs on, and the one who installs, backups and configures the redis. And you get the redis on button click, so if you suddenly need three more, you're also replacing the person who automates building redises. You are right, that it is not that the operational support is just gone. Some of it is gone. Some of it is replace by doing cloud stuff (like thinking of buying reserved instances instead of actual physical servers). Some of it is just more efficient. Now if any of this doesn't fit you're use case because you have too small or too large scale, then the Cloud is of course a bad idea.

hyperdimension 6y ago

> Managed Airflow Scheduler Management decided get rid of all the on-prem, capex-heavy HVAC systems (that are depreciating as-we-speak!) for budget-friendly cloud thermostats? Maybe they can send you the climate-controlled blown air on your office's Amazon Day. ;)