4 ms·

I think the cloud can reduce the operations team even at their scale if serverless is used on GCP. Many of the serverless offerings on GCP just get it right. I

I think the cloud can reduce the operations team even at their scale if serverless is used on GCP. Many of the serverless offerings on GCP just get it right. I have never had to think about machine sizes, clusters, etc when using BigQuery. My team and I focus on the SQL. Cloud Run gives you a web or API server without having to really think about machines other than number of cores and memory size. I am currently designing a data pipeline architecture that is 100% serverless to move away from Airflow/Composer. Why? Because my team and I can stay focused on the code and logic of moving data around instead of the operations of managing a managed product. Would this work on AWS or Azure? Theoretically yes, but I personally think their serverless offerings are hard to use and don't make things easier.

flyinglizard 4y ago
Azure Functions is as easy to use as writing an "Hello World" console application. I have a few services running like clockwork for a few years now, very happy with this service overall. The one thing that complicates matters is Microsoft's eagerness to deprecate libraries, frameworks and best practices every now and then.
icedchai 4y ago
The developer experience for AWS serverless offerings, like Lambda, is awful. You can spend hours going down a rabbit hole of API gateway configuration, IAM roles, security policies, VPC endpoints, etc. that gives you very little in terms of useful error messages when it's not quite configured right. But, yeah, once you do set it up, the per-request cost is quite cheap. Still, I think if you consider the lost developer productivity dealing with slow deploys and environment/configuration issues, the costs are much, much higher.