3 ms·
The author has given 1 specific example: "Another time our Data Science team told us that they needed an orchestration tool for their data pipelines. I steer
by bg24 4y ago
The author has given 1 specific example:
"Another time our Data Science team told us that they needed an orchestration tool for their data pipelines. I steered the selection process towards Argo Workflows (which runs inside Kubernetes) instead of Prefect (a SaaS offering), which they already used for a PoC. There were all sorts of justifications in my head for this decision. Unfortunately, they were all based on premature optimisations. In the end, our team needed to build a new set of Terraform and Helm charts to automate the deployment of Argo Workflows and integrate it into our SSO etc. I regret this decision. I think we lost weeks or even months shipping something to end-users because of this decision. Premature optimisation!"
There is absolutely NO issue in adopting kubernetes, and then using SAAS for everything possible. In the above case, if you had a data pipeline proficient engineer in the team, you would not spend months to set up one. It comes down to making decisions based on your current situation (business urgency, team skills, future plans).