3 ms·

Thanks for the feedback! We aren't trying to invent another infrastructure provisioning language, and I agree that Terraform would be the right choice if that w

Thanks for the feedback! We aren't trying to invent another infrastructure provisioning language, and I agree that Terraform would be the right choice if that was the case. Our YAML is more similar to the configuration of deployment tools like Netlify or CircleCI. We use CloudFormation and Kubernetes under the hood but our goal is to provide a much higher abstraction for data scientists / ML engineers.

sandGorgon 7y ago
Not entirely. The abstractions are different between infrastructure deployment.... versus configuration yml of circleci. The declaration of deployment state is a very BIG and hard problem that has had millions of collective man hours spent over decades. I urge you not to think of it as a simple configuration. In fact it is so hard that AWS has to build a new language on top of typescript ..versus cloudformation templates that it already had. https://docs.aws.amazon.com/cdk/latest/guide/home.html https://docs.aws.amazon.com/cdk/latest/guide/home.html What you are building makes sense - I would drop cloudformation and surface Terraform right till the top. So the way to use your tool is to install and use a new Terraform "provider".
- ospillinger 7y ago
  The Terraform provider idea is interesting, I'll think about it more carefully. Almost all of our deployment configuration under the hood is done with Kubernetes (which is focused on the declaration of deployment state). We modeled our configuration after Kubernetes for that reason, and we want to go beyond low-level infrastructure configuration by allowing users to configure prediction tracking, model retraining thresholds, and other more ML specific features using the same declarative paradigm and in the same configuration files.
  - sandGorgon 7y ago
    Terraform has first class support for K8s - https://www.terraform.io/docs/providers/kubernetes/index.html https://www.terraform.io/docs/providers/kubernetes/index.htm... In fact, i would say that's what Terraform was built around, so there's nothing more maintained than the k8s provider. In addition, it has EKS (https://www.terraform.io/docs/providers/aws/r/eks_cluster.html https://www.terraform.io/docs/providers/aws/r/eks_cluster.ht...) as well as GKE providers (https://www.terraform.io/docs/providers/google/r/container_cluster.html https://www.terraform.io/docs/providers/google/r/container_c...) in case you are so inclined.
    - dankohn1 7y ago
      Terraform slightly predates Kubernetes, May 21 vs. June 6, 2014. They were developed independently. Links at: https://landscape.cncf.io/category=automation-configuration,scheduling-orchestration&format=card-mode&grouping=no&organization=cloud-native-computing-foundation-cncf,hashi-corp&sort=first-commit https://landscape.cncf.io/category=automation-configuration,...
- joseph 7y ago
  Well, CDK actually produces CloudFormation templates. Sorry, but I always feel the urge to jump in when people claim Terraform should be used instead of CloudFormation because of personal preferences. If you are AWS native and already using CloudFormation, I see no reason to switch. CloudFormation provides a ton of functionality out of the box and Amazon handles it for you. Rollbacks alone are a huge reason one might want to use it over Terraform.
  - sandGorgon 7y ago
    well the reason to switch from CDK to Terraform is that your infrastructure management becomes a lot more cloud agnostic. That's basically what Terraform is for anyway. If you wanted, you could have scripted using the AWS SDK in python or something. if that's not a concern, then i suppose CDK is as good as any (probably better, since its in Typescript...but then so is Pulumi)

sandGorgon 7y ago

Not entirely. The abstractions are different between infrastructure deployment.... versus configuration yml of circleci. The declaration of deployment state is a very BIG and hard problem that has had millions of collective man hours spent over decades. I urge you not to think of it as a simple configuration. In fact it is so hard that AWS has to build a new language on top of typescript ..versus cloudformation templates that it already had. https://docs.aws.amazon.com/cdk/latest/guide/home.html https://docs.aws.amazon.com/cdk/latest/guide/home.html What you are building makes sense - I would drop cloudformation and surface Terraform right till the top. So the way to use your tool is to install and use a new Terraform "provider".

ospillinger 7y ago
The Terraform provider idea is interesting, I'll think about it more carefully. Almost all of our deployment configuration under the hood is done with Kubernetes (which is focused on the declaration of deployment state). We modeled our configuration after Kubernetes for that reason, and we want to go beyond low-level infrastructure configuration by allowing users to configure prediction tracking, model retraining thresholds, and other more ML specific features using the same declarative paradigm and in the same configuration files.
- sandGorgon 7y ago
  Terraform has first class support for K8s - https://www.terraform.io/docs/providers/kubernetes/index.html https://www.terraform.io/docs/providers/kubernetes/index.htm... In fact, i would say that's what Terraform was built around, so there's nothing more maintained than the k8s provider. In addition, it has EKS (https://www.terraform.io/docs/providers/aws/r/eks_cluster.html https://www.terraform.io/docs/providers/aws/r/eks_cluster.ht...) as well as GKE providers (https://www.terraform.io/docs/providers/google/r/container_cluster.html https://www.terraform.io/docs/providers/google/r/container_c...) in case you are so inclined.
  - dankohn1 7y ago
    Terraform slightly predates Kubernetes, May 21 vs. June 6, 2014. They were developed independently. Links at: https://landscape.cncf.io/category=automation-configuration,scheduling-orchestration&format=card-mode&grouping=no&organization=cloud-native-computing-foundation-cncf,hashi-corp&sort=first-commit https://landscape.cncf.io/category=automation-configuration,...
joseph 7y ago
Well, CDK actually produces CloudFormation templates. Sorry, but I always feel the urge to jump in when people claim Terraform should be used instead of CloudFormation because of personal preferences. If you are AWS native and already using CloudFormation, I see no reason to switch. CloudFormation provides a ton of functionality out of the box and Amazon handles it for you. Rollbacks alone are a huge reason one might want to use it over Terraform.
- sandGorgon 7y ago
  well the reason to switch from CDK to Terraform is that your infrastructure management becomes a lot more cloud agnostic. That's basically what Terraform is for anyway. If you wanted, you could have scripted using the AWS SDK in python or something. if that's not a concern, then i suppose CDK is as good as any (probably better, since its in Typescript...but then so is Pulumi)