5 ms·

Stripe open-sources Skycfg, a configuration builder for Kubernetes

97 points by jmillikin 8y ago

houqp 8y ago
How is this compared to pulumi k8s module?
- jmillikin 8y ago
  Skycfg[0] is hermetic and deterministic. Evaluating a Skycfg configuration file can't execute local processes or access system properties such as the current time, so it can be treated as inert data similar to a JSON or YAML file. This important for Stripe because we have lots of internal security boundaries. A rich configuration language that can't break out of its sandbox is useful when a highly-trusted system needs to expose a subset of its power to less trusted agents. [0] More specifically, the Starlark language that Skycfg extends.
simplify 8y ago
It doesn't have functions, but I've also released a type-checked configuration language recently[0]. The main difference is it's designed to be easy to edit, even by non-technical users. [0] https://github.com/gilbert/zaml https://github.com/gilbert/zaml
- jmillikin 8y ago
  Note that Skycfg isn't a language itself; we're using Google's Starlark (a deterministic Python dialect) for syntax and evaluation. This means that we get IDE support and tooling for ~free, such as Python syntax highlighting and Starlark formatting with Buildifier.
  - simplify 8y ago
    A deterministic Python dialect is indeed very cool. I've dabbled in writing a deterministic language myself. Syntax highlighting and formatting is nice, but I think error reporting is more important for the end user. Zaml auto-validates configuration structure, removing the need to write most boilerplate on the programmer's side. Can you help me understand Skycfg's type safety and validation? When I see `return [pb.StringValue(value = 123)]`, does this imply the schema is written inline, alongside the values?
    - jmillikin 8y ago
      The schema is defined as a Protocol Buffer (https://developers.google.com/protocol-buffers/ https://developers.google.com/protocol-buffers/) package. This lets us construct values protobuf-based APIs without having to manually copy over the schema. For example, if you had this schema: package my.example.schema; message Person { string name = 1; int32 id = 2; string email = 3; } Then you could build a value using Skycfg like this: pb = proto.package("my.example.schema") msg = pb.Person( id = 1234, name = "Jane Doe", email = "jdoe@example.com", ) print(msg) For Kubernetes, we're using the schema at https://github.com/kubernetes/api https://github.com/kubernetes/api and for Envoy we use https://github.com/envoyproxy/data-plane-api/tree/master/envoy https://github.com/envoyproxy/data-plane-api/tree/master/env...
- corwin7 8y ago
  > The main difference is it's designed to be easy to edit, even by non-technical users. Yaml is beautiful and easy, this looks like a nightmare.
  - kjeetgill 8y ago
    Many uses of YAML are beautiful and easy but the spec/parsers are anything but. 5 notations for things, unclear whitespace delimiters, graph/labels, etc. https://news.ycombinator.com/item?id=17358103 https://news.ycombinator.com/item?id=17358103
justicezyx 8y ago
I am seeing K8s ecosystem repeating Borg's own history of configuration churns. It's just there seems so many configuration tools for K8s now, which seems a much worse situation than Borg (started with only one configuration tool).
- ithkuil 8y ago
  Here is one of those: https://github.com/ksonnet/kubecfg https://github.com/ksonnet/kubecfg
  - atombender 8y ago
    For those who are as confused as me why Kubecfg exists in the Ksonnet org on Github, when they also have Ksonnet itself, I found this explanation: https://blog.heptio.com/the-next-chapter-for-ksonnet-1dcbbad30cb https://blog.heptio.com/the-next-chapter-for-ksonnet-1dcbbad.... Short summary: They're different but somewhat complementary tools. Kubecfg was made by Bitnami, Ksonnet by Heptio. Kubecfg uses Jsonnet and is less opinioned than Ksonnet, but (as far as I can see) can also use Ksonnet.
    - guslees 8y ago
      The fork is my fault. I agree it's confusing. I wrote (and still maintain) kubecfg. Heptio joined the project and started adding lots of great stuff. Eventually it was clear they wanted to take it in a direction that was different to the original borgcfg-like vision. I suggested we split that new functionality into a different tool so we could keep exploring both directions without trying to mash both into the same cli flags. Hence ksonnet/kubecfg and ksonnet/ksonnet. They both use jsonnet internally, generate the same k8s resources in the end, and have that common code heritage, so have many similarities. ksonnet/ksonnet has a bunch of extra ("rails-like") tooling to hold your hand while you generate jsonnet, and assumes it uses the k.libsonnet library (https://github.com/ksonnet/ksonnet-lib https://github.com/ksonnet/ksonnet-lib). ksonnet/kubecfg is much dumber and really just conceptually `jsonnet | kubectl apply`. In particular kubecfg avoids having any opinion about which jsonnet libraries you use. We use it extensively with https://github.com/bitnami-labs/kube-libsonnet https://github.com/bitnami-labs/kube-libsonnet, but you can also use it with https://github.com/ksonnet/ksonnet-lib https://github.com/ksonnet/ksonnet-lib or anything else that is valid jsonnet.
greggdonovan 8y ago
This looks great. We have a similar stack (Bazel, Terraform, Kubernetes) and have struggled with configuration. I'd love to learn more detail about how you've use skycfg with Terraform. Do Terraform module definitions drive the proto config (something we've considered)? Or does skycfg generate JSON or HCL? Thanks!
- jmillikin 8y ago
  Our use of Skycfg with Terraform is still very basic, using Terraform's JSON syntax[0] and the "external" data source[1] for particular configurations that are difficult to express in HCL. I personally would like to have most of our Terraform config in Skycfg format for testing/analysis purposes, but that's blocked on having some sort of schema for the providers we use. [0] https://www.terraform.io/docs/configuration/syntax.html#json-syntax https://www.terraform.io/docs/configuration/syntax.html#json... [1] https://www.terraform.io/docs/providers/external/data_source.html https://www.terraform.io/docs/providers/external/data_source...
atombender 8y ago
Looks interesting, but why is it a library and not a command line tool? For Kubernetes you don't really want to write a program (one that calls this library) for every single Kubernetes deploy you want to do. Seeing as Protobuf schemes are parseable, why not just have a command line tool that parses the schema, then combines it with the Starlark declarations to generate Kubernetes manifests? It could be an interesting competitor to Ksonnet if done right.
- jmillikin 8y ago
  Internally we use Skycfg as a library linked into other tooling, the same way you might link in a YAML parser. The ideal adoption model would see "skycfg support" as a checkmark feature of user-facing tools like kubectl or Helm. The specific case of a sky-to-yaml utility binary is one we want to write (and have started in the `_examples/repl/` directory). Unfortunately go-protobuf doesn't support dynamic protobuf schemas yet (https://github.com/golang/protobuf/issues/199 https://github.com/golang/protobuf/issues/199). Until it does, every message type must be linked directly into the binary.
  - atombender 8y ago
    Thanks for clearing that up. The story title and the Github repo together makes it unclear what this is intended for. I'd love to see a Helm-like tool built on Skycfg. I'm really surprised that the Go Protobuf implementation doesn't support dynamic schemas, which seems like such a core feature, but looks like work is underway.
- justicezyx 8y ago
  Why is this downvoted?
corwin7 8y ago
If you can't edit a yaml (or HCL) file, you shouldn't be using Kubernetes / Terraform. All these "you too can be an artist, just draw inside the lines of this coloring book" configuration systems (ie helm) are just distracting people from doing things the right way.
- londons_explore 8y ago
  Plain yaml files have far too much repetition for common use cases. If I have 100 microservices, and I want to run all of them in a production and staging cluster, but I want all the staging cluster jobs to have an extra environment variable/tag and have lower CPU limits, there's no way to do that without 200 yaml files, which quickly get out of sync and inconsistent.
  - deleted 8y ago
    [deleted]
- atombender 8y ago
  That's pretty extreme. What if you have a deployment that should have a 100GB PVC in production, but only 10GB in staging? Do you duplicate the entire config, or pass around patches, just to make a single line change? What if you have different HPAs, different replica counts, different resource requests/limits, etc.? People use tools like Helm and Ksonnet because they allow you to write general-purpose manifests that can be parameterized. It's an extremely powerful tool. Our charts are full of things like this for that reason: resources: {{ if .Values.kubernetes.resources }} requests: {{ if .Values.kubernetes.resources.requests.cpu }}cpu: {{ .Values.kubernetes.resources.requests.cpu }}{{ end }} {{ if .Values.kubernetes.resources.requests.memory }}memory: {{ .Values.kubernetes.resources.requests.memory }}{{ end }} limits: {{ if .Values.kubernetes.resources.limits.cpu }}cpu: {{ .Values.kubernetes.resources.limits.cpu }}{{ end }} {{ if .Values.kubernetes.resources.limits.memory }}memory: {{ .Values.kubernetes.resources.limits.memory }}{{ end }} {{ end }} Or stuff like dynamically built ingresses: - host: {{ .Values.host }} http: paths: {{ range .Values.apps }} - path: {{ .path }} backend: serviceName: {{ .name }} servicePort: 80 {{ end }} Admittedly, templating sucks, and a tool like Skycfg or Jsonnet would be significantly better here, but the fundamental problem to solve is the same. I'm a big fan of configurator generators. I think it's wise to let apps be simple and rely o static, pre-baked configs (JSON, YAML, whatever), but a deployment system needs to be able to generate dynamic ones.
londons_explore 8y ago
I'm just waiting for someone to make a kubernetes config generator based on googles GCL (not opensource, but this is an opensource clone of it, although without k8s integration/examples: https://github.com/rix0rrr/gcl https://github.com/rix0rrr/gcl). I feel it has almost limitless flexibility, without requiring either boilerplate or complexity where you don't need it. Nothing else comes close in that regard.
- justicezyx 8y ago
  Disclaimer: I maintain borg’s configuration tool, which was gcl originally. Gcl is terse, dead simple in most common areas. It appears the best general purpose configuration language I’ve seen so far.
  - londons_explore 8y ago
    The main caveat of the above is that GCL is so powerful, it's easy to write crazily complex puzzles in it. Just because you can write a mandlebrot fractal generator in your config language, does not mean you should. And you can bet the person to write the configs of that service you inherited did enjoy seeing quite how far they could stretch the flexibility of GCL...
    - justicezyx 8y ago
      Well we have good ways to solve these problem, Complexity management in programming language is a well understood problem, I see plenty of low hanging fruits. Also trrseness has nothing to do with complexity. Gcl did not have much language syntax sugar intended for terseness. It’s just naturally simple in choices of right abstractions.
- dilyevsky 8y ago
  Hashicorp comes pretty close with their new edition of HCL at least in terms of language. It’s still more flexible in its output (unlike gcl where the output is protos). If you are familiar with google’s piccolo this is closer to that.
- 8y ago
gregwebs 8y ago
Dhall and its Kubernetes support are another potential option [1]. I am not using them yet, but looking to switch from helm templates to something that isn't just text mash-up but has modules and type-safety. Skycfg looks like a nice option to have: more main-stream for those used to Python or Bazel even if it is missing some of Dhall's interesting features (such as reduction). [1]: https://github.com/dhall-lang/dhall-kubernetes https://github.com/dhall-lang/dhall-kubernetes

97 points by jmillikin 8y ago

houqp 8y ago

How is this compared to pulumi k8s module?

jmillikin 8y ago
Skycfg[0] is hermetic and deterministic. Evaluating a Skycfg configuration file can't execute local processes or access system properties such as the current time, so it can be treated as inert data similar to a JSON or YAML file. This important for Stripe because we have lots of internal security boundaries. A rich configuration language that can't break out of its sandbox is useful when a highly-trusted system needs to expose a subset of its power to less trusted agents. [0] More specifically, the Starlark language that Skycfg extends.

simplify 8y ago

It doesn't have functions, but I've also released a type-checked configuration language recently[0]. The main difference is it's designed to be easy to edit, even by non-technical users. [0] https://github.com/gilbert/zaml https://github.com/gilbert/zaml

jmillikin 8y ago
Note that Skycfg isn't a language itself; we're using Google's Starlark (a deterministic Python dialect) for syntax and evaluation. This means that we get IDE support and tooling for ~free, such as Python syntax highlighting and Starlark formatting with Buildifier.
- simplify 8y ago
  A deterministic Python dialect is indeed very cool. I've dabbled in writing a deterministic language myself. Syntax highlighting and formatting is nice, but I think error reporting is more important for the end user. Zaml auto-validates configuration structure, removing the need to write most boilerplate on the programmer's side. Can you help me understand Skycfg's type safety and validation? When I see `return [pb.StringValue(value = 123)]`, does this imply the schema is written inline, alongside the values?
  - jmillikin 8y ago
    The schema is defined as a Protocol Buffer (https://developers.google.com/protocol-buffers/ https://developers.google.com/protocol-buffers/) package. This lets us construct values protobuf-based APIs without having to manually copy over the schema. For example, if you had this schema: package my.example.schema; message Person { string name = 1; int32 id = 2; string email = 3; } Then you could build a value using Skycfg like this: pb = proto.package("my.example.schema") msg = pb.Person( id = 1234, name = "Jane Doe", email = "jdoe@example.com", ) print(msg) For Kubernetes, we're using the schema at https://github.com/kubernetes/api https://github.com/kubernetes/api and for Envoy we use https://github.com/envoyproxy/data-plane-api/tree/master/envoy https://github.com/envoyproxy/data-plane-api/tree/master/env...
corwin7 8y ago
> The main difference is it's designed to be easy to edit, even by non-technical users. Yaml is beautiful and easy, this looks like a nightmare.
- kjeetgill 8y ago
  Many uses of YAML are beautiful and easy but the spec/parsers are anything but. 5 notations for things, unclear whitespace delimiters, graph/labels, etc. https://news.ycombinator.com/item?id=17358103 https://news.ycombinator.com/item?id=17358103

justicezyx 8y ago

I am seeing K8s ecosystem repeating Borg's own history of configuration churns. It's just there seems so many configuration tools for K8s now, which seems a much worse situation than Borg (started with only one configuration tool).

ithkuil 8y ago
Here is one of those: https://github.com/ksonnet/kubecfg https://github.com/ksonnet/kubecfg
- atombender 8y ago
  For those who are as confused as me why Kubecfg exists in the Ksonnet org on Github, when they also have Ksonnet itself, I found this explanation: https://blog.heptio.com/the-next-chapter-for-ksonnet-1dcbbad30cb https://blog.heptio.com/the-next-chapter-for-ksonnet-1dcbbad.... Short summary: They're different but somewhat complementary tools. Kubecfg was made by Bitnami, Ksonnet by Heptio. Kubecfg uses Jsonnet and is less opinioned than Ksonnet, but (as far as I can see) can also use Ksonnet.
  - guslees 8y ago
    The fork is my fault. I agree it's confusing. I wrote (and still maintain) kubecfg. Heptio joined the project and started adding lots of great stuff. Eventually it was clear they wanted to take it in a direction that was different to the original borgcfg-like vision. I suggested we split that new functionality into a different tool so we could keep exploring both directions without trying to mash both into the same cli flags. Hence ksonnet/kubecfg and ksonnet/ksonnet. They both use jsonnet internally, generate the same k8s resources in the end, and have that common code heritage, so have many similarities. ksonnet/ksonnet has a bunch of extra ("rails-like") tooling to hold your hand while you generate jsonnet, and assumes it uses the k.libsonnet library (https://github.com/ksonnet/ksonnet-lib https://github.com/ksonnet/ksonnet-lib). ksonnet/kubecfg is much dumber and really just conceptually `jsonnet | kubectl apply`. In particular kubecfg avoids having any opinion about which jsonnet libraries you use. We use it extensively with https://github.com/bitnami-labs/kube-libsonnet https://github.com/bitnami-labs/kube-libsonnet, but you can also use it with https://github.com/ksonnet/ksonnet-lib https://github.com/ksonnet/ksonnet-lib or anything else that is valid jsonnet.

greggdonovan 8y ago

This looks great. We have a similar stack (Bazel, Terraform, Kubernetes) and have struggled with configuration. I'd love to learn more detail about how you've use skycfg with Terraform. Do Terraform module definitions drive the proto config (something we've considered)? Or does skycfg generate JSON or HCL? Thanks!

jmillikin 8y ago
Our use of Skycfg with Terraform is still very basic, using Terraform's JSON syntax[0] and the "external" data source[1] for particular configurations that are difficult to express in HCL. I personally would like to have most of our Terraform config in Skycfg format for testing/analysis purposes, but that's blocked on having some sort of schema for the providers we use. [0] https://www.terraform.io/docs/configuration/syntax.html#json-syntax https://www.terraform.io/docs/configuration/syntax.html#json... [1] https://www.terraform.io/docs/providers/external/data_source.html https://www.terraform.io/docs/providers/external/data_source...

atombender 8y ago

Looks interesting, but why is it a library and not a command line tool? For Kubernetes you don't really want to write a program (one that calls this library) for every single Kubernetes deploy you want to do. Seeing as Protobuf schemes are parseable, why not just have a command line tool that parses the schema, then combines it with the Starlark declarations to generate Kubernetes manifests? It could be an interesting competitor to Ksonnet if done right.

jmillikin 8y ago
Internally we use Skycfg as a library linked into other tooling, the same way you might link in a YAML parser. The ideal adoption model would see "skycfg support" as a checkmark feature of user-facing tools like kubectl or Helm. The specific case of a sky-to-yaml utility binary is one we want to write (and have started in the `_examples/repl/` directory). Unfortunately go-protobuf doesn't support dynamic protobuf schemas yet (https://github.com/golang/protobuf/issues/199 https://github.com/golang/protobuf/issues/199). Until it does, every message type must be linked directly into the binary.
- atombender 8y ago
  Thanks for clearing that up. The story title and the Github repo together makes it unclear what this is intended for. I'd love to see a Helm-like tool built on Skycfg. I'm really surprised that the Go Protobuf implementation doesn't support dynamic schemas, which seems like such a core feature, but looks like work is underway.
justicezyx 8y ago
Why is this downvoted?

corwin7 8y ago

If you can't edit a yaml (or HCL) file, you shouldn't be using Kubernetes / Terraform. All these "you too can be an artist, just draw inside the lines of this coloring book" configuration systems (ie helm) are just distracting people from doing things the right way.

londons_explore 8y ago
Plain yaml files have far too much repetition for common use cases. If I have 100 microservices, and I want to run all of them in a production and staging cluster, but I want all the staging cluster jobs to have an extra environment variable/tag and have lower CPU limits, there's no way to do that without 200 yaml files, which quickly get out of sync and inconsistent.
- deleted 8y ago
  [deleted]
atombender 8y ago
That's pretty extreme. What if you have a deployment that should have a 100GB PVC in production, but only 10GB in staging? Do you duplicate the entire config, or pass around patches, just to make a single line change? What if you have different HPAs, different replica counts, different resource requests/limits, etc.? People use tools like Helm and Ksonnet because they allow you to write general-purpose manifests that can be parameterized. It's an extremely powerful tool. Our charts are full of things like this for that reason: resources: {{ if .Values.kubernetes.resources }} requests: {{ if .Values.kubernetes.resources.requests.cpu }}cpu: {{ .Values.kubernetes.resources.requests.cpu }}{{ end }} {{ if .Values.kubernetes.resources.requests.memory }}memory: {{ .Values.kubernetes.resources.requests.memory }}{{ end }} limits: {{ if .Values.kubernetes.resources.limits.cpu }}cpu: {{ .Values.kubernetes.resources.limits.cpu }}{{ end }} {{ if .Values.kubernetes.resources.limits.memory }}memory: {{ .Values.kubernetes.resources.limits.memory }}{{ end }} {{ end }} Or stuff like dynamically built ingresses: - host: {{ .Values.host }} http: paths: {{ range .Values.apps }} - path: {{ .path }} backend: serviceName: {{ .name }} servicePort: 80 {{ end }} Admittedly, templating sucks, and a tool like Skycfg or Jsonnet would be significantly better here, but the fundamental problem to solve is the same. I'm a big fan of configurator generators. I think it's wise to let apps be simple and rely o static, pre-baked configs (JSON, YAML, whatever), but a deployment system needs to be able to generate dynamic ones.

londons_explore 8y ago

I'm just waiting for someone to make a kubernetes config generator based on googles GCL (not opensource, but this is an opensource clone of it, although without k8s integration/examples: https://github.com/rix0rrr/gcl https://github.com/rix0rrr/gcl). I feel it has almost limitless flexibility, without requiring either boilerplate or complexity where you don't need it. Nothing else comes close in that regard.

justicezyx 8y ago
Disclaimer: I maintain borg’s configuration tool, which was gcl originally. Gcl is terse, dead simple in most common areas. It appears the best general purpose configuration language I’ve seen so far.
- londons_explore 8y ago
  The main caveat of the above is that GCL is so powerful, it's easy to write crazily complex puzzles in it. Just because you can write a mandlebrot fractal generator in your config language, does not mean you should. And you can bet the person to write the configs of that service you inherited did enjoy seeing quite how far they could stretch the flexibility of GCL...
  - justicezyx 8y ago
    Well we have good ways to solve these problem, Complexity management in programming language is a well understood problem, I see plenty of low hanging fruits. Also trrseness has nothing to do with complexity. Gcl did not have much language syntax sugar intended for terseness. It’s just naturally simple in choices of right abstractions.
dilyevsky 8y ago
Hashicorp comes pretty close with their new edition of HCL at least in terms of language. It’s still more flexible in its output (unlike gcl where the output is protos). If you are familiar with google’s piccolo this is closer to that.
8y ago

gregwebs 8y ago

Dhall and its Kubernetes support are another potential option [1]. I am not using them yet, but looking to switch from helm templates to something that isn't just text mash-up but has modules and type-safety. Skycfg looks like a nice option to have: more main-stream for those used to Python or Bazel even if it is missing some of Dhall's interesting features (such as reduction). [1]: https://github.com/dhall-lang/dhall-kubernetes https://github.com/dhall-lang/dhall-kubernetes