9 ms·

Cue, an open-source data validation language

190 points by benhoyt 5y ago

tyingq 5y ago
It looks like it would be great to use locally within a browser, since so much data entry happens there. But it doesn't seem like that's currently possible. I wonder if they have a WASM port (or similar) on a roadmap.
- dqpb 5y ago
  Cue definitions export to json schema which you can use to validate form input.
- rjrodger 5y ago
  Try cuelang.org/play - it's exactly that. Seems to be just a demo though.
wikibob 5y ago
Some context: Cue is a project originally started by Marcel van Lohuizen who previously was part of BCL (Borg Config Lang) at Google. The main use is to generate config files. See the Kubernetes examples at: https://cuelang.org/docs/tutorials/ https://cuelang.org/docs/tutorials/ Here are two posts discussing the motivations for Cue over BCL/Jsonnet: - https://github.com/cuelang/cue/issues/33#issuecomment-483615374 https://github.com/cuelang/cue/issues/33#issuecomment-483615... - https://github.com/cuelang/cue/discussions/669 https://github.com/cuelang/cue/discussions/669 A very interesting development is that Grafana appears to be adopting Cue as a first-class configuration option. See: "Bring new CUE-based config schema system to release-readiness" https://github.com/grafana/grafana/issues/33139 https://github.com/grafana/grafana/issues/33139 This could mean that a future where Grafana dashboards can be two-way synced with a git repo will eventually exist. ---- Other tools with some industry adoption in the "Infrastructure as Code" space include - Dhall - Jsonnet (from BCL) - kustomize - Helm - kubecfg - Tanka - SkyCfg - jkcfg - Krane - HCL (Terraform) And two tools that fall into a separate class of enabling "Infrastructure as Software" - Pulumi (TypeScript/Go/Python/.NET) - CDK
- purpleidea 5y ago
  cough how can you leave out https://github.com/purpleidea/mgmt/ https://github.com/purpleidea/mgmt/ =D It's in golang, and is the only reactive DSL.
  - dolmen 5y ago
    The language is named Go.
- tamalsaha001 5y ago
  It might sound a bit pedantic, but kustomize strictly avoids the "Infrastructure as Code" space and stays in the "Infrastructure as Data" space. The main difference is that since it just deals with "data", you can build any higher level tooling on this. One of the major proponents of this idea is Brian Grant from Google. He tweets about this from time to time. Here is a recent one: https://twitter.com/bgrant0607/status/1404461906186833927 https://twitter.com/bgrant0607/status/1404461906186833927
  - chalst 5y ago
    Is this distinction really about whether the customisation language is declarative? It seems to me that Dhall has the advantages Brian Grant attributes to "Infrastructure as Data", although it is an executable specification.
    - throwaway894345 5y ago
      I also think “executable” is more important. Our configs are so large and complex that we need to DRY them up. A type system alone is insufficient.
- sdboyer 5y ago
  re: Grafana (i'm the author of the linked issue) - i'm quite excited, i do think there's a world of possibilities here. Two-way sync with a git repo is one possible path, and we've talked a lot internally at GL about how to best support it. My sense is that we can do it with relatively little friction and likely will - but if you're just syncing with a git repo, there's still a lot of arbitrary, opaque repo layout decisions that still have to be made (how do you map a filesystem position for a dashboard to a position in Grafana? In a way that places the dashboards next to the systems they're intended to observe? With many teams? With many Grafana instances?) which induce new kinds of friction at scale. Fortunately - and not mutually exclusively with the above - by building the system for schema in CUE, we've made a composable thing that we can make into larger systems. That's what we're starting to do with Polly: https://github.com/pollypkg/polly https://github.com/pollypkg/polly Conveniently, my parts of a Grafanaconline talk tomorrow discusses both of these https://grafana.com/go/grafanaconline/2021/dashboards-as-code/ https://grafana.com/go/grafanaconline/2021/dashboards-as-cod... :D
Kinrany 5y ago
I really like the language and look forward to seeing more adoption. Is there a second implementation in another programming language? As I understand it, it would make sense to have libraries for processing Cue data in every language. I'm a bit concerned about Cue relying on Go too much. A data validation language should be independent of the implementation language.
- rjrodger 5y ago
  Started working on a JS version a few weeks ago [1]. Even with 20% of the features it’s already so useful we’re building systems with it. And not just config - model all the things! Overrides and inheritance are a world of pain. Unification and commutative operations restore sanity to the actual work of coding with a domain representation language because WYSIWYG. And you get type safety for your domain model. The project is still at the “Read the Source, Luke” stage so caveat emptor until we get a respectable release out. * https://github.com/rjrodger/aontu https://github.com/rjrodger/aontu
  - wdb 5y ago
    Nice, I was looking for a Javascript version. I will check it out. Regarding "Read the Source, Luke" you are in good company with Apple and its Swift ABI :)
    - rjrodger 5y ago
      And we kinda leverage the docs on cuelang.org since the syntax and semantics are mostly the same!
  - sdboyer 5y ago
    Very excited to see someone doing this! Right now, Grafana is [planned to] relying on an anemic CUE->Typescript translator for getting its schema to the frontend - https://github.com/sdboyer/cuetsy https://github.com/sdboyer/cuetsy. (Somebody also pointed me to Project Cambria recently, which could be an interesting compilation target for what we have https://www.inkandswitch.com/cambria.html https://www.inkandswitch.com/cambria.html) Being able to work with CUE natively in TS, though, would be a huge gamechanger for what we can do with CUE in Grafana
throwamon 5y ago
Can a useful comparison be made between Cue and Clojure.spec?
- dgb23 5y ago
  Philosophically they seem very similar, but at a glance[0] it seems like clojure.spec is quite a bit more expressive. Also spec lives in your REPL session and in your source code, while Cue is meant to be used as a CLI, so there is a completely different approach. [0] https://cuelang.org/docs/references/spec/ https://cuelang.org/docs/references/spec/
  - sdboyer 5y ago
    CUE is currently centered around its CLI, but AIUI, that's not the long-term goal. i read in some CUE issue somewhere that the goal is shifting towards enabling frameworks rather than driving people to the CLI, though i don't have a link. Our use of CUE in Grafana is an example of framework-style usage. It is a hard requirement that users never have to install the CUE CLI to perform any of our planned CUE-related tasks; rather, the needed tooling is baked into Go packages we export, and things like grafana-cli. (Avoiding a dependency on the CUE CLI also gives us a defense mechanism against breaking changes)
thom 5y ago
This looks interesting, and I’d love to know more, for example what is Cue’s approach to validating sequences? I’d spend more time trying to answer that question myself but this site wants me to read an awful lot of philosophy before showing me any code. I dislike homepages like this. I’d rather you assume I buy into your philosophy, otherwise I’d leave, so you’re free to just show me what the language is like. When you do that, I’m more interested in examples than EBNF.
- jozvolskyef 5y ago
  https://cuetorials.com/ https://cuetorials.com/ is probably more hands-on than the homepage which is slightly more theoretical. For validating sequences see https://cuelang.org/docs/tutorials/tour/types/lists/ https://cuelang.org/docs/tutorials/tour/types/lists/ .
  - thom 5y ago
    Brilliant, ta! I guess I was hoping for something more like linear temporal logic perhaps. What we’ve found with lots of validation libraries is that doing “if this then expect that” kind of rules on sequences is quite difficult.
andix 5y ago
It looks quite cool. I think it would be really useful if you have a lot of integrations into different programming languages, frameworks, and maybe even SQL servers. So you could do data validation on the frontend, backend and the database server based on the same definitions. It would save us a lot of bugs caused by different opinions of valid data in different layers of software.
bitfield 5y ago
A quick overview of CUE for those who are wondering what all the fuss is about: https://bitfieldconsulting.com/golang/cuelang-exciting https://bitfieldconsulting.com/golang/cuelang-exciting
- wstrange 5y ago
  This is a fantastic read - well done. I'd recommend starting with this article as it nicely lays out the motivations for Cue.
abraxaz 5y ago
Is there any plans to support model generation in future as can be done with JSON schema through something like https://github.com/quicktype/quicktype https://github.com/quicktype/quicktype ?
- dqpb 5y ago
  Cue definitions export to json schema.
  - abraxaz 5y ago
    Can't see how, and this seems to suggest this functionality is not implemented yet: https://github.com/cuelang/cue/discussions/663 https://github.com/cuelang/cue/discussions/663 Mind sharing a reference?
    - deleted 5y ago
      [deleted]
    - dqpb 5y ago
      I do it the way they discuss in that issue. Converting from openapi to json schema.
      - abraxaz 5y ago
        Thanks for the tip.
debarshri 5y ago
I had met Marcel van Lohuizen when he was in the board of my previous company. One of the passionate techie and down to earth guy. He was actually working on cuelang and had not released it yet. After he gave us a presentation on Cue, one of my thoughts was that it is not easy for beginners to grasp it but then the language is not meant for beginners. My Second thought which was completely whack, was may be you could use it as a add-on for Protobufs, as the schema definitions in Cuelang has validations builts into it, which might remove boilerplate validation code in grpc services.
bketelsen 5y ago
Related hobby project - we've been building cueblox (cueblox.com) as a way to create, consume, and validate data in YAML and Markdown using cue. Cue is very powerful, and it's been fun working on this project.
hawaiianSpork 5y ago
If you are looking to do data validation from the JVM, you may try Baleen (written in Kotlin): https://github.com/ShopRunner/baleen/ https://github.com/ShopRunner/baleen/ I'm one of the contributors. We created a DSL in the language to describe the data and create tests. You can then use that data description to validate against json, csv, avro... One of the neat things we came up with was the concept of a data trace which is like a stack trace but is a path through the data to a particular error.
mirekrusin 5y ago
We use cuelang in big corp in prod, beautiful tool.
syats 5y ago
Just dropping this here https://www.w3.org/TR/shacl/ https://www.w3.org/TR/shacl/ Shacl is a language for defining constraints on data. While it is focused on RDF data, it is also possible to use with JSON/CSV data using an RML mapper (so far only PoC on this side).
billfruit 5y ago
Haven't taken a detailed look, but can it work on binary data?
carlosf 5y ago
At this point one might consider using a real language and common software practices for type checking, extending, modularization, testing, etc... Instead of building an ecosystem just to keep Infrastructure as Yaml sane. My experience with Pulumi and AWS CDK is absolutely brilliant in this regard, hopefully good DevOps/SRE/WhateverNewTerm practices and patterns will reassemble good software development practices in the future.
- sdboyer 5y ago
  > real language In what sense is CUE not a real language?
  - heywherelogingo 5y ago
    I think he means one language that includes config, instead of yaml plus yaml-taming ecosystem.
  - Micoloth 5y ago
    I think parent means a General Purpose language, i.e. capable of computations. Personally, on one hand I know allowing computations into configuration immediatly destroys any hope of having a tidy, rational schema in real word projects. On the other hand though, i do believe configuration and code should be build with related tools, possibly the same tool- or at least tools using the same syntax! (a bit like the json syntax is the same as a Python dict syntax, except this is the terrible example that is so poorly thought out that does more harm than good) This unlocks a much greater degree of freedom and power than all the gluing together technologies that we have to do...
    - sdboyer 5y ago
      If this is the sense in which the comment is intended - CUE is capable of some kinds of computation. It's just not Turing complete. My 2c - thus far, i've found the language features enabled by this constraint much more useful than the expressiveness lost - at least, for the purposes i've chosen.
- 5y ago
sdboyer 5y ago
It's not necessarily easy to see, just from general descriptions of CUE, what's possible. Dagger was demo'd at the recent Dockercon - gives a bit more of a sense of the possible with CUE: https://docker.events.cube365.net/dockercon-live/2021/content/Videos/TamRA3F2zT8fycvq5 https://docker.events.cube365.net/dockercon-live/2021/conten...
- moderation 5y ago
  Vector [0] leverages CUE for their documentation [1]. 0. https://vector.dev/ https://vector.dev/ 1. https://github.com/timberio/vector/blob/master/CONTRIBUTING.md#documentation https://github.com/timberio/vector/blob/master/CONTRIBUTING....
_asummers 5y ago
The Cue integrated Kubernetes project I'm most excited about is KubeVela[0]. Effectively, you can create an "operator" for just the YAML bits to narrow your Kubernetes API and provide best practices via the Components and Trait overrides, and it should allow platform teams to standardize how their teams are deploying software on large Kubernetes installations. [0] https://kubevela.io/ https://kubevela.io/
shykes 5y ago
If you’ve ever had to wrangle yaml configuration files… do yourself a favor and learn Cue. It’s still young and the website can seem intimidating; but it’s simpler than it looks, and the language is unbelievably powerful. There simply isn’t anything else like it. In my opinion it’s in a league of its own compared to other configuration languages like HCL, Jsonnet, Dhall, Starlark etc. Marcel, the creator of Cue, is basically the godfather of configuration languages - most of the state of the art can be traced back to his work at Google. Despite his deep knowledge of the subject and unparalleled experience, he is modest, pragmatic and responsive to questions and feedback. The momentum behind Cue reminds me of Go in its early days. I’ve been using Cue for over a year now, using it as the foundation for a new projet; and will gladly answer questions about our experience.
- throwaway894345 5y ago
  I tried dabbling with Cue, but it doesn't seem to solve the problem that I care about, which is that I have a whole bunch of configs that vary only slightly and I want to DRY them up. For example, for any given application we have several fixed environments--dev, staging, prod--as well as "on demand" environments for things like pull requests or individual developer environments. The configs for these environments are almost the same, but they vary based on a handful of parameters. I want to be able to write a "generic environment" module for each application and then parameterize it accordingly for each environment. Cue doesn't seem to care much about this problem, but rather it's just trying to make sure your data is type checked. It seems more like an advanced JSONSchema rather than a typed Starlark. I think the latter would be more powerful (albeit Cue's type system is more powerful than an ordinary generic type system with things like range types). Cue almost has an answer to the DRY problem, but you can't quite emulate functions as far as I can tell (due, I think, to shadowing problems). I wonder what people who are convinced that Cue is the future would say to this? Am I just thinking about the problem wrong?
  - deleted 5y ago
    [deleted]
  - Kinrany 5y ago
evancordell 5y ago
CUE is great - it's the first config language I've used that feels like it's getting things right. I've been playing with some different ways of using CUE: - https://github.com/cuebernetes/cuebectl https://github.com/cuebernetes/cuebectl (declarative, continuous syncing with a kube cluster) - https://github.com/ecordell/cuezel https://github.com/ecordell/cuezel (bazel/make but with CUE) both have some overlap philosophically with the "command" subsystem in CUE, but it's been fun to play with different approaches to the same problems.
arthurcolle 5y ago
Goldman had a library like this when I was there. I wonder if it ended up catching on.
phissenschaft 5y ago
Wonder if https://hydra.cc/ https://hydra.cc/ would be a better choice
wodenokoto 5y ago
When I hear data validation I think “csv or excel file generated by other department that I need to import”, how is cue with that sort of thing? Eg I want to import data file and get a report that tells me 20 values in column A are negative, column B contains floats etc. And can it munge stuff? Column C are stored as floats, but should have been of type int. If all values are whole numbers, accept it.

190 points by benhoyt 5y ago

It looks like it would be great to use locally within a browser, since so much data entry happens there. But it doesn't seem like that's currently possible. I wonder if they have a WASM port (or similar) on a roadmap.

dqpb 5y ago
Cue definitions export to json schema which you can use to validate form input.
rjrodger 5y ago
Try cuelang.org/play - it's exactly that. Seems to be just a demo though.

wikibob 5y ago

Some context: Cue is a project originally started by Marcel van Lohuizen who previously was part of BCL (Borg Config Lang) at Google. The main use is to generate config files. See the Kubernetes examples at: https://cuelang.org/docs/tutorials/ https://cuelang.org/docs/tutorials/ Here are two posts discussing the motivations for Cue over BCL/Jsonnet: - https://github.com/cuelang/cue/issues/33#issuecomment-483615374 https://github.com/cuelang/cue/issues/33#issuecomment-483615... - https://github.com/cuelang/cue/discussions/669 https://github.com/cuelang/cue/discussions/669 A very interesting development is that Grafana appears to be adopting Cue as a first-class configuration option. See: "Bring new CUE-based config schema system to release-readiness" https://github.com/grafana/grafana/issues/33139 https://github.com/grafana/grafana/issues/33139 This could mean that a future where Grafana dashboards can be two-way synced with a git repo will eventually exist. ---- Other tools with some industry adoption in the "Infrastructure as Code" space include - Dhall - Jsonnet (from BCL) - kustomize - Helm - kubecfg - Tanka - SkyCfg - jkcfg - Krane - HCL (Terraform) And two tools that fall into a separate class of enabling "Infrastructure as Software" - Pulumi (TypeScript/Go/Python/.NET) - CDK

purpleidea 5y ago
cough how can you leave out https://github.com/purpleidea/mgmt/ https://github.com/purpleidea/mgmt/ =D It's in golang, and is the only reactive DSL.
- dolmen 5y ago
  The language is named Go.
tamalsaha001 5y ago
It might sound a bit pedantic, but kustomize strictly avoids the "Infrastructure as Code" space and stays in the "Infrastructure as Data" space. The main difference is that since it just deals with "data", you can build any higher level tooling on this. One of the major proponents of this idea is Brian Grant from Google. He tweets about this from time to time. Here is a recent one: https://twitter.com/bgrant0607/status/1404461906186833927 https://twitter.com/bgrant0607/status/1404461906186833927
- chalst 5y ago
  Is this distinction really about whether the customisation language is declarative? It seems to me that Dhall has the advantages Brian Grant attributes to "Infrastructure as Data", although it is an executable specification.
  - throwaway894345 5y ago
    I also think “executable” is more important. Our configs are so large and complex that we need to DRY them up. A type system alone is insufficient.
sdboyer 5y ago
re: Grafana (i'm the author of the linked issue) - i'm quite excited, i do think there's a world of possibilities here. Two-way sync with a git repo is one possible path, and we've talked a lot internally at GL about how to best support it. My sense is that we can do it with relatively little friction and likely will - but if you're just syncing with a git repo, there's still a lot of arbitrary, opaque repo layout decisions that still have to be made (how do you map a filesystem position for a dashboard to a position in Grafana? In a way that places the dashboards next to the systems they're intended to observe? With many teams? With many Grafana instances?) which induce new kinds of friction at scale. Fortunately - and not mutually exclusively with the above - by building the system for schema in CUE, we've made a composable thing that we can make into larger systems. That's what we're starting to do with Polly: https://github.com/pollypkg/polly https://github.com/pollypkg/polly Conveniently, my parts of a Grafanaconline talk tomorrow discusses both of these https://grafana.com/go/grafanaconline/2021/dashboards-as-code/ https://grafana.com/go/grafanaconline/2021/dashboards-as-cod... :D

Kinrany 5y ago

I really like the language and look forward to seeing more adoption. Is there a second implementation in another programming language? As I understand it, it would make sense to have libraries for processing Cue data in every language. I'm a bit concerned about Cue relying on Go too much. A data validation language should be independent of the implementation language.

rjrodger 5y ago
Started working on a JS version a few weeks ago [1]. Even with 20% of the features it’s already so useful we’re building systems with it. And not just config - model all the things! Overrides and inheritance are a world of pain. Unification and commutative operations restore sanity to the actual work of coding with a domain representation language because WYSIWYG. And you get type safety for your domain model. The project is still at the “Read the Source, Luke” stage so caveat emptor until we get a respectable release out. * https://github.com/rjrodger/aontu https://github.com/rjrodger/aontu
- wdb 5y ago
  Nice, I was looking for a Javascript version. I will check it out. Regarding "Read the Source, Luke" you are in good company with Apple and its Swift ABI :)
  - rjrodger 5y ago
    And we kinda leverage the docs on cuelang.org since the syntax and semantics are mostly the same!
- sdboyer 5y ago
  Very excited to see someone doing this! Right now, Grafana is [planned to] relying on an anemic CUE->Typescript translator for getting its schema to the frontend - https://github.com/sdboyer/cuetsy https://github.com/sdboyer/cuetsy. (Somebody also pointed me to Project Cambria recently, which could be an interesting compilation target for what we have https://www.inkandswitch.com/cambria.html https://www.inkandswitch.com/cambria.html) Being able to work with CUE natively in TS, though, would be a huge gamechanger for what we can do with CUE in Grafana

throwamon 5y ago

Can a useful comparison be made between Cue and Clojure.spec?

dgb23 5y ago
Philosophically they seem very similar, but at a glance[0] it seems like clojure.spec is quite a bit more expressive. Also spec lives in your REPL session and in your source code, while Cue is meant to be used as a CLI, so there is a completely different approach. [0] https://cuelang.org/docs/references/spec/ https://cuelang.org/docs/references/spec/
- sdboyer 5y ago
  CUE is currently centered around its CLI, but AIUI, that's not the long-term goal. i read in some CUE issue somewhere that the goal is shifting towards enabling frameworks rather than driving people to the CLI, though i don't have a link. Our use of CUE in Grafana is an example of framework-style usage. It is a hard requirement that users never have to install the CUE CLI to perform any of our planned CUE-related tasks; rather, the needed tooling is baked into Go packages we export, and things like grafana-cli. (Avoiding a dependency on the CUE CLI also gives us a defense mechanism against breaking changes)

thom 5y ago

This looks interesting, and I’d love to know more, for example what is Cue’s approach to validating sequences? I’d spend more time trying to answer that question myself but this site wants me to read an awful lot of philosophy before showing me any code. I dislike homepages like this. I’d rather you assume I buy into your philosophy, otherwise I’d leave, so you’re free to just show me what the language is like. When you do that, I’m more interested in examples than EBNF.

jozvolskyef 5y ago
https://cuetorials.com/ https://cuetorials.com/ is probably more hands-on than the homepage which is slightly more theoretical. For validating sequences see https://cuelang.org/docs/tutorials/tour/types/lists/ https://cuelang.org/docs/tutorials/tour/types/lists/ .
- thom 5y ago
  Brilliant, ta! I guess I was hoping for something more like linear temporal logic perhaps. What we’ve found with lots of validation libraries is that doing “if this then expect that” kind of rules on sequences is quite difficult.

andix 5y ago

It looks quite cool. I think it would be really useful if you have a lot of integrations into different programming languages, frameworks, and maybe even SQL servers. So you could do data validation on the frontend, backend and the database server based on the same definitions. It would save us a lot of bugs caused by different opinions of valid data in different layers of software.

bitfield 5y ago

A quick overview of CUE for those who are wondering what all the fuss is about: https://bitfieldconsulting.com/golang/cuelang-exciting https://bitfieldconsulting.com/golang/cuelang-exciting

wstrange 5y ago
This is a fantastic read - well done. I'd recommend starting with this article as it nicely lays out the motivations for Cue.

abraxaz 5y ago

Is there any plans to support model generation in future as can be done with JSON schema through something like https://github.com/quicktype/quicktype https://github.com/quicktype/quicktype ?

dqpb 5y ago
Cue definitions export to json schema.
- abraxaz 5y ago
  Can't see how, and this seems to suggest this functionality is not implemented yet: https://github.com/cuelang/cue/discussions/663 https://github.com/cuelang/cue/discussions/663 Mind sharing a reference?
  - deleted 5y ago
    [deleted]
  - dqpb 5y ago
    I do it the way they discuss in that issue. Converting from openapi to json schema.
    - abraxaz 5y ago
      Thanks for the tip.

debarshri 5y ago

I had met Marcel van Lohuizen when he was in the board of my previous company. One of the passionate techie and down to earth guy. He was actually working on cuelang and had not released it yet. After he gave us a presentation on Cue, one of my thoughts was that it is not easy for beginners to grasp it but then the language is not meant for beginners. My Second thought which was completely whack, was may be you could use it as a add-on for Protobufs, as the schema definitions in Cuelang has validations builts into it, which might remove boilerplate validation code in grpc services.

bketelsen 5y ago

Related hobby project - we've been building cueblox (cueblox.com) as a way to create, consume, and validate data in YAML and Markdown using cue. Cue is very powerful, and it's been fun working on this project.

hawaiianSpork 5y ago

If you are looking to do data validation from the JVM, you may try Baleen (written in Kotlin): https://github.com/ShopRunner/baleen/ https://github.com/ShopRunner/baleen/ I'm one of the contributors. We created a DSL in the language to describe the data and create tests. You can then use that data description to validate against json, csv, avro... One of the neat things we came up with was the concept of a data trace which is like a stack trace but is a path through the data to a particular error.

mirekrusin 5y ago

We use cuelang in big corp in prod, beautiful tool.

syats 5y ago

Just dropping this here https://www.w3.org/TR/shacl/ https://www.w3.org/TR/shacl/ Shacl is a language for defining constraints on data. While it is focused on RDF data, it is also possible to use with JSON/CSV data using an RML mapper (so far only PoC on this side).

billfruit 5y ago

Haven't taken a detailed look, but can it work on binary data?

carlosf 5y ago

At this point one might consider using a real language and common software practices for type checking, extending, modularization, testing, etc... Instead of building an ecosystem just to keep Infrastructure as Yaml sane. My experience with Pulumi and AWS CDK is absolutely brilliant in this regard, hopefully good DevOps/SRE/WhateverNewTerm practices and patterns will reassemble good software development practices in the future.

sdboyer 5y ago
> real language In what sense is CUE not a real language?
- heywherelogingo 5y ago
  I think he means one language that includes config, instead of yaml plus yaml-taming ecosystem.
- Micoloth 5y ago
  I think parent means a General Purpose language, i.e. capable of computations. Personally, on one hand I know allowing computations into configuration immediatly destroys any hope of having a tidy, rational schema in real word projects. On the other hand though, i do believe configuration and code should be build with related tools, possibly the same tool- or at least tools using the same syntax! (a bit like the json syntax is the same as a Python dict syntax, except this is the terrible example that is so poorly thought out that does more harm than good) This unlocks a much greater degree of freedom and power than all the gluing together technologies that we have to do...
  - sdboyer 5y ago
    If this is the sense in which the comment is intended - CUE is capable of some kinds of computation. It's just not Turing complete. My 2c - thus far, i've found the language features enabled by this constraint much more useful than the expressiveness lost - at least, for the purposes i've chosen.
5y ago

sdboyer 5y ago

It's not necessarily easy to see, just from general descriptions of CUE, what's possible. Dagger was demo'd at the recent Dockercon - gives a bit more of a sense of the possible with CUE: https://docker.events.cube365.net/dockercon-live/2021/content/Videos/TamRA3F2zT8fycvq5 https://docker.events.cube365.net/dockercon-live/2021/conten...

moderation 5y ago
Vector [0] leverages CUE for their documentation [1]. 0. https://vector.dev/ https://vector.dev/ 1. https://github.com/timberio/vector/blob/master/CONTRIBUTING.md#documentation https://github.com/timberio/vector/blob/master/CONTRIBUTING....

_asummers 5y ago

The Cue integrated Kubernetes project I'm most excited about is KubeVela[0]. Effectively, you can create an "operator" for just the YAML bits to narrow your Kubernetes API and provide best practices via the Components and Trait overrides, and it should allow platform teams to standardize how their teams are deploying software on large Kubernetes installations. [0] https://kubevela.io/ https://kubevela.io/

shykes 5y ago

If you’ve ever had to wrangle yaml configuration files… do yourself a favor and learn Cue. It’s still young and the website can seem intimidating; but it’s simpler than it looks, and the language is unbelievably powerful. There simply isn’t anything else like it. In my opinion it’s in a league of its own compared to other configuration languages like HCL, Jsonnet, Dhall, Starlark etc. Marcel, the creator of Cue, is basically the godfather of configuration languages - most of the state of the art can be traced back to his work at Google. Despite his deep knowledge of the subject and unparalleled experience, he is modest, pragmatic and responsive to questions and feedback. The momentum behind Cue reminds me of Go in its early days. I’ve been using Cue for over a year now, using it as the foundation for a new projet; and will gladly answer questions about our experience.

throwaway894345 5y ago
I tried dabbling with Cue, but it doesn't seem to solve the problem that I care about, which is that I have a whole bunch of configs that vary only slightly and I want to DRY them up. For example, for any given application we have several fixed environments--dev, staging, prod--as well as "on demand" environments for things like pull requests or individual developer environments. The configs for these environments are almost the same, but they vary based on a handful of parameters. I want to be able to write a "generic environment" module for each application and then parameterize it accordingly for each environment. Cue doesn't seem to care much about this problem, but rather it's just trying to make sure your data is type checked. It seems more like an advanced JSONSchema rather than a typed Starlark. I think the latter would be more powerful (albeit Cue's type system is more powerful than an ordinary generic type system with things like range types). Cue almost has an answer to the DRY problem, but you can't quite emulate functions as far as I can tell (due, I think, to shadowing problems). I wonder what people who are convinced that Cue is the future would say to this? Am I just thinking about the problem wrong?
- deleted 5y ago
  [deleted]
- Kinrany 5y ago

evancordell 5y ago

CUE is great - it's the first config language I've used that feels like it's getting things right. I've been playing with some different ways of using CUE: - https://github.com/cuebernetes/cuebectl https://github.com/cuebernetes/cuebectl (declarative, continuous syncing with a kube cluster) - https://github.com/ecordell/cuezel https://github.com/ecordell/cuezel (bazel/make but with CUE) both have some overlap philosophically with the "command" subsystem in CUE, but it's been fun to play with different approaches to the same problems.

arthurcolle 5y ago

Goldman had a library like this when I was there. I wonder if it ended up catching on.

phissenschaft 5y ago

Wonder if https://hydra.cc/ https://hydra.cc/ would be a better choice

wodenokoto 5y ago

When I hear data validation I think “csv or excel file generated by other department that I need to import”, how is cue with that sort of thing? Eg I want to import data file and get a report that tells me 20 values in column A are negative, column B contains floats etc. And can it munge stuff? Column C are stored as floats, but should have been of type int. If all values are whole numbers, accept it.