10 ms·
Cue: A new language for data validation
- kjgkjhfkjf 5y ago> CUE [...] makes an effort to incorporate lessons learned from 15 years of GCL usage. The main lesson learned by many people at Google, including the operators of many mission-critical services, was that GCL was far more trouble than it was worth. They used an alternative more mainstream configuration language instead.
- ithkuil 5y agowhich mainstream configuration language?
- captn3m0 5y agoJust a guess, https://github.com/google/starlark-go https://github.com/google/starlark-go perhaps? It is a subset of Python.
- bjackman 5y agoStarlark has taken over the build system (as it was originally built for) and a small number of other systems at Google. There are a couple of other internal languages that found success in specific domains. But GCL remains the most common config language at Google unfortunately.
- verdverm 5y agoI hear the GCL lambdas make for a bad day. Marcel, who created CUE, helped develop Borg, borgcfg, GCL. He talks about the "unfortunate" sentiment in the last video on this page https://cuetorials.com/videos https://cuetorials.com/videos
- bjackman 5y agoFunny detail: I really did not understand GCL until I read about CUE and also JSonnet, which I believe are both by post-GCL ex/Googlers. Once I understood those languages, I started to get a handle on GCL. "GCL is like a fucked up JSonnet" is a better angle than "JSonnet is like GCL if it wasn't torture"
- ithkuil 5y agoYeah, GCL's scoping rules are confusing. Jsonnet got rid of "up" and a few other details and suddenly things are much less likely to blow up in your hands. curiosity: I once wrote a GCL function that computed a PNG data url (the thing that looks like '<img src=data:image/png;base64,iVBORw...' and can be natively opened by your browser) whose content was a rendering of the mandelbrot set. It took several GB of RAM to evaluate. Didn't port it to jsonnet yet...
- IshKebab 5y agoMaybe Javascript? A lot of web tools support Javascript config files. There's this nice-looking effort to provide a hermetic execution environment for them: https://github.com/jkcfg/jk https://github.com/jkcfg/jk and if you use Typescript you get an extremely good static type system too. Plus the language is already very well known with loads of tool support and documentation. Definitely what I would use today.
- encryptluks2 5y agoLooks nice. I guess my question is how easy would it be to implement this into into the providers directly? Reminds me of OPA which I believe does something similar already.
- verdverm 5y agoDepends on what you mean by providers. There is only a Go implementation at present. The more complicated thing is that it supports imports and modules. A lot of programs may need some adjustment for this concept. For example, what do you put in a k8s ConfigMap? Should a container have an init step to fetch modules? Grafana and Sigstore are making Cuelang first class. There are some other CUE native applications developing in the community.
- charlieegan3 5y agoI wrote a blog post comparing Rego (OPA), Cue and Polar for authorization in Go applications. You might find it interesting: https://charlieegan3.com/posts/2021-05-08-authorization-dsls-in-go/ https://charlieegan3.com/posts/2021-05-08-authorization-dsls...
- verdverm 5y agoThere's an issue on the Cuelang GH that outlines the creators thoughts on Rego, CUE, and admission policies. https://github.com/cue-lang/cue/discussions/818 https://github.com/cue-lang/cue/discussions/818
- rmrfchik 5y agoNot related to Cue, but "All is good now" simply shows how to shoot yourself in the foot with yaml and apply a nice patch to a wound. That's not good at all.
- 2mol 5y agoI'm conflicted between - this looks very useful, and - "parse, don't validate" has proven to me to be a far better approach for all my use-cases. I guess validation that is language/consumer-agnostic is more useful in a big-tech context? Or maybe the advantage is to put constraints on data that you are sending to a system out of your control? Keen to hear opinions of people that worked with similar tools.
- verdverm 5y agoPutting constraints on configuration is a big use case. CUE is much nicer to write than Yaml or JSON, and then you can output those formats for the programs which use the "old" formats. There is a vet command if you want to validate Yaml or JSON before moving to generating them. It has helped me catch errors and I use CUE instead of Helm for a lot of things now. I'm looking into generating Terraform JSON rather than HCL. If there was going to be a language, CUE is preferable to HCL. Another thing I have been working on is a tool to diff, mask, upsert, and transform config or data files in bulk using CUE. https://github.com/hofstadter-io/cuetuls https://github.com/hofstadter-io/cuetuls
- BiteCode_dev 5y agoDo you use Cue for generating config, or for more for validating it. I kinda like Jsonnet, but it only generates.
- frumiousirc 5y agoTrue, though one may turn that around to say Cue's scope is too large while Jsonnet's is more "pure" while still enabling the larger scope. Schema that describes data is itself just data and Jsonnet can describe both. While the various Jsonnet compilers do not "know" about any specific meta-schema in which to express schema nor of course any way to apply that schema to data for validation, one can create such systems with Jsonnet as the language and its compilers as a component.
- 5y ago
- verdverm 5y agoCUE is having it's first town hall / community meeting in a couple of days. The roadmap and new foundation will be hot topics. https://github.com/cue-lang/cue/discussions/1300 https://github.com/cue-lang/cue/discussions/1300
- mirekrusin 5y agoCue is great, we’re using it in prod to manage deployment manifests.
- notwedtm 5y agoWhy? What does Cue have that older, more production-tested systems do not? I'm curious as to what problems I'm missing today that may come up in the future that Cue is solving.
- mirekrusin 5y agoThere is no noise. Size of configs is close to optimum. Cross cutting concerns are easy to express. Quality of configs is very high similar to well typed code with algebra on types, this idea of values are types and lattice based unification is very intuitive. ...and we are generating manifests for "older, more production-tested systems", ie. docker compose for docker vms/swarm services - it's really great for that.
- verdverm 5y agoManaging configuration at scale or when the person SRE'n is not the same person who wrote, configured, and deployed the application. Marcels latest talk has a good overview of these problems solved with CUE (https://cuetorials.com/videos https://cuetorials.com/videos)
- xyzzy_plugh 5y agoA commenter in another thread raises a good point: why not Starlark? https://github.com/google/starlark-go https://github.com/google/starlark-go Starlark is so, so good by comparison to Cue.
- mohanmcgeek 5y agoI have not used starlark, but to me it looks like a cleaner competitor to hcl. It can be used to generate verbose configs. It doesn't seem to enforce types and constraints on the fields. Surely there must be a way to do that using skylark..? But in cue types and constraints are just another form of values. Cue is a nicer jsonSchema competitor in which valid json is also valid cue
- throwaway894345 5y agoI’ve done a lot with Starlark, but my big beef is the lack of types. Cue aspires to be declarative which I never find myself reaching for, but maybe that’s just for lack of experience with the declarative category of tools.
- ootsootsoots 5y agoI’ve worked with CUE in production. It’s a very friendly Googler’s fever dream. It’s HCL without a Terraform command. You will be invested in the workflow and toolchain building. Barely anyone uses it, so have fun training up new hires. Some folks just straight up left the company rather than get sucked into a new DSL Frankly, the AWS SDK works great. The documentation includes working examples, is available in familiar syntax, I avoid CF’s opinions, and state is stored in the release branch. I can successfully rebuild any version of our infra by checking out a git hash. Meanwhile month old Terraform and cue break because a middleman had a brilliant idea at 1AM or some reason. IMO devops as an idea jumped the shark almost immediately. Folks decided developer habits were to be added to ops habits, but on the ground it just meant “use source control”. No other wisdom, like don’t repeat yourself and write less code, came along. I think SRE was meant to fix that but the same people and perspectives just took that over with DSL hell too I’m don’t burn out on tech work. I burn out on tech worker bullshit
- typical182 5y agoFWIW, the best intro to CUE I’ve found so far is: https://bitfieldconsulting.com/golang/cuelang-exciting https://bitfieldconsulting.com/golang/cuelang-exciting …which does a nice job of starting from the problem, and building up from "how could we improve JSON for use as a config language" to arriving at CUE. (Those types of explanations happen to resonate with me, including understanding the "why" first at least for me helps the mechanics and details later make more sense and feel more intuitive…)
- peterb 5y agoI really enjoyed the article. Thank you for posting it.
- angelzen 5y agoCUE is really promising and the article is a good first introduction to CUE. After reading it, there are a few things that don't fully click: * How does CUE compare with a standard structural typing system, i.e. Typescript. Specifically, what is the deep difference between john: #Person & { age: 29 hobbies: [ "physics", "reading", ] } and type Person = { age: number hobbies: string[] } const john: Person = { age: 29 hobbies: [ "physics", "reading", ] } * How do constraints relate with the rest of the language? Specifically, why a limited set of ad-hoc operators `>=18` / `=~ regex` vs (Scala) lambda forms `_ >= 18` / `_ =~ regex` * Where is the line between a data language and a full fledged programming language? For example, the list package, while (intentionally?) missing map / fold, indicates a strong demand for rich list processing capabilities. https://pkg.go.dev/cuelang.org/go@v0.4.0/pkg/list https://pkg.go.dev/cuelang.org/go@v0.4.0/pkg/list
- throwaway894345 5y agoI don't fully have my head around it either, but in my understanding you can have two mutually independent files that define `john: <constraint>` where <constraint> differs in each file, and in Cue these will be unified: the `john` value must satisfy both constraints. For example, one file might have the constraint "john is an int" and the other "john is between 1 and 7". If the constraints clash (e.g., john is an int and john is a string), then it's an error. That said, I think my disconnect is "when is this useful?" or "what can I do with this property?".
- znt 5y agoHow do we handle interdependent validation between different fields? Let's say I got begin date and end date, and I'd like to ensure end date always comes after begin date.
- gregwebs 5y agoThere’s a separate API for dates but for numbers: begin: … end: … #before: < end v: before & begin
- k37dhn3 5y agoHere is a generic example: https://cuelang.org/play/?id=LerWxMboqQQ#cue@export@cue https://cuelang.org/play/?id=LerWxMboqQQ#cue@export@cue Try changing the values of before and after
- verdverm 5y agoMore generally speaking, CUE has overlap with propagator networks. Given x = y + 1 y = x - 1 You can specify either x or y and the other will be solved for. A more powerful PN system can be built on CUE, especially given the DAG solver in the code base
- jsnell 5y agoPrevious discussions: https://news.ycombinator.com/item?id=20362951 https://news.ycombinator.com/item?id=20362951 https://news.ycombinator.com/item?id=20847943 https://news.ycombinator.com/item?id=20847943 https://news.ycombinator.com/item?id=27510919 https://news.ycombinator.com/item?id=27510919
- erezsh 5y ago> For instance, there is no need for generics and enums, sum types and null coalescing are all the same thing. Can anyone qualify that claim? How does one translate type generics to Cue?
- atombender 5y agoI like Cue and Jsonnet and Starlark and so on. But all of these have very low mindshare (though Starlark has the most momentum thanks to Bazel), and who knows if they will be dead by next year. Being an early adopter is difficult both in terms of the immaturity of the tooling — Cue, for example, only has a Go implementation at the moment — and in terms of the risk of betting on an evolutionary dead end, which can cause a lot of unnecessary churn when you want to standardize on something across an entire organization. As a concrete example, I'd love to replace Kubernetes's use of YAML with something like the above. But the tooling is immature, and almost nobody is using any of it. For example, there's Isopod [1], which is a nice-looking tool to use Starlark with Kubernetes. But it might go the same way as Ksonnet. Anyone actually using these tools? [1] https://github.com/cruise-automation/isopod https://github.com/cruise-automation/isopod
- SEJeff 5y agoI use them to manage infrastructure running in Kubernetes. One of my coworker’s previous companies actually funded the cue support in IntelliJ. For kubernetes, it actually introspects the structs and interfaces to create the validation bits. It has a high learning curve but is surprisingly nice once you get the hang of it. There is a lot less duplication due to it in my use.
- asymmetric 5y agoThere are plenty of well supported Jsonnet libraries for Kubrnetes. Have a look at [1] as a starting point. I’ve used it in production with great success, with tanka[2] as the glue . [1]: https://github.com/grafana/jsonnet-libs https://github.com/grafana/jsonnet-libs [2]: https://tanka.dev https://tanka.dev
- verdverm 5y agoGrafana is adopting CUE for dashboard schemas in the mainline. I believe it is internal validation with the intention to expose it to end users
- dqpb 5y agoMy recommendation is to make your choice based on the merits of the options, rather than trying to keep up with the latest trend. It's more satisfying to choose something with less mindshare for your own reason, than something with more mindshare for no reason of your own. Take a look at the logic of cue: https://cuelang.org/docs/concepts/logic/ https://cuelang.org/docs/concepts/logic/ If nothing else, it will make you smarter, and that will make you more conscious, and that will make you more alive.
- fasteo 5y agoStarlark A Starlark interpreter is typically embedded within a larger application, and the application may define additional domain-specific functions and data types beyond those provided by the core language Pretty much what Lua provides. I settled for Lua many years ago.
- yawnxyz 5y agoThis looks super promising, but I'm a coding and Cuelang newbie. I'm confused as to how to use it. Let's say I'm building a tool for people to upload a lot of data, in the form of CSVs and JSON files, POSTing them into my node API. I'm writing those files to a document database. I'm trying to figure out how to use Cue in this process. I'm planning on using JSON Schema definitions, and Ajv as the validator — would I purely be using Cue to generate and maintain my JSON schemas, which I use to validate with Ajv? Or can Cue be used more directly in my node API, e.g. using some npm validation tool that lets me use Cue to validate a bunch of CSV files converted to JSON? Thanks so much in advance!
- 0xFACEFEED 5y agoI think Cue is more aimed at configuration file management. Think Kubernetes, Ansible, Puppet, Chef, and Salt. There you have tens to hundreds to thousands of YAML/JSON files that describe your server configuration and there's no way to validate any of it out of the box. Even more broadly, most configuration files in general don't have any validation. For example things in the /etc/ directory on a Linux server. That fits under configuration file management too. I'm not sure about your specific use case, but if you haven't considered something like Protocol Buffers -- I'd highly recommend at least looking into it. JSON sucks, sometimes.
- verdverm 5y agoExecute is in the name. There is a DAG engine (tool/flow) that automatically calculated dependencies. It powers the scripting layer and https://dagger.io https://dagger.io
- Kinrany 5y agoIn theory you could use CUE to validate requests and insert defaults and implied data, replacing Ajv. 1. Get the CUE value representing the schema endpoint 2. Convert the JSON request into a CUE value 3. Unify the request with the schema If unification fails, return 400 Bad Request. If it succeeds, deserialize the resulting third CUE value into a programming language value. CUE will fill fields missing from JSON by either using the default values from schema or inferring fields that have only one possible value (which may depend on all the other fields of request and schema).
- sandGorgon 5y agothe most interesting summary explanation of cue lang and its differences is from a bug filing - https://github.com/cuelang/cue/issues/33 https://github.com/cuelang/cue/issues/33 >CUE is a bit different from the languages used in linguistics and more tailored to the general configuration issue as we've seen it at Google. But under the hood it adheres strictly to the concepts and principles of these approaches and we have been careful not to make the same mistakes made in BCL (which then were copied in all its offshoots). It also means that CUE can benefit from 30 years of research on this topic. For instance, under the hood, CUE uses a first-order unification algorithm, allowing us to build template extractors based on anti-unification (see issue #7 and #15), something that is not very meaningful or even possible with languages like BCL and Jsonnet.
- heywherelogingo 5y agoI'm afraid, just as some people don't like the name of CockroachDb, which I like, I won't use Cue because of its bottom value (_|_) -- awful.
- verdverm 5y agoWhile the concept of bottom is not going away, the lexical symbol may be replaced with the word. I'm particularly amused that bottom in parentheses looks like the bottom of a human... (_|_)
- corysama 5y agoDoes anyone know enough to compare Cue and https://dhall-lang.org/ https://dhall-lang.org/ ? For example: Is using them together even a sensical idea?
- verdverm 5y agoThe biggest difference is probably CUE is Turing Incomplete, which means no programming or functions. CUE is logical and proves that what you have written is valid and correct. CUE is a superset of JSON, so all JSON is valid CUE. The other main thing is the value lattice where everything from abstract types to concrete data lives. I haven't actually used Dhall, the syntax rubs me the wrong way or something.
- dreamer7 5y agoPotentially naive question - If I had a non-user-facing CRUD admin backend, why should I not use this for validating the data while creating or editing records?
- verdverm 5y agoIf you already have validation, hand tuned and correct, it will perform better. You will have overhead during the encoding to validate in CUE. This may not be a concern, also CUE has some performance goals that should make it a non concern as well. Why not give it a try with some experimentation?
- snassr 5y agoWe are trying an interesting use-case at Overseed (https://www.overseed.io/ https://www.overseed.io/). We recently replaced JSON with CUE as our data configuration language. To summarize, our application allows users to describe their fake data's attributes and behavior and then output that data as a file or stream API. At the moment, we mainly use CUE for the structure, variables, and validation. It replaced our custom JSON and validation logic and cleaned up a lot of our code! We are still exploring moving more of our specifications to CUE style configuration and validation. We are still early-stage users (a few weeks). So far, we love the concise syntax, strong typing, and validation. I plan to spend some time this week checking out the querying, scripting, and external tools. I liked this tutorial site for those who are starting out: https://cuetorials.com/ https://cuetorials.com/
- verdverm 5y agoAre you using CUE from Go? I'm working on the go-api section of Cuetorials right now! Glad you have found the current content helpful.
- wvh 5y agoI played with Cue last year and while I think the concept is interesting, I've been struggling a bit with actually finding a good application for the technology. I'd be more interested in using it for validation as a less brain-dead alternative to e.g. json-schema but the documentation talks mostly about manifest generation and Kubernetes-oriented workflows, which already has Kustomize and Jsonnet and templating solutions like Helm.
- verdverm 5y agoThere is "cue vet" for command line validation. Are you more interested in validation in APIs? There are interesting use cases for ETL. I have written a tool that generates full-stack code using CUE as the declarative input for your types. (https://github.com/hofstadter-io/hof https://github.com/hofstadter-io/hof) For CI/CD from source to production, see https://dagger.io https://dagger.io by the creators of Docker. That is a super cool project. I no longer use Helm now that I use CUE, at least I no longer write charts. We do need a Helm like workflow tool built on CUE to replace the text/templating of Yaml Once "cue mod" is managing deps, there is potential for increased and better sharing of module configuration, much like Helm is for kubernetes applications.
- joshspankit 5y agoAlways respect a demo that teaches me something new about the basics: `cat > file.yaml` Handy!
- kats 5y ago> A key thing that sets CUE apart from its peer languages is that it merges types and values into a single concept. Just use XML...