9 ms·
Yml Coding
- tuankiet65 6y agoTrying to turn YAML into a programming language reminds me of Ansible playbooks.
- dhosek 6y agoWe went through great lengths to be able to do parameterized YAML using jinja2 templates. I designed a system that let us effectively create a new yaml out of two other yamls which let us handle repetitive configuration tasks reasonably well but there were occasional unexpected challenges thanks to the syntactically significant white space of yaml.
- hbogert 6y agosincerely, why do this in the first place?
- dhosek 6y agoWe were targetting configurations of Concourse for deploying software. We would have to do essentially the same deployment to multiple environments (dev, qa, staging, performance, prod) and Concourse doesn't provide a sufficiently rich configuration in its use of yaml. For a single app cut and paste is acceptable but once you get into multiple microservices, it becomes more efficient to do it this way.
- q3k 6y agoHave you looked into Jsonnet/CUE/Dhall? They attempt to solve this problem, but give you an actual programming language instead. I've been using Jsonnet to successfully escape the 'yaml-templating-yaml' hell for years now. But yeah, concourse configuration files are probably the worst YAML verbosity offenders, even worse than k8s manifests.
- IshKebab 6y agoThat sounds ridiculous.
- kabacha 6y agoIsn't that what Helm[1] is? 1 - https://helm.sh/ https://helm.sh/
- icameherefor 6y agoUs too (well, along the lines). Did you end up finding some better solution or ended up having a bad programming language with a buggy interpreter in the end (bc of serious time constraints) as well?
- deleted 6y ago[deleted]
- oblio 6y agohttp://ant-contrib.sourceforge.net/tasks/tasks/if.html http://ant-contrib.sourceforge.net/tasks/tasks/if.html
- deleted 6y ago[deleted]
- taylorlapeyre 6y agoI am surprised at how well this actually reads!
- perfunctory 6y agoAnd here we come. Google, in the 21st century, is trying to sell us BASIC - define: assign: - array: ["foo", "ba", "r"] - result: "" - i: 0 - check_condition: switch: - condition: ${len(array) > i} next: iterate next: exit_loop - iterate: assign: - result: ${result + array[i]} - i: ${i+1} next: check_condition - exit_loop: return: concat_result: ${result} edit: it's not April 1st yet, is it?
- recuter 6y agoCome on, this is bound to revolutionise your Kubernetes configuration.
- baq 6y agoi've seen this before, as has my father and probably my grandfather, except it was thought out better and was called lisp instead of yaml.
- mahmoudimus 6y agoI completely agree with this statement and made a similar realization as well. Basically, modern devops and application development boils down to writing Lisp in YAML. Config languages for Ansible, K8S, etc.are just basically a bad implementation of a Lisp-ish language.
- Jtsummers 6y agoIt’s not a lisp. It’s similar to s-expressions, but it is not lisp. It’s a tree based language and that’s the entire similarity. There is nothing else here that is reminiscent of lisp.
- timgilbert 6y agoYeah, among other things Lisp isn't big on labels and GOTO (but BASIC is).
- orf 6y agoI like this. Go "har har har yaml" all you like, but this reads a damn lot better than AWS step functions and seems way more powerful. There are a number of tasks that benefit from something like this, and there are a huge number of advantages from being able to encode steps in domain specific languages such as YAML and have them execute with truly "no code".
- baq 6y agoyaml is not a language. it's a serialization format. this thing here is a language that serializes into yaml. there are zero benefits of using it except perhaps 'it is yaml so i don't have to compile my configuration'. note that xml with xslt would have handle it better.
- gnulinux 6y agoWell, you don't have to write a parser. The parser is already written for you (yaml parser). But I have no idea why that'd be an advantage, especially for Google, they surely know how to write a parser...
- orf 6y ago> there are zero benefits of using it There are many, many benefits from having steps defined in a restrictive, tightly-controlled language that doesn't require sandboxing. There are also many benefits to not using XML and XSLT, not least of all user experience.
- osmarks 6y agoThis is not "no code". This is code, imperative code even, in a really bizarre format.
- orf 6y agoYou're not wrong, but there is always going to be code involved when instructing a machine to perform some actions. By "no code" (and the reason it was in quotes) was that it's almost no superfluous code at all. No libraries to bundle, functions to define, bundles to upload, etc. "I want to call an API, and if it's a Tuesday and the response contains 'foo' trigger another API" seems pretty simple with this. That's not to say it wasn't "simple" before, but there was definitely _more code_.
- abeppu 6y agoI get the appeal of something like this. It's locked down enough that it's perhaps challenging to do something really unsafe with it. It's maybe usable by someone who doesn't feel comfortable with a "real" programming language. But it seems like this falls apart as soon as one service you need to interact with creates a requirement not anticipated by this very constrained tool-set. You need to query service A, extract something with a regex, base64 encode something else before you post to service B? Well we didn't include regexes, a module/import system, or the ability to introduce UDFs in a different language. And if you had the resources to make all your services play into the expectations of this workflow system, you might not need to use this workflow system.
- rightbyte 6y agoI don't see how writing a kinda abstract syntax tree with the opacity of Enterprise JavaBeans is easier than say writing the workflow in some Basic dialect with only GOTOs and IFs.
- dpc_pw 6y agoPeak Yaml?
- throwaway4889 6y agoYaml is the new xml
- IshKebab 6y agoI dunno, XML may have been verbose and overused but at least it had a fairly solid design. YAML is terrible.
- Spivak 6y agoYAML “won” for some definition because it’s really natural to write. Mix colons and brackets, multiline strings that aren’t weird, type annotations when desired, add your own types, basic but functional-ish DRY support that’s transparent to the reader, dict merging. Throw in some value templating a la Ansible and you have a really solid language to express annoying data shapes.
- zo1 6y agoI always saw it as "JSON was the new XML". And JSON Schema was the new XSD. And then they went ahead and also re-invented SOAP using "Swagger" / OpenAPI. Luckily, JSON/YAML is mostly interchangeable these days as it's just a nested hierarchy of a few basic types. Heck, I mostly treat XML the same way as well.
- ses1984 6y agoDid you know that json is actually valid yaml?
- osmarks 6y agoThis seems like some sort of horrible inner platform effect. It would be nice if they would use an actual embeddable programming language, like Lua.
- baq 6y agoturing complete configuration? no thanks. i like it when my configs are guaranteed to halt.
- osmarks 6y agoFair, and there's this neat Dhall (https://dhall-lang.org/ https://dhall-lang.org/) thing for config files (not really this, since it seems like it is designed to actually encode some amount of logic) which enforces that while still being pretty programmable. This might actually already be TC, though; at the very least it is able to do infinite loops.
- SomaticPirate 6y agoDhall really feels like the future of configuration as code. It provides strong typing and outputs to a flat map. It’s still early though and the tooling is a bit ugly. You can embed enough logic that it feels usable. The biggest selling point for me was the typing though.
- zingplex 6y agoI've used Dhall in production, pushed it fairly hard and can say with utmost certainty it's be an absolute pleasure. We use it as our application configuration format and derive fairly intricate Kubernetes resources from our app config.
- Gravityloss 6y agoWe also use Dhall. The home page has a very nice intro indeed. https://dhall-lang.org/ https://dhall-lang.org/
- thangalin 6y agoTangentially related is my YAML pre-processor that performs string interpolation: https://bitbucket.org/djarvis/yamlp/ https://bitbucket.org/djarvis/yamlp/ Along with Pandoc, it allows common prose from Markdown to be de-duplicated, as described in my Typesetting Markdown series: https://dave.autonoma.ca/blog/2019/07/06/typesetting-markdown-part-5/#interpolated https://dave.autonoma.ca/blog/2019/07/06/typesetting-markdow... Obviously it's laborious insert to YAML keys everywhere, so I'm developing an editor that integrates YAML and plain text document formats (such as Markdown): https://www.youtube.com/watch?v=u_dFd6UhdV8 https://www.youtube.com/watch?v=u_dFd6UhdV8
- jpxw 6y agoEven writing CI configuration is onerous enough in YAML, let alone a program
- wickedOne 6y agodespite the readability of the yaml syntax, for me the big disadvantage is the inability to use meta / type info. for a straightforward collection of strings: probably good enough, for anything else xml is more likely to cover the usecase better
- Spivak 6y agoYAML has really good typing support with type annotations being a first class citizen. But you’re dependent on the hosting software (adding or understanding) the types. For example start: !date 2020-09-12 could actually map to a native date object.
- bonestormii_ 6y agoData is data. Code is logic. So the the thought process goes... ...what if I could store my configuration (data! right?!) in a way that is nicely separated from the logic?! No more scripting for me! Oh wait. Some of the configuration can't be generalized about universally. Configurations fundamentally contain logic, I guess. Oh, well then why don't I just represent the logic in the data?! That will be much better than representing the data in the logic! ....but now, you are back where you started, only instead of using something nice, standard, and powerful like Python, you have to use this... language... thing. This YAML convention you cooked up. Separate the data from logic. Read the data into the logic. Make a nice organized place to call custom logic from... like, you know, a file directory full of scripts, which are called according to some scheme. It could be another data file. Stop there. Like this: $ ls -Ra ./config/do_something.yaml ./config/do_something_else.yaml ./config/config.yaml ./logs/ping.log ./scripts/do_something.py ./scripts/do_something_else.py --- $ cat ./config/config.yaml do_something: target: my.stupid.server exec_frequency: daily ... do_something_else: target: my.stupid.server exec_frequency: monthly ... --- $ cat ./config/do_something.yaml ping: true output: directory: ./logs/ping.log mode: append --- $ cat ./config/do_something_else.yaml delete: ./logs/ping.log --- $ cat ./scripts/do_something.py from stupid.library.task import Task, subscribe class DoSomething(Task): @subscribe def ping(self, target): return super().ping(target) Then you code the program that runs collects all of the task methods, put them in an ordered list, and run them if the conditions in the config.yaml is met. Every other thing is done in a task method in python, or something like it. I don't think we need more abstraction than that.
- pydry 6y agoThis is more or less the structure I followed with this testing framework: https://github.com/hitchdev/hitchstory https://github.com/hitchdev/hitchstory The whole idea being that you don't want the story to be turing complete (there are no loops or conditionals with a story), but the code that executes it needs to be turing complete.
- bonestormii_ 6y ago
- xiaq 6y agoSaying this is reinventing BASIC in YAML seems to miss the point here; the programming model here is essentially a finite state machine with optional data flowing between nodes, and AFAIK there isn't really a widespread language that targets this model. And a restricted model is not there just so that non-programmers can use it. Restricting what you can do in the program means that there are more things you can do with the program. I haven't checked if GCP's Workflows supports everything below, but here are some things you can do in principle: * You can visualize the entire program as a flowchart, and visualize the state of the program simply by pointing at a node in the flowchart. This is not possible with a general purpose language since there could be arbitrary levels of call frames. * You can implement retry policies for each step entirely transparently, and possibly other things like authentication. Aspect-oriented programming is more practical when the programming model is restricted. * You can schedule the steps onto different hosts, possibly in parallel. * You can suspend and resume the workflow, since its entire state is just which step is being executed, plus a handful of variables (which presumably are always serializable). Re the problem of extension: the idea seems to be that you put all "smartness" inside HTTP services that are written in real languages and only use this as a dumb glue language.
- mfateev 6y agoLook at my project temporal.io. It does the same thing using general purpose programming languages. Currently Java and Go are supported with Python, Ruby and C# under development. This is absolutely valid workflow code: public void execute(String customerId) { activities.onboardToFreeTrial(customerId); try { Workflow.sleep(Duration.ofDays(60)); activities.upgradeFromTrialToPaid(customerId); while (true) { Workflow.sleep(Duration.ofDays(30)); activities.chargeMonthlyFee(customerId); } } catch (CancellationException e) { activities.processSubscriptionCancellation(customerId); } }
- xiaq 6y agoBut what this does under the hood is still sending instructions to a workflow engine that does the actual work, as opposed to this code doing the actual work directly, right? If that's the case, then what you could do is still restricted by the protocol of the workflow engine - building the instructions in code gives you some dynamicism, but not a whole world of difference, and complicates things that are easier done statically. It is definitely a valid approach, but it doesn't invalidate the approach of writing out the workflow definition statically, especially if the paradigm is "put all smartness inside HTTP service and only use the workflow as a glue".
- cordite 6y agoHow about we use resumable lua scripting
- swiley 6y agoIMO: most of these weird DSLs come from wanting a data structure but getting confused and writing a bad programming language instead.
- jpalomaki 6y agoDo they have a graphical tool for defining these workflows? For me this looks like the business process management tools from the past. With those the idea was not to use them to write program logic, but connect the dots on very high level. See for example the use case pictures at [1]. [1] https://cloud.google.com/workflows https://cloud.google.com/workflows
- fmakunbound 6y agoGood god it’s stuff like this makes me contemplate leaving professional programming.