12 ms·

DSLs are a waste of time

95 points by jaxxstorm 3y ago

yegle 3y ago
My favorite example of Terraform being a terrible DSL for infrastructure: code to find ipv6 addresses in a list of v4/v6 addresses [for x in x.addresses: x if length(regexall(".*[:].*", x))>0]
turtleyacht 3y ago
Why don't we have a tool that lets us associate monetary budgets to concrete hardware specs, and then generates a cross-cloud (or cloud-agnostic) configuration? budget: $23000 pricing: cheapest cpu: preferred: xeon cores: 8 mem: 64gb disk: preferred: ssd And some --import-usage flag to specify a folder of historical CPU and memory usage from Grafana.
- sgt 3y ago
  Found a great option for you at $23000. Amazingly we managed to accommodate your budget. /s
  - schoen 3y ago
    This is a nice reminder of the importance of having your software be a "user-agent"!
  - dale_glass 3y ago
    Obviously you don't want to tell AWS what your budget is. You want your local tool that does work for your benefit to have a way to sanity check the configuration, and warn you if the only suitable VM at the provider turns out to cost a lot more than you expected.
    - turtleyacht 3y ago
      Thank-you. > local tool... to sanity check Yes, exactly. Some open-source project as a layer between the cloud providers, and budget and other elements are just knobs to twiddle. If this "open pricing" project could swarm various known constraints between money and virtual hardware, would it be a kind of constraint satisfaction problem?
  - turtleyacht 3y ago
    It does sound foolish to share that directly to the cloud provider. They have an incentive to maximize spend--although they may also provide more than enough oomph with that much spend. It would lie on the payor to collect on all the available discounts (multi-year commitment, etc). Assuming the tool is more or less one step removed from an elaborate spreadsheet and more of a crowdsourced, open optimizer, do you feel there would be any gains to generating multi-cloud setups, or just unnecessary complexity? Is it possible to have a good-faith network of operators reporting monetary info along optimal routes? Routes less like distances, and more like "given your compute and memory profile, a tier 2 storage bucket in AWS and 4gb VM in GCP would fit your budget."
- imtringued 3y ago
  This is backwards. What you want is to specify resource limits and then have the maximum possible bill calculated for you. If you want services to turn off if they exceed a threshold then you will need some sort of priority scheme.
  - turtleyacht 3y ago
    A capped resource limit and/or budget does sound easier to manage. Here is your possible max; after that, shut stuff down in this order. If it were any easier, it would likely be automated as a layer above Infra as Code (IaC). Someone may need to manually check on teams and ensure critical apps are not throttled before less critical ones.
intrasight 3y ago
DSLs are more succinct. The are declarative. They are closer to English. Perhaps if no relational database existed before today, the standard grammar would just be javascript - but I think not.
- jen20 3y ago
  DSLs need not be declarative. DSLs need not be succinct. DSLs need not be close to English at all. All of these are properties which may or may not apply to a _specific_ DSL, they are not defining features.
  - im3w1l 3y ago
    But they can be. They can have nice properties that general purpose languages cannot.
    - jen20 3y ago
      Why not?
      - im3w1l 3y ago
        Well the reasons for each point are a bit different so let's go through them one by one. A declarative language cannot be general purpose by definition. Succinctness. If you know exactly the domain you are talking about it means you can use many shortcuts and abbreviations that would be unclear otherwise. We all know this is true for natural languages, and it is also true for programming. Close to English. In this case the reasons are more about aesthetics. English is not particularly good for general purpose computation as it struggles to express complex ideas with enough precision that it is crystal clear what is meant. A reader must apply some common sense to disambiguate. A general purpose language close to English would either be a Turing tarpit, or it would be a close-to-english language with some general purpose computation things bolted on.
        jen20 3y ago
OJFord 3y ago
I think calling terraform a DSL in comparison to general purpose programming languages misses the point. If it used a declarative GPPL instead of HCL I wouldn't care, but imperative alternatives general purpose or not are 'a waste of time' & not fit for the domain, IMO.
- jen20 3y ago
  CDK for Prolog would be quite the trip. As it turns out, what matters is whether the model is declarative. As some uses of YAML go to show, declarative vs not is an orthogonal axis to how general purpose or otherwise a language is, and orthogonal again to whether it looks like a configuration file or code.
  - ljm 3y ago
    I could see a prolog based solution being quite powerful if used to its advantage. Someone would say…why not hold your state in SQLite and make it queryable that way? And it would be fair. But nowhere near as exciting a problem.
  - OhMeadhbh 3y ago
    CDK for Prolog sounds sick and disgusting. I LIKE IT! I'm a little more familiar with DSLs w/ Lisp, but I built a simple make replacement with Prolog a few decades ago and it turned out to be useful. Replacing CloudFormation or TerraForm with a Prolog based DSL seems like it could be a win. I like Prolog much more than Pulimi in this role as you can succinctly and unambiguously declare dependency relations.
    - jen20 3y ago
      > I like Prolog much more than Pulimi in this role as you can succinctly and unambiguously declare dependency relations. Prolog for Pulumi is an absolutely doable thing - and indeed sounds sufficiently diabolical that someone should do it ;-)
      - OhMeadhbh 3y ago
waffletower 3y ago
This is a great takeaway from the article-- "programming languages are the better authoring model for cloud infrastructure". I was surprised by the simplicity of the "complex" terraform example the author offered, I have seen much thornier Terraform. The title, while appropriate for an article inspired by Terraform, I think misses the benefits DSLs have paricularly in LISP-derived ecosystems. A much better title for the article would definitely be "Terraform(ing) is a Waste of Time"
- PaulHoule 3y ago
  I like DSLs for many things but not for building out cloud infrastructure. It seems like the whole point of those things to build layer of complexity on top of layer of complexity. If any if those things meant you didn’t need to know all the thorny details of bash scripting, charsets, quoting and all, they might be a met positive but as they don’t you might as well write some bash scripts.
- OhMeadhbh 3y ago
  Well... it's part of the cycle. First you create a declarative language to describe WHAT you want the infrastructure planner to generate. Then when your customers discover its limitations, you create a pulumi style imperative "infrastructure builder" language (or library). Then when people complain they have to do everything by hand, you create a different declarative language using whatever transfer syntax is popular that month. Then you keep repeating the cycle to ensure you have something to talk about on your blog every funding cycle. Though... yes... I've had decent luck with DSLs in Lisp-land. It's not a panacea, and there's a couple months of training when we pick up new engineers, but for our app, it seems to work out fine.
phendrenad2 3y ago
Yes DSLs have been a terrible failure in the infrastructure space. But that's just because DSLs are suited to a class of problem where the language of the domain is well-defined and changes infrequently. What we've seen with devops is an ecosystem that changes even more rapidly than frontend software development (in other words, a brakeneck pace). DSLs are great in any area where you want to allow less-technical or untechnical users generate some simple rules. So think finance teams using Microsoft Excel, or video game story writers using in-house DSLs to define game events.
- skywhopper 3y ago
  What specific DSLs are you talking about being failures? Because this is absolutely ludicrous claim.
_hcuq 3y ago
Oops, there goes the first 10 years of my career, where I used DSLs almost exclusively.
OhMeadhbh 3y ago
DSLs aren't a waste of time. But when you create a DSL, you limit the size of the community that will interact with it. So if you create a DSL, make sure that the benefit in expressiveness is worth the down-side of a smaller community.
marcosdumay 3y ago
So, the author considers platform specification languages a waste of time, and has no idea what DSLs are (except that he knows about plartform specification languages)... Yes, platform specification languages are a mess. Making them without the "language" part keeps all of the mess (the language is absolutely not the source of it). There are people trying to fix this, although I'm not sure if I'd place Pulumi on that group. And DSLs are still a great software architecture technique with all kinds of applications.
- AtlasBarfed 3y ago
  DSLs are often the advanced step to a programmer guaranteeing his future employment: 1) don't document 2) obfuscate 3) pick obscure language 4) implement with your own language I will grant that the most succinct representation of a problem is a DSL, but DSLs are inherent obfuscation to some degree. Just because a perfect mathematical representation exists doesn't mean anyone besides the author will understand it inherently
specialist 3y ago
OC is arguing that Terraform is a suboptimal abstraction for infrastructure-as-code. I have no dog in this fight. But I do have opinions about DSLs. The declarative vs imperative, or some hybrid, debate is ongoing with build systems. James Duncan Davidson, author of Ant build system for Java, famously wrote a post-mortem. Mostly wrt to choice of XML for Ant's syntax and semantics. TLDR: Wheels fall off a declarative model once imperative logic is added. FWIW, Here's how I think about declarative DSLs. Such as a scenegraph. You have an exquisite mental model for a scene. It's probably hierarchical. You attempt to capture that mental model in code. The representation is probably a grove data structure. aka Directed Acyclic Graph where nodes can have key/value pairs for metadata. You need to serialize that grove. So you pick a suitable syntax. Hopefully something that looks like VRML-97. Basically s-expressions (JSON without the syntactic vinegar), with some way to represent object prototyping (define/use), and don't forget identifiers ("id" field). Bonus points if you include path expressions in your syntax. That (human readable) serialized grove is the declarative DSL. In an future perfect world, your mental model, runtime representation, serialization format, AND the real world will be isomorphic. Not isomorphic? Of course they're isomorphic. Why even mention it? Funny you should ask. In the Java world, dom4j for representing XML documents, and most of its progeny, is not isomorphic. Only JDOM2's class hierarchy design actually matches the serialization format. (As of the last time I checked, a few years ago.) dom4j's mismatch, leaky abstraction, poor design, what have you, begat untold additional labor and heartache. Which is all completely mooted by JDOM2's correct design. Ensuring correct isomorphism avoids errors in the same way that strong typing and garbage collection do. To wrap this up... In my experience, isomorphism is the exception. I have no idea if Terraform is a good declarative DSL. But I do know that it should be isomorphic with the runtime representation AND the real world. And that should be settled before weaving in imperative functionality.
eduction 3y ago
We stop calling the useful ones “DSLs” so this is a truism. Is SQL a “waste of time?” Regular expressions, HTML, Makefiles, CSS (and CSS selectors aka jquery selectors)? It’s the bad ones that are a waste of time. The ones still called “dsl” instead of just “language, “format,” or “syntax.”
- bob1029 3y ago
  SQL would be my #1 example too. It is probably the ultimate DSL if you are willing to get deviant with your tech stack.
  - OhMeadhbh 3y ago
    Sure. But you can create a declarative DSL without having to copy SQL.
  - throwawaymaths 3y ago
    it's kind of a catch-22, right? SQL is declarative. And while people seem to love declarative DSLs, they are generally really only awesome on the happy path. For SQL, sometimes, it can be incredibly obtuse when something goes horribly wrong (in terms of query speed) -- even with great tools like EXPLAIN. And a query that is fast on one SQL might not be so fast on another SQL. For other declarative DSLs, I've definitely bashed my head against the wall and gotten a few gray hairs over "why the f* doesn't this goddamn thing that should work work". Sometimes it's because I'm not grokking how the platform works. Sometimes it's a bug, or unimplemented feature. I suspect the reason why people don't absolutely abhor SQL is that the biggest players (MySQL, Postgres, Sqlite) have actually done a reasonably good job of making the 98% paths very good and 90% of the rest "good enough".
    - bob1029 3y ago
      > the biggest players (MySQL, Postgres, Sqlite) have actually done a reasonably good job of making the 98% paths very good and 90% of the rest "good enough". Exactly - Unless you are doing something really unusual (or wrong), the vendors of these engines have almost certainly encountered and optimized for some shape approximating your scenario. If you want to get into some extreme ends of the practice, "ancient" engines like DB2 are some of the most capable. Many have been around longer than most developers today have been alive (myself included). That is a lot of optimization legacy to argue against. The halloween problem & iceberg meme comes quickly to mind. Why wouldn't you want to stick with something that has already dealt with all of that bullshit?
gumby 3y ago
Ultimately, what is an API but a domain-specific language? This article could well have been written about that. People often talk about Lisp as a language in which people specify DSLs, but that acronym was something I only encountered outside the Lisp community. The way it always felt to me was that I "simply" designed a set of datastructures and operations around the abstractions and metaphors of the problem domain and then manipulated them to solve my problem. Isn't that the essence of programming in any language?
- kagevf 3y ago
  It seems to be more of a spectrum in Lisps, spanning from syntax enhancements to full blown languages.
fnordpiglet 3y ago
The title is overly broad. They mean a very specific set of infrastructure automation related DSL I tend to agree on that point. I think CDK and related approaches are much easier. I also find just writing SDK based scripts faster and more reliable. I find very little advantage in things like Terraform, despite my driving a few megacorps to adopt it. (Listen, in my defense, I couldn’t sell the idea of just scripting things, and generally they were repurposing admins who had no programming experience and had a mental block against learning a real language - but somehow a configuration file like language was ok) However I’ve written a ton of DSLs in my life. They’re always useful, because I write them when I find it cumbersome to not. The broad statement about DSL is of course absurd. However at one point in my life I built a system that made it easy to write DSLs in a monorepo, and it was a huge mistake. Everyone wrote dsls everywhere and you constantly encountered code in some weird language that you had to reverse engineer. So, there’s a DSL entropy principle to be aware of - DSLs are useful if they are bounded. But like anything too much of a good thing ain’t
deleted 3y ago
[deleted]
pgib 3y ago
This is where Chef got it right over Puppet, but of which were written in Ruby. But Chef recipes are just Ruby files, and they make use of the Ruby language to do the same thing because it's perfectly capable of describing things that way.
cogman10 3y ago
This push and pull on "programming language vs DSL" will be with us forever. Some people love maven because it's a concise DSL, easy to understand if you understand maven. Some people like gradle because it's a DSL with a major escape hatch in that it's also just the groovy programming language with all that entails. This dance has played out with the likes of the Javascript community. Just write a js script, no actually we need a pipeline so let's use grunt. Actually we hate grunt so let's use gulp, actually gulp is just wrong lets get declarative with webpack. Actually that's too inflexible and weedy let's just use a framework (like react-scripts) which just runs our code and gets out of the way. After 10+ years of programming, it seems like we are on an endless loop of "let the framework do everything" to "let's enable maximum flexibility with a full programming language". (see ant in the java world for an even earlier version of this). It seems like the push and pull is this constant problem that "describing the problem is hard, and messing up that description makes everything more complex to maintain". The lack of flexibility feels nice because there's less to mess up, but then when something really complex arises you usually end up doing something 10x more nasty than you would with an ultimate flexibility framework. On the flip side, super flexible frameworks lend themselves to being completely unique butterflies that require a heavy up front research to understand (as, you don't know what that foo method is doing and why it does it). They can 1x the complex situation but it feels like the tendency is to want to overcomplicate simple problems by reinventing a framework for your company. (My company is dealing with this and jenkins pipelines)
- denton-scratch 3y ago
  > After 10+ years of programming Ah - you're a young whippersnapper! I lived through those changes. I never understood why people thought it was a good idea to have a build-pipeline for an interpreted language like Javascript. But my colleagues did; grunt->gulp->laravel->nodejs->react, with a parallel succession of CSS compilers. That was in just the 5 years before I retired. It became a sort of Red Queen race; you have to run just to stand still. I'm glad I'm out of it.
  - em-bee 3y ago
    there are frameworks that can be used without a build-pipeline. but while i have been building websites that way for years, i only recently realized the significance of avoiding the build-pipeline. it's way more than just laziness
joshmarinacci 3y ago
This really rings true for me. I’ve used many DSLs over the last 20+ years. Build systems. UI definition files. Orchestration languages. They always start simple and easy, but they always eventually grow so complex that using a real language would be preferable. I’ve never seen a system remain at the same level of complexity (or much less shrink) over a multi year timeline.
- nly 3y ago
  This is simply because over times systems destructure. Some developer eventually can't be arsed to stop and _think_ for half an hour about how to solve a problem, so proposes replacing the carefully thought out DSL with a Python script. The typical programmer does have any respect for code they didn't write themselves or what it was trying to accomplish.
  - throwawaymaths 3y ago
    I don't think this is the root cause. Declarative languages really paper over details in many cases, and the mapping from the declaration to the imperative execution might not be obvious. When you're on the happy path, it's amazing (if it's a good DSL: because you can be confident you haven't forgotten details, that it's systematically correct, etc). When you encounter a regression, bug, or misunderstanding, in anger, it can be very very difficult to figure out what's going on.
ec109685 3y ago
Hyperbolic blog posts are so annoying. Really, if OpenTF happens, you’ll be back to square one? “ When the maintainers of OpenTF inevitably decide that they want to add enhancements that require language changes, you as the end user end up in a situation where you’re effectively back to square 1, learning (potentially) a new DSL.”
- manojlds 3y ago
  The post is from 2021
  - ec109685 3y ago
    September 4th.
    - manojlds 3y ago
      Oh no, I think I saw the copyright footer or something.
denton-scratch 3y ago
> I was left with an expertise and knowledge of a language that was now, effectively useless. VB did that to me; the lesson I learned is to avoid[0] languages with just a single implementation. As a consequence, I'm not very interested in languages like Ruby, Go and Rust. In the old days, every language had a train of compilers/interpreters following it. Pascal, COBOL, FORTRAN, even ALGOL. I believe that in those days, building a compiler for an existing language was a pretty standard hobby project. Everyone and his dog had built one, and manufacturers bundled their own ones with the hardware. [0] Learning is hard; I jumped from VB to PHP.
- tremon 3y ago
  Ruby has multiple implementations: https://github.com/planetruby/awesome-rubies https://github.com/planetruby/awesome-rubies The main implementations are: - MRI (aka CRuby, the reference implementation) - mruby (lightweight embedded engine) - JRuby (on top of the JVM) - IronRuby (on top of the .Net stack)
- em-bee 3y ago
  not sure why you are making that distinction. sure, a few decades ago, a single implementation meant that a language was not widely used. but that is no longer true today. the biggest example for a widely used single implementation language is probably java. and learning ruby, go or rust surely is not useless
  - TristanBall 3y ago
    The point, I think, is not how widely used it is, the point is that you become beholden to a single, often corporate vendor. I'm not actually sure what made the transition from VB classic to the DotNET version so disruptive, and so damaging to VB in terms of popularity and mindshare ( possibly that C# was better while still sharing enough of the benefits? ) But from everything I saw it was. VB mindshare just fell off a cliff from everything I could see. I mean, you could say similar things about the damage and disruption about perl->raku and python 2->3, which are both effectively single implementation ( ok, less true of python ), but the end result seems very different. Perl5 is still nearly standard in linux/unix, especially anything not embedded or real-time, while raku is niche, although perl5 is more and more niche despite that, with many, many haters.. Python 3 is probably as popular or more so than 2 was VB is an also ran in DotNET languages. So, I dunno, I think there might be more to it that just "single implementation is risky". "Single corporate owner" might be a better one, idk?
TheGRS 3y ago
I feel like I get the author's points pretty well. I had in fact chosen Terraform when it was newer as a greenfield project to deploy our infrastructure more quickly at another company. And I was happy to find out my current company had built all their infrastructure with Terraform, since by that point I knew it really well. But I still struggle with interacting with it, expanding it, and trying to get it to sing to me. The example given is pretty good at demonstrating how Terraform itself has some unique problems that they themselves created: using counts for enabling/disabling resources, using locals as proxies for data comprehension, writing long-ass resource descriptions, often with built-in functions and ternaries to get it to do what you want in various scenarios. Trying to be DRY in Terraform can often be a chore. I think Terraform's MO is actually that some duplicate code isn't the worst thing, so you end up with piles of duplicate lines that are extremely difficult to read, and where writing a function would be super helpful. We had an engineer do a bunch of Terragrunt work for a project and I found that code nigh-incomprehensible. So, I get it as well, I get where they're coming from. I also think working with DSLs isn't a huge waste of time, since many of the things we work on are abstractions of abstractions. It seems like everything I'm doing today is probably going to go away in due time, but I know how to program, read docs, and pivot quickly. I do feel like as long as you're gaining an understanding of the underlying elements, your time spent in a DSL (assuming you're getting paid or you end up using it for several years), is still well spent. They mention Puppet, and I remember getting a job offer once because I was able to read the Puppet documentation and write out a little script that could do the ask in Puppet, even though I had never used it in a professional capacity before. It was easy to pick up and I would've happily worked in it if I accepted the offer. The knowledge I gain around deploying to AWS or other cloud providers is often transferrable in that manner too. The clouds themselves are a sort of DSL often, but I can't say that my time spent with Azure was a waste just because I'm working in AWS now. No, we delivered stuff on that platform, I learned its pros and cons, I gained some wider understanding of what cloud platforms could do, and I got paid along the way :)
pyrolistical 3y ago
This is mainly a complaint about over abstraction and intrinsic complexity. Tools like terraform offer simple modules to make your devops life easy. But simple looking modules abstract a complex thing by hiding details. Details which are critical once your infrastructure reaches a critical complexity threshold. At this point the tool is a burden. It is actively getting in your way to understand and get at the the actual infrastructure you are maintaining. This is why imo alternative like CDK got popular. It doesn’t try to over abstract for you. It just gives you an api you write custom code over. The complexity of your code goes up with the intrinsic complex of the infrastructure.
spion 3y ago
The frontend ecosystem realized (at the time of React) that the (template) DSL isn't really the point. Its the declarative description of what the end result should look like (as well as applying the diff) which is the important bit, not how you generate that declarative description. The devops space is slowly moving in this direction too. TypeScript is a great fit to model the unwieldy yaml / json schemas with precision, getting less in the way compared to most languages with sum types. Deno could be a great player in this space that avoids the complexity of npm, especially with its sandbox to restrict side-effects, networking and external commands. CDKs come with their own multi-language compiler (jsii-rosetta) which results with a ~300MB npm install. Even when all you wanted was to use simple typescript functions to output some JSON and/or yaml.
chubot 3y ago
FWIW this wiki page I started may help frame discussions like this: Survey of Config Languages - https://github.com/oilshell/oil/wiki/Survey-of-Config-Languages https://github.com/oilshell/oil/wiki/Survey-of-Config-Langua... It puts a bunch of languages in 5 cateogires. Terraform is at least not a YAML DSL, a very common pattern which I don't understand the appeal of. I guess the main appeal is that you can copy and paste, and pretend it's data in simple cases. But even Terraform suffers from the problem of "external" DSLs -- you end up needing a general purpose language. And yes this pattern has repeated itself so many times. Awk and Make both grew into general purpose languages, and in Make's case it's particularly bad. Most SQL dialects have also grown a pretty bad stored procedure language. So I do think making internal DSLs in general purpose languages is more promising. But they have to be designed for it. There have been many attempts to use Python as a DSL/config language, but it's missing a few things for that, and gets awkward pretty quickly. Ruby does a bit better, but it's not perfect either. It's fairly big and heavy if you just want a config file.
- pydry 3y ago
  >Terraform is at least not a YAML DSL, a very common pattern which I don't understand the appeal of. I guess the main appeal is that you can copy and paste, and pretend it's data in simple cases. It's easy to write an interpreter for - everything maps to either a scalar, list or a hashmap in any language. It also looks pretty familiar to most people. It gets abused an awful lot but so does every language that gets popular.
  - saltcured 3y ago
    I thing you're getting to the heart of the matter here. I'm not sure if I'm being redundant, because I reloaded a few times and see that _many_ posts have been made in the short time since I started looking at this article and reading comments. But, the discussing seems to be bifurcating into different camps. Do you assume that the blog and discussion are narrowly scoped to sysop/devops problems, or a general diatribe on programming languages and product design? For product design, a big part of the problem is choosing your audience. The blog author mentions being happy with a prior ops DSL because it let him think he's not being a "software developer". In my opinion, to target this type of user is to always be on the treadmill of toy DSLs. You have to over-simplify something to appeal to someone who wants to use it without deeper engagement in a programming/software development mindset. But then you have other users who hit the toy's limits and want more, and you either provide two entirely different products or start compromising the simplicity to add escape hatches. In the end, I think you'll always find that you need more of an API/framework in a proper programming language if you want to support all the use cases with one solution. Or you make a poor approximate of a programming language, quite likely repeating most mistakes in the ~50 year history of scripting languages. There has been a lot of this churn in the sysop/devops space where system administrators are being given more complex and dynamic deployment problems. (Facing their devops mirror-world where programmers are being given more operations duties). This tension also exists in other domains, but I'd say the systems operations space has much more stark culture of "scripting" and "config files" as a distinct thing from "programming" and it shows in the tools and perspectives.
olvy0 3y ago
I agree. I wrote a long rant against DSLs here [0]. I won't repeat it here. This isn't an infrastructure DSL though, just a private DSL for our product. Since then the situation hasn't gotten any better, but sort of stayed the same. Two people on our team have been solving bugs in this DSL for the last few months, rather minor ones. Tomorrow in fact I have a meeting with one of them about adding a sort of limited pass-by-reference semantics into the language (which currently doesn't allow pass-by-reference). I'm very torn about it, as I feel this is a waste of time, and we should ditch this DSL. Users complain rightly that it isn't a "real" programming language. Our "standard library" is very very lacking, for example there's no I/O. We just added FFI 2 months ago. My team lead who was supposed to do this years ago didn't, and in fact effectively stopped developing it 2 years ago, feeling overwhelmed, and ended up leaving a couple of months ago. What this language or rather ecosystem DOES do well is having an internal build system that is completely transparent to the users (unless there are bugs). Well, I designed this build system so I'm partial to it. I had actually written the C# API which I wrote about at the end of [0], that was supposed to be a replacement of this DSL with a "real" language. However I got burned out a couple of months ago after my team lead left, and halted development. There was also a long series of heated discussions with some of my users, who wanted me to remove some of the (rather simple) guardrails I made, and give them the ability to build complex hierarchies above this API, an API which I designed for simplicity. I warned them those hierarchies would end up creating DLL hell and would make it necessary to build devops tooling and maybe even infrastructure as code [1], which IMO is silly for most of the internal projects, which contain at most 5-6 people. From 2 different organizational groups, but still. My users aren't developers per se, and have a more or less support role. I'm still torn as to what the correct direction here. Long term our small team (only 2 full time developers and 3 part time) cannot support 2 half-assed non-standard DSLs, plus that C# API for our many internal users. Something gotta give. [0] https://news.ycombinator.com/item?id=31205265 https://news.ycombinator.com/item?id=31205265 [1] There, I tied this response to the OP!
jamesblonde 3y ago
We have seen this trend in feature engineering. Some companies selling feature stores try to promote their DSL as the way to create features - instead of general purpose frameworks, such as Pandas, Spark, Flink, or event DBT/SQL. In this case, DSLs may have a short-term wow factor, that you don't have to build an maintain feature pipelines, but you're always playing catchup when the latest feature engineering technique is not available in your DSL. And then, you have your developers careers to think of - do they want to put a DSL for feature engineering on their resume?
jauntywundrkind 3y ago
The AWS cdk is a pretty good example of how nice it is to have actual code doing the things. People know JS and know how to use it & it's imminently flexible. By compare every dsl requires learning some new language. The languages seem to grow and grow. I know plenty of people with a little bit of Terraform experience, but almost no one with extensive knowledge. It's great that surfing along the top works as well as it does, and having such a confined language keeps a normativeness, but in general I so strongly feel it's just better to use available known well supported tools.
zamalek 3y ago
Buildah, specifically after struggling with containerfiles/dockerfiles nonsense, entrenched beliefs that agree with the author of the article. Dockerfiles are just an obnoxiously limited DSL that make all the worst assumptions (especially concerning layering), while providing nearly no flexibility. Buildah (when not using buildah bud) lets you write a container build script in the scripting language of your choice - most likely bash - and puts you in charge of layering. I mean, sure, you don't get that automagic in-between later reuse - but I've rarely come across a project that does the magic incantations to get that to work anyway. Rather delegate the rarely changing stuff to a base image, probably updating in CI.
AtlasBarfed 3y ago
Any api is kind of a dsl in the sense you need to figure out how to converse with/invoke it, but at least you can descend into the code to figure out what is happening. A lot of DSLs are a wall to understanding what each "figure of speech" actually does. You need to navigate a parser before you get to the underlying execution of the language
skywhopper 3y ago
Nothing new or interesting in this article. I find the idea that a general purpose programming language is better for infra/config-as-code to be utterly laughable. No one is out there saying DSLs are “easy”. They are just far more constrained in a way that’s often useful. His examples of confusing Terraform are not particularly interesting without some counterexample showing how his preferred CDK-style alternative is better (spoiler: it isn’t any less messy). In fact, CDK/Pulumi-style programming is actually just an imperative-style wrapper that generates the underlying declarative IaC code. You might even call it a reverse DSL. But adding one more layer of abstraction away from what’s actually happening isn’t actually useful. It just adds more seams and more opportunities for leaky abstractions. Using a generic programming language to wrap your declarative infrastructure actually encourages more layers of abstraction. Not a good idea. Could Terraform’s language be improved? Yes. But replacing it with JavaScript or Python is not the answer.
_Algernon_ 3y ago
Read the first couple of paragraphs, realized I have no idea what DSLs are or why I should care about them. Define your abbreviations, people, especially when its not the first hit when googling it. It's not that hard and makes your writing much more accessible.
SonOfLilit 3y ago
Author is arguing that parsed DSLs are worse than embedded DSLs, and he's not wrong pe se - if you have a choice, you should choose parsod DSLs 99% of the time. However, this is not at all an argument against DSLs. Using a "real" language to specify the cloud infrastructure, you'd still want your API to be declarative and ergonomic, i.e. you'd want a DSL.
mvaliente2001 3y ago
The alternative, using general programming languages, is a terrible idea. The last thing I want to do when dealing with a domain specific configuration is trying to figure the meaning out of hundreds of poorly written, badly abstracted, totally undocumented, lines of code.
thrashh 3y ago
The problems with DSLs is usually poor ass documentation. Throwing a full programming language in place of a DSL means now you deal with everyone and their mom inventing their own way of doing things.
tremon 3y ago
Over the years my opinion has grown in the same direction, but with a slightly different slant: domain-specific configuration syntaxes are a waste of time, and domain-specific processing instructions masquerading as a Turing-complete language are always inferior to a real programming language. If you want to represent a data structure, PLEASE pretty please just use an existing data format (json, yaml, toml, xml). Those structures can be parsed and validated by generic tools, and you won't need to write your own domain-specific parsers and syntax highlighters to deal with a problem of your own making. And if you want to allow for runtime evaluation of constructs, PLEASE pretty please just use a programming language that has friendly syntax (lua or python comes to mind). Don't get cute and try to invent your own language, just provide a good API on top.
jesseryoung 3y ago
Programming languages will always be better, more feature rich and more capable than any DSL. However, the whole reason things like terraform or puppet become popular is because the people who write the DSL don't consider themselves "programmers" and they don't want to be thought of as "programmers". As somebody who identifies as a "programmer" and often has to manage infrastructure I would absolutely love for a serious IaS library to pop up in my favorite language. I would switch to it immediately. However unless we start hammering home that if you are regularly writing in these DSLs that you are actually a programmer and there's nothing wrong with that people will still flock to these tools.
- systemvoltage 3y ago
  Very much agree, I do wonder how can things like crontab be specified? The cron job syntax is its own fricking thing that I have to always look up. But then, I wonder, how else can it be specified? yaml file?
  - eternityforest 3y ago
    I've spent way too many hours this week looking for a better model for configurable timers than cron. Cron is kind of awful, not just the syntax but the logic. You can't say "Every three days but not on Monday, if it would land on a Monday, defer that run till the next selected day". There is also too many ways to say the same thing, so GUI editors are hard.
    - systemvoltage 3y ago
      Yeah, basically it needs to be Turing complete but it falls short. Then why not just give us Lua or some real programming language. There is no real need for a cryptic single line syntax.
raffraffraff 3y ago
Hmmm. I just have to strongly disagree. I hit those hard spots in terraform, just like I did in puppet years ago. But I'd still prefer to use a DSL for configuration management (because that's what both are). Puppet actually managed to get to a very usable state by v5, if used with hiera, you could essentially define some types and fling data at them using 'create_resources', which is the equivalent of "for each" over a data set. If you stick to that pattern and make good use of hiera, your codebase will be compact and easy to reason about. The difference between your environments will be a few lines of yaml. In fact I use his with terraform, to that same end. I've looked at pulumi and I just don't like it. Terraform has it's quirks and limitations but I still prefer it. Maybe DSLs are a waste of Lee's time because he knows they'll be "dead languages" when the next major evolution of systems takes place. But meh, I'll learn the next DSL, and maybe even hook hiera up to it too.
julienfr112 3y ago
People, just try pyinfra. Pure python for infrastructure as code. Ansible, terraform, salt, never more.
kwhitefoot 3y ago
Surely all programming is essentially the creation of a domain specific language, or at least a domain specific vocabulary that describes entities in the domain, the actions that they can perform, and the messages that they can send and receive. You can be fluent in Fortran yet be completely bewildered by a Fortran program because the vast bulk of the meaning is in a language that you have yet to learn. As for being left with expertise in a language that no one uses any more, welcome to the club. My first language was Algol-60, the next was Leasco Basic, after that I wrote some Fortran IV, then whatever dialect of Dartmouth Basic was on the Exeter Uni time sharing system, then Z80 assembler, then 6502 assembler, followed by Turbo Pascal in its various versions (I skipped all the 'standard Pascals), more Fortran (but I forget which version) then VB4, 5, 6, then VB.Net then C#. SQL was involved for quite a lot of the time too. Then I retired and now I play with Nim. Were all of those a waste of time too? I certainly have no use for the specifics of Fortran, Algol, Dartmouth Basic, VB.Net, any assembly language, or even C# now, but they were all useful at the time and I created things that were important and valuable for the companies in which I worked.
aranchelk 3y ago
Ultimately learning a DSL is learning a syntax, a vocabulary, a model of execution, and the underlying domain. The syntax shouldn’t take you terribly long unless it was badly made. In my experience virtually any project whether you use a standalone DSL, an embedded DSL, framework, library, or writing something yourself, you will have to understand the underlying domain to be useful. The model of execution can be simple or possibly tricky. There may be a small benefit to having an embedded DSL or framework over a standalone DSL as information learned could be transferable, but model of execution could still be more complicated and different than the underlying language in the other case. I’m thinking of things like Chef, Angular, and Parsec. That leaves the vocabulary which you’ll either need to learn or develop. You’ll take all of the domain knowledge, which as already stated, is the important bit. If you work in the same domain, even if it’s with a very different language, model, etc., you’ll have a great insight into what’s going on. I feel like the opinions expressed in the article really just show a lack of experience of what happens after you’re done working with a piece of tech.
dunk010 3y ago
Steve Yegge made this same point ~20 years ago: > The whole nasty "configuration" problem becomes incredibly more convenient in the Lisp world. No more stanza files, apache-config, .properties files, XML configuration files, Makefiles — all those lame, crappy, half-language creatures that you wish were executable, or at least loaded directly into your program without specialized processing. I know, I know — everyone raves about the power of separating your code and your data. That's because they're using languages that simply can't do a good job of representing data as code. But it's what you really want, or all the creepy half-languages wouldn't all evolve towards being Turing-complete, would they? Via: https://sites.google.com/site/steveyegge2/the-emacs-problem https://sites.google.com/site/steveyegge2/the-emacs-problem
- eternityforest 3y ago
  I'd rather they just fix whatever problem or missing feature caused me to want logic in the config file. Ideally, why do I even need a nontrivial config file at all? I want more stuff that just works, and has standard interfaces to find and connect to all it's other pieces. Even manually executing SQL to make a database is too much. I don't want make files at all. They're only there because C/C++ doesn't handle it for you. Python doesn't need them. Node.js doesn't need them. They have build config stuff, but it's nowhere near as in depth as makefiles. Apache is insane with all the different configurations, and how it's separate from PHP and MySQL and it all has to be configured to work together. Lots of tools just work. They have few options, and all the important configuration options are set at runtime automatically based on input data, auto discovery, and things like that. Install and run. That's it. If I need a full programming language to configure, then whatever I'm doing probably isn't configuration, it's more like plugin writing.
  - dunk010 3y ago
    The point is that the config gets more and more complex, requiring more and more features, until you get a rather ugly turing-complete programming language. And the further point is that with Lisp, you don't need any of that. Code is data. Data is code.
deleted 3y ago
[deleted]
fishnchips 3y ago
Marcin here, one of OpenTF folks. I think the redeeming factor of Terraform is not the language itself, though I really like the fact that it's not a programming language. But I think the redeeming factor is the tool that interprets this language, and the fact that this tool can be taught new tricks - one of which is the CDKTF (not a fan, I think we can do better than that). Here's an analogy. Most folks I know are at least skeptical about Java. The same folks would then praise Kotlin, Scala or Clojure as "good" JVM languages. The relative success of these languages owes much to the Java ecosystem and the JVM interop. That's the reason I find Terraform worth saving. It's the JVM and the Java ecosystem of our cloud world. We can still build great things on it, whether you like HCL or not.
computerfriend 3y ago
> Understanding how to handle complex data structures in say, Python is something which might help you fix bug’s in application code, you can’t throw a DSL at that. True, but it's not an either/or situation. Maybe it's not optimal, but a lot of people do both. I consider myself a Python and Terraform expert and I'd say I reached that stage in Terraform in less than 10% of the time it took for Python.

95 points by jaxxstorm 3y ago

My favorite example of Terraform being a terrible DSL for infrastructure: code to find ipv6 addresses in a list of v4/v6 addresses [for x in x.addresses: x if length(regexall(".*[:].*", x))>0]

turtleyacht 3y ago

Why don't we have a tool that lets us associate monetary budgets to concrete hardware specs, and then generates a cross-cloud (or cloud-agnostic) configuration? budget: $23000 pricing: cheapest cpu: preferred: xeon cores: 8 mem: 64gb disk: preferred: ssd And some --import-usage flag to specify a folder of historical CPU and memory usage from Grafana.

sgt 3y ago
Found a great option for you at $23000. Amazingly we managed to accommodate your budget. /s
- schoen 3y ago
  This is a nice reminder of the importance of having your software be a "user-agent"!
- dale_glass 3y ago
  Obviously you don't want to tell AWS what your budget is. You want your local tool that does work for your benefit to have a way to sanity check the configuration, and warn you if the only suitable VM at the provider turns out to cost a lot more than you expected.
  - turtleyacht 3y ago
    Thank-you. > local tool... to sanity check Yes, exactly. Some open-source project as a layer between the cloud providers, and budget and other elements are just knobs to twiddle. If this "open pricing" project could swarm various known constraints between money and virtual hardware, would it be a kind of constraint satisfaction problem?
- turtleyacht 3y ago
  It does sound foolish to share that directly to the cloud provider. They have an incentive to maximize spend--although they may also provide more than enough oomph with that much spend. It would lie on the payor to collect on all the available discounts (multi-year commitment, etc). Assuming the tool is more or less one step removed from an elaborate spreadsheet and more of a crowdsourced, open optimizer, do you feel there would be any gains to generating multi-cloud setups, or just unnecessary complexity? Is it possible to have a good-faith network of operators reporting monetary info along optimal routes? Routes less like distances, and more like "given your compute and memory profile, a tier 2 storage bucket in AWS and 4gb VM in GCP would fit your budget."
imtringued 3y ago
This is backwards. What you want is to specify resource limits and then have the maximum possible bill calculated for you. If you want services to turn off if they exceed a threshold then you will need some sort of priority scheme.
- turtleyacht 3y ago
  A capped resource limit and/or budget does sound easier to manage. Here is your possible max; after that, shut stuff down in this order. If it were any easier, it would likely be automated as a layer above Infra as Code (IaC). Someone may need to manually check on teams and ensure critical apps are not throttled before less critical ones.

intrasight 3y ago

DSLs are more succinct. The are declarative. They are closer to English. Perhaps if no relational database existed before today, the standard grammar would just be javascript - but I think not.

jen20 3y ago
DSLs need not be declarative. DSLs need not be succinct. DSLs need not be close to English at all. All of these are properties which may or may not apply to a _specific_ DSL, they are not defining features.
- im3w1l 3y ago
  But they can be. They can have nice properties that general purpose languages cannot.
  - jen20 3y ago
    Why not?
    - im3w1l 3y ago
      Well the reasons for each point are a bit different so let's go through them one by one. A declarative language cannot be general purpose by definition. Succinctness. If you know exactly the domain you are talking about it means you can use many shortcuts and abbreviations that would be unclear otherwise. We all know this is true for natural languages, and it is also true for programming. Close to English. In this case the reasons are more about aesthetics. English is not particularly good for general purpose computation as it struggles to express complex ideas with enough precision that it is crystal clear what is meant. A reader must apply some common sense to disambiguate. A general purpose language close to English would either be a Turing tarpit, or it would be a close-to-english language with some general purpose computation things bolted on.
      - jen20 3y ago

OJFord 3y ago

I think calling terraform a DSL in comparison to general purpose programming languages misses the point. If it used a declarative GPPL instead of HCL I wouldn't care, but imperative alternatives general purpose or not are 'a waste of time' & not fit for the domain, IMO.

jen20 3y ago
CDK for Prolog would be quite the trip. As it turns out, what matters is whether the model is declarative. As some uses of YAML go to show, declarative vs not is an orthogonal axis to how general purpose or otherwise a language is, and orthogonal again to whether it looks like a configuration file or code.
- ljm 3y ago
  I could see a prolog based solution being quite powerful if used to its advantage. Someone would say…why not hold your state in SQLite and make it queryable that way? And it would be fair. But nowhere near as exciting a problem.
- OhMeadhbh 3y ago
  CDK for Prolog sounds sick and disgusting. I LIKE IT! I'm a little more familiar with DSLs w/ Lisp, but I built a simple make replacement with Prolog a few decades ago and it turned out to be useful. Replacing CloudFormation or TerraForm with a Prolog based DSL seems like it could be a win. I like Prolog much more than Pulimi in this role as you can succinctly and unambiguously declare dependency relations.
  - jen20 3y ago
    > I like Prolog much more than Pulimi in this role as you can succinctly and unambiguously declare dependency relations. Prolog for Pulumi is an absolutely doable thing - and indeed sounds sufficiently diabolical that someone should do it ;-)
    - OhMeadhbh 3y ago

waffletower 3y ago

This is a great takeaway from the article-- "programming languages are the better authoring model for cloud infrastructure". I was surprised by the simplicity of the "complex" terraform example the author offered, I have seen much thornier Terraform. The title, while appropriate for an article inspired by Terraform, I think misses the benefits DSLs have paricularly in LISP-derived ecosystems. A much better title for the article would definitely be "Terraform(ing) is a Waste of Time"

PaulHoule 3y ago
I like DSLs for many things but not for building out cloud infrastructure. It seems like the whole point of those things to build layer of complexity on top of layer of complexity. If any if those things meant you didn’t need to know all the thorny details of bash scripting, charsets, quoting and all, they might be a met positive but as they don’t you might as well write some bash scripts.
OhMeadhbh 3y ago
Well... it's part of the cycle. First you create a declarative language to describe WHAT you want the infrastructure planner to generate. Then when your customers discover its limitations, you create a pulumi style imperative "infrastructure builder" language (or library). Then when people complain they have to do everything by hand, you create a different declarative language using whatever transfer syntax is popular that month. Then you keep repeating the cycle to ensure you have something to talk about on your blog every funding cycle. Though... yes... I've had decent luck with DSLs in Lisp-land. It's not a panacea, and there's a couple months of training when we pick up new engineers, but for our app, it seems to work out fine.

phendrenad2 3y ago

Yes DSLs have been a terrible failure in the infrastructure space. But that's just because DSLs are suited to a class of problem where the language of the domain is well-defined and changes infrequently. What we've seen with devops is an ecosystem that changes even more rapidly than frontend software development (in other words, a brakeneck pace). DSLs are great in any area where you want to allow less-technical or untechnical users generate some simple rules. So think finance teams using Microsoft Excel, or video game story writers using in-house DSLs to define game events.

skywhopper 3y ago
What specific DSLs are you talking about being failures? Because this is absolutely ludicrous claim.

_hcuq 3y ago

Oops, there goes the first 10 years of my career, where I used DSLs almost exclusively.

OhMeadhbh 3y ago

DSLs aren't a waste of time. But when you create a DSL, you limit the size of the community that will interact with it. So if you create a DSL, make sure that the benefit in expressiveness is worth the down-side of a smaller community.

marcosdumay 3y ago

So, the author considers platform specification languages a waste of time, and has no idea what DSLs are (except that he knows about plartform specification languages)... Yes, platform specification languages are a mess. Making them without the "language" part keeps all of the mess (the language is absolutely not the source of it). There are people trying to fix this, although I'm not sure if I'd place Pulumi on that group. And DSLs are still a great software architecture technique with all kinds of applications.

AtlasBarfed 3y ago
DSLs are often the advanced step to a programmer guaranteeing his future employment: 1) don't document 2) obfuscate 3) pick obscure language 4) implement with your own language I will grant that the most succinct representation of a problem is a DSL, but DSLs are inherent obfuscation to some degree. Just because a perfect mathematical representation exists doesn't mean anyone besides the author will understand it inherently

specialist 3y ago

OC is arguing that Terraform is a suboptimal abstraction for infrastructure-as-code. I have no dog in this fight. But I do have opinions about DSLs. The declarative vs imperative, or some hybrid, debate is ongoing with build systems. James Duncan Davidson, author of Ant build system for Java, famously wrote a post-mortem. Mostly wrt to choice of XML for Ant's syntax and semantics. TLDR: Wheels fall off a declarative model once imperative logic is added. FWIW, Here's how I think about declarative DSLs. Such as a scenegraph. You have an exquisite mental model for a scene. It's probably hierarchical. You attempt to capture that mental model in code. The representation is probably a grove data structure. aka Directed Acyclic Graph where nodes can have key/value pairs for metadata. You need to serialize that grove. So you pick a suitable syntax. Hopefully something that looks like VRML-97. Basically s-expressions (JSON without the syntactic vinegar), with some way to represent object prototyping (define/use), and don't forget identifiers ("id" field). Bonus points if you include path expressions in your syntax. That (human readable) serialized grove is the declarative DSL. In an future perfect world, your mental model, runtime representation, serialization format, AND the real world will be isomorphic. Not isomorphic? Of course they're isomorphic. Why even mention it? Funny you should ask. In the Java world, dom4j for representing XML documents, and most of its progeny, is not isomorphic. Only JDOM2's class hierarchy design actually matches the serialization format. (As of the last time I checked, a few years ago.) dom4j's mismatch, leaky abstraction, poor design, what have you, begat untold additional labor and heartache. Which is all completely mooted by JDOM2's correct design. Ensuring correct isomorphism avoids errors in the same way that strong typing and garbage collection do. To wrap this up... In my experience, isomorphism is the exception. I have no idea if Terraform is a good declarative DSL. But I do know that it should be isomorphic with the runtime representation AND the real world. And that should be settled before weaving in imperative functionality.

eduction 3y ago

We stop calling the useful ones “DSLs” so this is a truism. Is SQL a “waste of time?” Regular expressions, HTML, Makefiles, CSS (and CSS selectors aka jquery selectors)? It’s the bad ones that are a waste of time. The ones still called “dsl” instead of just “language, “format,” or “syntax.”

bob1029 3y ago
SQL would be my #1 example too. It is probably the ultimate DSL if you are willing to get deviant with your tech stack.
- OhMeadhbh 3y ago
  Sure. But you can create a declarative DSL without having to copy SQL.
- throwawaymaths 3y ago
  it's kind of a catch-22, right? SQL is declarative. And while people seem to love declarative DSLs, they are generally really only awesome on the happy path. For SQL, sometimes, it can be incredibly obtuse when something goes horribly wrong (in terms of query speed) -- even with great tools like EXPLAIN. And a query that is fast on one SQL might not be so fast on another SQL. For other declarative DSLs, I've definitely bashed my head against the wall and gotten a few gray hairs over "why the f* doesn't this goddamn thing that should work work". Sometimes it's because I'm not grokking how the platform works. Sometimes it's a bug, or unimplemented feature. I suspect the reason why people don't absolutely abhor SQL is that the biggest players (MySQL, Postgres, Sqlite) have actually done a reasonably good job of making the 98% paths very good and 90% of the rest "good enough".
  - bob1029 3y ago
    > the biggest players (MySQL, Postgres, Sqlite) have actually done a reasonably good job of making the 98% paths very good and 90% of the rest "good enough". Exactly - Unless you are doing something really unusual (or wrong), the vendors of these engines have almost certainly encountered and optimized for some shape approximating your scenario. If you want to get into some extreme ends of the practice, "ancient" engines like DB2 are some of the most capable. Many have been around longer than most developers today have been alive (myself included). That is a lot of optimization legacy to argue against. The halloween problem & iceberg meme comes quickly to mind. Why wouldn't you want to stick with something that has already dealt with all of that bullshit?

gumby 3y ago

Ultimately, what is an API but a domain-specific language? This article could well have been written about that. People often talk about Lisp as a language in which people specify DSLs, but that acronym was something I only encountered outside the Lisp community. The way it always felt to me was that I "simply" designed a set of datastructures and operations around the abstractions and metaphors of the problem domain and then manipulated them to solve my problem. Isn't that the essence of programming in any language?

kagevf 3y ago
It seems to be more of a spectrum in Lisps, spanning from syntax enhancements to full blown languages.

fnordpiglet 3y ago

The title is overly broad. They mean a very specific set of infrastructure automation related DSL I tend to agree on that point. I think CDK and related approaches are much easier. I also find just writing SDK based scripts faster and more reliable. I find very little advantage in things like Terraform, despite my driving a few megacorps to adopt it. (Listen, in my defense, I couldn’t sell the idea of just scripting things, and generally they were repurposing admins who had no programming experience and had a mental block against learning a real language - but somehow a configuration file like language was ok) However I’ve written a ton of DSLs in my life. They’re always useful, because I write them when I find it cumbersome to not. The broad statement about DSL is of course absurd. However at one point in my life I built a system that made it easy to write DSLs in a monorepo, and it was a huge mistake. Everyone wrote dsls everywhere and you constantly encountered code in some weird language that you had to reverse engineer. So, there’s a DSL entropy principle to be aware of - DSLs are useful if they are bounded. But like anything too much of a good thing ain’t

deleted 3y ago

[deleted]

pgib 3y ago

This is where Chef got it right over Puppet, but of which were written in Ruby. But Chef recipes are just Ruby files, and they make use of the Ruby language to do the same thing because it's perfectly capable of describing things that way.

cogman10 3y ago

This push and pull on "programming language vs DSL" will be with us forever. Some people love maven because it's a concise DSL, easy to understand if you understand maven. Some people like gradle because it's a DSL with a major escape hatch in that it's also just the groovy programming language with all that entails. This dance has played out with the likes of the Javascript community. Just write a js script, no actually we need a pipeline so let's use grunt. Actually we hate grunt so let's use gulp, actually gulp is just wrong lets get declarative with webpack. Actually that's too inflexible and weedy let's just use a framework (like react-scripts) which just runs our code and gets out of the way. After 10+ years of programming, it seems like we are on an endless loop of "let the framework do everything" to "let's enable maximum flexibility with a full programming language". (see ant in the java world for an even earlier version of this). It seems like the push and pull is this constant problem that "describing the problem is hard, and messing up that description makes everything more complex to maintain". The lack of flexibility feels nice because there's less to mess up, but then when something really complex arises you usually end up doing something 10x more nasty than you would with an ultimate flexibility framework. On the flip side, super flexible frameworks lend themselves to being completely unique butterflies that require a heavy up front research to understand (as, you don't know what that foo method is doing and why it does it). They can 1x the complex situation but it feels like the tendency is to want to overcomplicate simple problems by reinventing a framework for your company. (My company is dealing with this and jenkins pipelines)

denton-scratch 3y ago
> After 10+ years of programming Ah - you're a young whippersnapper! I lived through those changes. I never understood why people thought it was a good idea to have a build-pipeline for an interpreted language like Javascript. But my colleagues did; grunt->gulp->laravel->nodejs->react, with a parallel succession of CSS compilers. That was in just the 5 years before I retired. It became a sort of Red Queen race; you have to run just to stand still. I'm glad I'm out of it.
- em-bee 3y ago
  there are frameworks that can be used without a build-pipeline. but while i have been building websites that way for years, i only recently realized the significance of avoiding the build-pipeline. it's way more than just laziness

joshmarinacci 3y ago

This really rings true for me. I’ve used many DSLs over the last 20+ years. Build systems. UI definition files. Orchestration languages. They always start simple and easy, but they always eventually grow so complex that using a real language would be preferable. I’ve never seen a system remain at the same level of complexity (or much less shrink) over a multi year timeline.

nly 3y ago
This is simply because over times systems destructure. Some developer eventually can't be arsed to stop and _think_ for half an hour about how to solve a problem, so proposes replacing the carefully thought out DSL with a Python script. The typical programmer does have any respect for code they didn't write themselves or what it was trying to accomplish.
- throwawaymaths 3y ago
  I don't think this is the root cause. Declarative languages really paper over details in many cases, and the mapping from the declaration to the imperative execution might not be obvious. When you're on the happy path, it's amazing (if it's a good DSL: because you can be confident you haven't forgotten details, that it's systematically correct, etc). When you encounter a regression, bug, or misunderstanding, in anger, it can be very very difficult to figure out what's going on.

ec109685 3y ago

Hyperbolic blog posts are so annoying. Really, if OpenTF happens, you’ll be back to square one? “ When the maintainers of OpenTF inevitably decide that they want to add enhancements that require language changes, you as the end user end up in a situation where you’re effectively back to square 1, learning (potentially) a new DSL.”

manojlds 3y ago
The post is from 2021
- ec109685 3y ago
  September 4th.
  - manojlds 3y ago
    Oh no, I think I saw the copyright footer or something.

denton-scratch 3y ago

> I was left with an expertise and knowledge of a language that was now, effectively useless. VB did that to me; the lesson I learned is to avoid[0] languages with just a single implementation. As a consequence, I'm not very interested in languages like Ruby, Go and Rust. In the old days, every language had a train of compilers/interpreters following it. Pascal, COBOL, FORTRAN, even ALGOL. I believe that in those days, building a compiler for an existing language was a pretty standard hobby project. Everyone and his dog had built one, and manufacturers bundled their own ones with the hardware. [0] Learning is hard; I jumped from VB to PHP.

tremon 3y ago
Ruby has multiple implementations: https://github.com/planetruby/awesome-rubies https://github.com/planetruby/awesome-rubies The main implementations are: - MRI (aka CRuby, the reference implementation) - mruby (lightweight embedded engine) - JRuby (on top of the JVM) - IronRuby (on top of the .Net stack)
em-bee 3y ago
not sure why you are making that distinction. sure, a few decades ago, a single implementation meant that a language was not widely used. but that is no longer true today. the biggest example for a widely used single implementation language is probably java. and learning ruby, go or rust surely is not useless
- TristanBall 3y ago
  The point, I think, is not how widely used it is, the point is that you become beholden to a single, often corporate vendor. I'm not actually sure what made the transition from VB classic to the DotNET version so disruptive, and so damaging to VB in terms of popularity and mindshare ( possibly that C# was better while still sharing enough of the benefits? ) But from everything I saw it was. VB mindshare just fell off a cliff from everything I could see. I mean, you could say similar things about the damage and disruption about perl->raku and python 2->3, which are both effectively single implementation ( ok, less true of python ), but the end result seems very different. Perl5 is still nearly standard in linux/unix, especially anything not embedded or real-time, while raku is niche, although perl5 is more and more niche despite that, with many, many haters.. Python 3 is probably as popular or more so than 2 was VB is an also ran in DotNET languages. So, I dunno, I think there might be more to it that just "single implementation is risky". "Single corporate owner" might be a better one, idk?

TheGRS 3y ago

I feel like I get the author's points pretty well. I had in fact chosen Terraform when it was newer as a greenfield project to deploy our infrastructure more quickly at another company. And I was happy to find out my current company had built all their infrastructure with Terraform, since by that point I knew it really well. But I still struggle with interacting with it, expanding it, and trying to get it to sing to me. The example given is pretty good at demonstrating how Terraform itself has some unique problems that they themselves created: using counts for enabling/disabling resources, using locals as proxies for data comprehension, writing long-ass resource descriptions, often with built-in functions and ternaries to get it to do what you want in various scenarios. Trying to be DRY in Terraform can often be a chore. I think Terraform's MO is actually that some duplicate code isn't the worst thing, so you end up with piles of duplicate lines that are extremely difficult to read, and where writing a function would be super helpful. We had an engineer do a bunch of Terragrunt work for a project and I found that code nigh-incomprehensible. So, I get it as well, I get where they're coming from. I also think working with DSLs isn't a huge waste of time, since many of the things we work on are abstractions of abstractions. It seems like everything I'm doing today is probably going to go away in due time, but I know how to program, read docs, and pivot quickly. I do feel like as long as you're gaining an understanding of the underlying elements, your time spent in a DSL (assuming you're getting paid or you end up using it for several years), is still well spent. They mention Puppet, and I remember getting a job offer once because I was able to read the Puppet documentation and write out a little script that could do the ask in Puppet, even though I had never used it in a professional capacity before. It was easy to pick up and I would've happily worked in it if I accepted the offer. The knowledge I gain around deploying to AWS or other cloud providers is often transferrable in that manner too. The clouds themselves are a sort of DSL often, but I can't say that my time spent with Azure was a waste just because I'm working in AWS now. No, we delivered stuff on that platform, I learned its pros and cons, I gained some wider understanding of what cloud platforms could do, and I got paid along the way :)

pyrolistical 3y ago

This is mainly a complaint about over abstraction and intrinsic complexity. Tools like terraform offer simple modules to make your devops life easy. But simple looking modules abstract a complex thing by hiding details. Details which are critical once your infrastructure reaches a critical complexity threshold. At this point the tool is a burden. It is actively getting in your way to understand and get at the the actual infrastructure you are maintaining. This is why imo alternative like CDK got popular. It doesn’t try to over abstract for you. It just gives you an api you write custom code over. The complexity of your code goes up with the intrinsic complex of the infrastructure.

spion 3y ago

The frontend ecosystem realized (at the time of React) that the (template) DSL isn't really the point. Its the declarative description of what the end result should look like (as well as applying the diff) which is the important bit, not how you generate that declarative description. The devops space is slowly moving in this direction too. TypeScript is a great fit to model the unwieldy yaml / json schemas with precision, getting less in the way compared to most languages with sum types. Deno could be a great player in this space that avoids the complexity of npm, especially with its sandbox to restrict side-effects, networking and external commands. CDKs come with their own multi-language compiler (jsii-rosetta) which results with a ~300MB npm install. Even when all you wanted was to use simple typescript functions to output some JSON and/or yaml.

chubot 3y ago

FWIW this wiki page I started may help frame discussions like this: Survey of Config Languages - https://github.com/oilshell/oil/wiki/Survey-of-Config-Languages https://github.com/oilshell/oil/wiki/Survey-of-Config-Langua... It puts a bunch of languages in 5 cateogires. Terraform is at least not a YAML DSL, a very common pattern which I don't understand the appeal of. I guess the main appeal is that you can copy and paste, and pretend it's data in simple cases. But even Terraform suffers from the problem of "external" DSLs -- you end up needing a general purpose language. And yes this pattern has repeated itself so many times. Awk and Make both grew into general purpose languages, and in Make's case it's particularly bad. Most SQL dialects have also grown a pretty bad stored procedure language. So I do think making internal DSLs in general purpose languages is more promising. But they have to be designed for it. There have been many attempts to use Python as a DSL/config language, but it's missing a few things for that, and gets awkward pretty quickly. Ruby does a bit better, but it's not perfect either. It's fairly big and heavy if you just want a config file.

pydry 3y ago
>Terraform is at least not a YAML DSL, a very common pattern which I don't understand the appeal of. I guess the main appeal is that you can copy and paste, and pretend it's data in simple cases. It's easy to write an interpreter for - everything maps to either a scalar, list or a hashmap in any language. It also looks pretty familiar to most people. It gets abused an awful lot but so does every language that gets popular.
- saltcured 3y ago
  I thing you're getting to the heart of the matter here. I'm not sure if I'm being redundant, because I reloaded a few times and see that _many_ posts have been made in the short time since I started looking at this article and reading comments. But, the discussing seems to be bifurcating into different camps. Do you assume that the blog and discussion are narrowly scoped to sysop/devops problems, or a general diatribe on programming languages and product design? For product design, a big part of the problem is choosing your audience. The blog author mentions being happy with a prior ops DSL because it let him think he's not being a "software developer". In my opinion, to target this type of user is to always be on the treadmill of toy DSLs. You have to over-simplify something to appeal to someone who wants to use it without deeper engagement in a programming/software development mindset. But then you have other users who hit the toy's limits and want more, and you either provide two entirely different products or start compromising the simplicity to add escape hatches. In the end, I think you'll always find that you need more of an API/framework in a proper programming language if you want to support all the use cases with one solution. Or you make a poor approximate of a programming language, quite likely repeating most mistakes in the ~50 year history of scripting languages. There has been a lot of this churn in the sysop/devops space where system administrators are being given more complex and dynamic deployment problems. (Facing their devops mirror-world where programmers are being given more operations duties). This tension also exists in other domains, but I'd say the systems operations space has much more stark culture of "scripting" and "config files" as a distinct thing from "programming" and it shows in the tools and perspectives.

olvy0 3y ago

I agree. I wrote a long rant against DSLs here [0]. I won't repeat it here. This isn't an infrastructure DSL though, just a private DSL for our product. Since then the situation hasn't gotten any better, but sort of stayed the same. Two people on our team have been solving bugs in this DSL for the last few months, rather minor ones. Tomorrow in fact I have a meeting with one of them about adding a sort of limited pass-by-reference semantics into the language (which currently doesn't allow pass-by-reference). I'm very torn about it, as I feel this is a waste of time, and we should ditch this DSL. Users complain rightly that it isn't a "real" programming language. Our "standard library" is very very lacking, for example there's no I/O. We just added FFI 2 months ago. My team lead who was supposed to do this years ago didn't, and in fact effectively stopped developing it 2 years ago, feeling overwhelmed, and ended up leaving a couple of months ago. What this language or rather ecosystem DOES do well is having an internal build system that is completely transparent to the users (unless there are bugs). Well, I designed this build system so I'm partial to it. I had actually written the C# API which I wrote about at the end of [0], that was supposed to be a replacement of this DSL with a "real" language. However I got burned out a couple of months ago after my team lead left, and halted development. There was also a long series of heated discussions with some of my users, who wanted me to remove some of the (rather simple) guardrails I made, and give them the ability to build complex hierarchies above this API, an API which I designed for simplicity. I warned them those hierarchies would end up creating DLL hell and would make it necessary to build devops tooling and maybe even infrastructure as code [1], which IMO is silly for most of the internal projects, which contain at most 5-6 people. From 2 different organizational groups, but still. My users aren't developers per se, and have a more or less support role. I'm still torn as to what the correct direction here. Long term our small team (only 2 full time developers and 3 part time) cannot support 2 half-assed non-standard DSLs, plus that C# API for our many internal users. Something gotta give. [0] https://news.ycombinator.com/item?id=31205265 https://news.ycombinator.com/item?id=31205265 [1] There, I tied this response to the OP!

jamesblonde 3y ago

We have seen this trend in feature engineering. Some companies selling feature stores try to promote their DSL as the way to create features - instead of general purpose frameworks, such as Pandas, Spark, Flink, or event DBT/SQL. In this case, DSLs may have a short-term wow factor, that you don't have to build an maintain feature pipelines, but you're always playing catchup when the latest feature engineering technique is not available in your DSL. And then, you have your developers careers to think of - do they want to put a DSL for feature engineering on their resume?

jauntywundrkind 3y ago

The AWS cdk is a pretty good example of how nice it is to have actual code doing the things. People know JS and know how to use it & it's imminently flexible. By compare every dsl requires learning some new language. The languages seem to grow and grow. I know plenty of people with a little bit of Terraform experience, but almost no one with extensive knowledge. It's great that surfing along the top works as well as it does, and having such a confined language keeps a normativeness, but in general I so strongly feel it's just better to use available known well supported tools.

zamalek 3y ago

Buildah, specifically after struggling with containerfiles/dockerfiles nonsense, entrenched beliefs that agree with the author of the article. Dockerfiles are just an obnoxiously limited DSL that make all the worst assumptions (especially concerning layering), while providing nearly no flexibility. Buildah (when not using buildah bud) lets you write a container build script in the scripting language of your choice - most likely bash - and puts you in charge of layering. I mean, sure, you don't get that automagic in-between later reuse - but I've rarely come across a project that does the magic incantations to get that to work anyway. Rather delegate the rarely changing stuff to a base image, probably updating in CI.

AtlasBarfed 3y ago

Any api is kind of a dsl in the sense you need to figure out how to converse with/invoke it, but at least you can descend into the code to figure out what is happening. A lot of DSLs are a wall to understanding what each "figure of speech" actually does. You need to navigate a parser before you get to the underlying execution of the language

skywhopper 3y ago

Nothing new or interesting in this article. I find the idea that a general purpose programming language is better for infra/config-as-code to be utterly laughable. No one is out there saying DSLs are “easy”. They are just far more constrained in a way that’s often useful. His examples of confusing Terraform are not particularly interesting without some counterexample showing how his preferred CDK-style alternative is better (spoiler: it isn’t any less messy). In fact, CDK/Pulumi-style programming is actually just an imperative-style wrapper that generates the underlying declarative IaC code. You might even call it a reverse DSL. But adding one more layer of abstraction away from what’s actually happening isn’t actually useful. It just adds more seams and more opportunities for leaky abstractions. Using a generic programming language to wrap your declarative infrastructure actually encourages more layers of abstraction. Not a good idea. Could Terraform’s language be improved? Yes. But replacing it with JavaScript or Python is not the answer.

_Algernon_ 3y ago

Read the first couple of paragraphs, realized I have no idea what DSLs are or why I should care about them. Define your abbreviations, people, especially when its not the first hit when googling it. It's not that hard and makes your writing much more accessible.

SonOfLilit 3y ago

Author is arguing that parsed DSLs are worse than embedded DSLs, and he's not wrong pe se - if you have a choice, you should choose parsod DSLs 99% of the time. However, this is not at all an argument against DSLs. Using a "real" language to specify the cloud infrastructure, you'd still want your API to be declarative and ergonomic, i.e. you'd want a DSL.

mvaliente2001 3y ago

The alternative, using general programming languages, is a terrible idea. The last thing I want to do when dealing with a domain specific configuration is trying to figure the meaning out of hundreds of poorly written, badly abstracted, totally undocumented, lines of code.

thrashh 3y ago

The problems with DSLs is usually poor ass documentation. Throwing a full programming language in place of a DSL means now you deal with everyone and their mom inventing their own way of doing things.

tremon 3y ago

Over the years my opinion has grown in the same direction, but with a slightly different slant: domain-specific configuration syntaxes are a waste of time, and domain-specific processing instructions masquerading as a Turing-complete language are always inferior to a real programming language. If you want to represent a data structure, PLEASE pretty please just use an existing data format (json, yaml, toml, xml). Those structures can be parsed and validated by generic tools, and you won't need to write your own domain-specific parsers and syntax highlighters to deal with a problem of your own making. And if you want to allow for runtime evaluation of constructs, PLEASE pretty please just use a programming language that has friendly syntax (lua or python comes to mind). Don't get cute and try to invent your own language, just provide a good API on top.

jesseryoung 3y ago

Programming languages will always be better, more feature rich and more capable than any DSL. However, the whole reason things like terraform or puppet become popular is because the people who write the DSL don't consider themselves "programmers" and they don't want to be thought of as "programmers". As somebody who identifies as a "programmer" and often has to manage infrastructure I would absolutely love for a serious IaS library to pop up in my favorite language. I would switch to it immediately. However unless we start hammering home that if you are regularly writing in these DSLs that you are actually a programmer and there's nothing wrong with that people will still flock to these tools.

systemvoltage 3y ago
Very much agree, I do wonder how can things like crontab be specified? The cron job syntax is its own fricking thing that I have to always look up. But then, I wonder, how else can it be specified? yaml file?
- eternityforest 3y ago
  I've spent way too many hours this week looking for a better model for configurable timers than cron. Cron is kind of awful, not just the syntax but the logic. You can't say "Every three days but not on Monday, if it would land on a Monday, defer that run till the next selected day". There is also too many ways to say the same thing, so GUI editors are hard.
  - systemvoltage 3y ago
    Yeah, basically it needs to be Turing complete but it falls short. Then why not just give us Lua or some real programming language. There is no real need for a cryptic single line syntax.

raffraffraff 3y ago

Hmmm. I just have to strongly disagree. I hit those hard spots in terraform, just like I did in puppet years ago. But I'd still prefer to use a DSL for configuration management (because that's what both are). Puppet actually managed to get to a very usable state by v5, if used with hiera, you could essentially define some types and fling data at them using 'create_resources', which is the equivalent of "for each" over a data set. If you stick to that pattern and make good use of hiera, your codebase will be compact and easy to reason about. The difference between your environments will be a few lines of yaml. In fact I use his with terraform, to that same end. I've looked at pulumi and I just don't like it. Terraform has it's quirks and limitations but I still prefer it. Maybe DSLs are a waste of Lee's time because he knows they'll be "dead languages" when the next major evolution of systems takes place. But meh, I'll learn the next DSL, and maybe even hook hiera up to it too.

julienfr112 3y ago

People, just try pyinfra. Pure python for infrastructure as code. Ansible, terraform, salt, never more.

kwhitefoot 3y ago

Surely all programming is essentially the creation of a domain specific language, or at least a domain specific vocabulary that describes entities in the domain, the actions that they can perform, and the messages that they can send and receive. You can be fluent in Fortran yet be completely bewildered by a Fortran program because the vast bulk of the meaning is in a language that you have yet to learn. As for being left with expertise in a language that no one uses any more, welcome to the club. My first language was Algol-60, the next was Leasco Basic, after that I wrote some Fortran IV, then whatever dialect of Dartmouth Basic was on the Exeter Uni time sharing system, then Z80 assembler, then 6502 assembler, followed by Turbo Pascal in its various versions (I skipped all the 'standard Pascals), more Fortran (but I forget which version) then VB4, 5, 6, then VB.Net then C#. SQL was involved for quite a lot of the time too. Then I retired and now I play with Nim. Were all of those a waste of time too? I certainly have no use for the specifics of Fortran, Algol, Dartmouth Basic, VB.Net, any assembly language, or even C# now, but they were all useful at the time and I created things that were important and valuable for the companies in which I worked.

aranchelk 3y ago

Ultimately learning a DSL is learning a syntax, a vocabulary, a model of execution, and the underlying domain. The syntax shouldn’t take you terribly long unless it was badly made. In my experience virtually any project whether you use a standalone DSL, an embedded DSL, framework, library, or writing something yourself, you will have to understand the underlying domain to be useful. The model of execution can be simple or possibly tricky. There may be a small benefit to having an embedded DSL or framework over a standalone DSL as information learned could be transferable, but model of execution could still be more complicated and different than the underlying language in the other case. I’m thinking of things like Chef, Angular, and Parsec. That leaves the vocabulary which you’ll either need to learn or develop. You’ll take all of the domain knowledge, which as already stated, is the important bit. If you work in the same domain, even if it’s with a very different language, model, etc., you’ll have a great insight into what’s going on. I feel like the opinions expressed in the article really just show a lack of experience of what happens after you’re done working with a piece of tech.

dunk010 3y ago

Steve Yegge made this same point ~20 years ago: > The whole nasty "configuration" problem becomes incredibly more convenient in the Lisp world. No more stanza files, apache-config, .properties files, XML configuration files, Makefiles — all those lame, crappy, half-language creatures that you wish were executable, or at least loaded directly into your program without specialized processing. I know, I know — everyone raves about the power of separating your code and your data. That's because they're using languages that simply can't do a good job of representing data as code. But it's what you really want, or all the creepy half-languages wouldn't all evolve towards being Turing-complete, would they? Via: https://sites.google.com/site/steveyegge2/the-emacs-problem https://sites.google.com/site/steveyegge2/the-emacs-problem

eternityforest 3y ago
I'd rather they just fix whatever problem or missing feature caused me to want logic in the config file. Ideally, why do I even need a nontrivial config file at all? I want more stuff that just works, and has standard interfaces to find and connect to all it's other pieces. Even manually executing SQL to make a database is too much. I don't want make files at all. They're only there because C/C++ doesn't handle it for you. Python doesn't need them. Node.js doesn't need them. They have build config stuff, but it's nowhere near as in depth as makefiles. Apache is insane with all the different configurations, and how it's separate from PHP and MySQL and it all has to be configured to work together. Lots of tools just work. They have few options, and all the important configuration options are set at runtime automatically based on input data, auto discovery, and things like that. Install and run. That's it. If I need a full programming language to configure, then whatever I'm doing probably isn't configuration, it's more like plugin writing.
- dunk010 3y ago
  The point is that the config gets more and more complex, requiring more and more features, until you get a rather ugly turing-complete programming language. And the further point is that with Lisp, you don't need any of that. Code is data. Data is code.

deleted 3y ago

[deleted]

fishnchips 3y ago

Marcin here, one of OpenTF folks. I think the redeeming factor of Terraform is not the language itself, though I really like the fact that it's not a programming language. But I think the redeeming factor is the tool that interprets this language, and the fact that this tool can be taught new tricks - one of which is the CDKTF (not a fan, I think we can do better than that). Here's an analogy. Most folks I know are at least skeptical about Java. The same folks would then praise Kotlin, Scala or Clojure as "good" JVM languages. The relative success of these languages owes much to the Java ecosystem and the JVM interop. That's the reason I find Terraform worth saving. It's the JVM and the Java ecosystem of our cloud world. We can still build great things on it, whether you like HCL or not.

computerfriend 3y ago

> Understanding how to handle complex data structures in say, Python is something which might help you fix bug’s in application code, you can’t throw a DSL at that. True, but it's not an either/or situation. Maybe it's not optimal, but a lot of people do both. I consider myself a Python and Terraform expert and I'd say I reached that stage in Terraform in less than 10% of the time it took for Python.