7 ms·

The main challenge I've observed with DSLs is maintaining and growing them over time. It's not unusual for a DSL to grow organically based on individual use cas

by bbrazil 11y ago

The main challenge I've observed with DSLs is maintaining and growing them over time. It's not unusual for a DSL to grow organically based on individual use cases that seem like a good idea, and the end result is a language that's not quite consistent or orthogonal. "A foolish consistency is the hobgoblin of little minds" is the rallying cry of those who only have to deal with a small number of libraries/languages/tools/systems, and don't realise the true externalities of having to memorise hundreds to thousands of special cases. Small consistent languages are good languages.

sklogic 11y ago
To avoid this issue DSLs must be highly modular, which is easily achivable if you're implementing them as eDSLs on top of a meta-language (i.e., mixing DSLs is trivial and interop is not a problem).
- bbrazil 11y ago
  I'm not following you. I'm thinking of things like https://github.com/prometheus/prometheus/issues/482 https://github.com/prometheus/prometheus/issues/482 where core elements of a language don't behave in a predictable manner (in this case == has two distinct modes). That's a simple example, more structural issues can be compound by workarounds and lead to a lot of complexity growing over time.
  - sklogic 11y ago
    In order for DSLs not to evolve into a mess they must be made of high quality semantic components, which you simply mix together and alter syntax a bit in order to get a new DSL. When you develop your DSLs this way the problems you're talking about are nearly impossible to run into. Essentially, if you build DSLs on top of an elaborate hierarchy of DSLs, every single DSL adds only a tiny bit of functionality which makes it maintainable and protects from evolving into a pile of messy layers. If you have any questions about technical details of this approach - feel free to ask.
    - nmrm2 11y ago
      > they must be made of high quality semantic components, which you simply mix together > the problems you're talking about are nearly impossible to run into. If you mean this in any non-fluffy sense, such a property would depend upon the set of semantic components having some incredibly strong properties. It's not clear to me that those properties don't immediately conflict with the entire point of creating DSLs -- that the host language's semantic model isn't the one you want. You can probably get trivial properties about the static semantics without a ton of machinery, and more dynamic properties might be possible to read off from very constrained examples. But it's entirely unclear to me why semantic models should easily combine without creating unintended behavior. Could you share some concrete examples?
      - sklogic 11y ago
        > such a property would depend upon the set of semantic components having some incredibly strong properties. Why? You can have as many semantic building blocks as you like. My current tool belt contains, among the others, a generic high performance imperative backend, generic functional backend, generic typing engine (which easily include various Hindley-Milner implementations as well as simple type propagation forms), generic lazy functional, generic Prolog and Datalog - these are the low level semantic building blocks which can be mixed together in any proportion. On top of them there is a large hierarchy of high level building blocks, which include, for example, generic optimisation components (e.g., IR-agnostic SSA-based optimisations), generic parsing frontends, data representation and transformation languages, and many more. > But it's entirely unclear to me why semantic models should easily combine without creating unintended behavior. Why would they? As long as the interop rules are clear, there is no problem in mixing different things together. > Could you share some concrete examples? Take a look at my github repo (linked in some other comments, do not want to keep spamming it).
        nmrm2 11y ago
        > Why? Because I care about soundness. And especially soundness for combinations of under-specified semantic models, because eDSLs are useful exactly when the semantic models at hand aren't 30-60 years old with mature implementations. > imperative backend, generic functional backend, generic typing engine (which easily include various Hindley-Milner implementations as well as simple type propagation forms), generic lazy functional... which can be mixed together in any proportion. That's just so totally not true. There are a ridiculous number of ways to combine these things in ways that totally kill type soundness, confluence, etc. The reason you didn't have a hard time combining them is because they are 1) really truly extraordinarily well-understood and well-defined languages; 2) we've known exactly how to combine these things for a really long time; and 3) you knew a priori the reasonable interaction points. None of these three assumptions are true for most of the eDSLs engineers might want to write down in general. In general, a framework that lets you get unsound bullcrap from two perfectly sound things isn't a solution to the (e)DSL problem.
ExpiredLink 11y ago
Not only maintaining and growing a DSL is challenging but also designing it at first. You need a lot of both domain and technical experience to achieve a half-decent result. That's why the "hype" around 2007 didn't lead to widespread practice. DSLs are costly.
- scottious 11y ago
  I'm currently developing a DSL. I definitely agree with what you're saying. Domain and technical expertise are needed in abundance. Writing a spec is HARD. It takes a certain perseverance to learn good parser design (I've written a great parser generator and I still feel inadequate in this area sometimes). DSLs are costly. My concern is that the cost will outweigh the benefit and people won't like it and we'll be suck with it. My hope is that it'll stabilize and be expressive enough to fit everybody's needs. Did you ever write a DSL used in a production environment? How complex was the DSL, more declarative or closer to a general purpose language?
  - sklogic 11y ago
    > Writing a spec is HARD. Chances are that you're severely over-engineering it. Just take the English specification for any particular problem your DSL is supposed to solve and slightly tweak it into looking like a formal language. That's it. Extremely easy with a bit of practice.
    - ExpiredLink 11y ago
      You have no idea.
      - sklogic 11y ago
        Yes, tell me more. I build eDSLs professionally. You can take a look at some of the stuff I'm building: https://github.com/combinatorylogic https://github.com/combinatorylogic
        11y ago
taeric 11y ago
If the language reflects the domain of the problem being solved, and it still grows; not sure there is really a better way. Artificially constraining it to a consistent API is just going to get you into the same problem many functional advocates are having. Which is the confusion of everything being covered by a few names.

by bbrazil 11y ago

sklogic 11y ago

To avoid this issue DSLs must be highly modular, which is easily achivable if you're implementing them as eDSLs on top of a meta-language (i.e., mixing DSLs is trivial and interop is not a problem).

bbrazil 11y ago
I'm not following you. I'm thinking of things like https://github.com/prometheus/prometheus/issues/482 https://github.com/prometheus/prometheus/issues/482 where core elements of a language don't behave in a predictable manner (in this case == has two distinct modes). That's a simple example, more structural issues can be compound by workarounds and lead to a lot of complexity growing over time.
- sklogic 11y ago
  In order for DSLs not to evolve into a mess they must be made of high quality semantic components, which you simply mix together and alter syntax a bit in order to get a new DSL. When you develop your DSLs this way the problems you're talking about are nearly impossible to run into. Essentially, if you build DSLs on top of an elaborate hierarchy of DSLs, every single DSL adds only a tiny bit of functionality which makes it maintainable and protects from evolving into a pile of messy layers. If you have any questions about technical details of this approach - feel free to ask.
  - nmrm2 11y ago
    > they must be made of high quality semantic components, which you simply mix together > the problems you're talking about are nearly impossible to run into. If you mean this in any non-fluffy sense, such a property would depend upon the set of semantic components having some incredibly strong properties. It's not clear to me that those properties don't immediately conflict with the entire point of creating DSLs -- that the host language's semantic model isn't the one you want. You can probably get trivial properties about the static semantics without a ton of machinery, and more dynamic properties might be possible to read off from very constrained examples. But it's entirely unclear to me why semantic models should easily combine without creating unintended behavior. Could you share some concrete examples?
    - sklogic 11y ago
      > such a property would depend upon the set of semantic components having some incredibly strong properties. Why? You can have as many semantic building blocks as you like. My current tool belt contains, among the others, a generic high performance imperative backend, generic functional backend, generic typing engine (which easily include various Hindley-Milner implementations as well as simple type propagation forms), generic lazy functional, generic Prolog and Datalog - these are the low level semantic building blocks which can be mixed together in any proportion. On top of them there is a large hierarchy of high level building blocks, which include, for example, generic optimisation components (e.g., IR-agnostic SSA-based optimisations), generic parsing frontends, data representation and transformation languages, and many more. > But it's entirely unclear to me why semantic models should easily combine without creating unintended behavior. Why would they? As long as the interop rules are clear, there is no problem in mixing different things together. > Could you share some concrete examples? Take a look at my github repo (linked in some other comments, do not want to keep spamming it).
      - nmrm2 11y ago
        > Why? Because I care about soundness. And especially soundness for combinations of under-specified semantic models, because eDSLs are useful exactly when the semantic models at hand aren't 30-60 years old with mature implementations. > imperative backend, generic functional backend, generic typing engine (which easily include various Hindley-Milner implementations as well as simple type propagation forms), generic lazy functional... which can be mixed together in any proportion. That's just so totally not true. There are a ridiculous number of ways to combine these things in ways that totally kill type soundness, confluence, etc. The reason you didn't have a hard time combining them is because they are 1) really truly extraordinarily well-understood and well-defined languages; 2) we've known exactly how to combine these things for a really long time; and 3) you knew a priori the reasonable interaction points. None of these three assumptions are true for most of the eDSLs engineers might want to write down in general. In general, a framework that lets you get unsound bullcrap from two perfectly sound things isn't a solution to the (e)DSL problem.

ExpiredLink 11y ago

Not only maintaining and growing a DSL is challenging but also designing it at first. You need a lot of both domain and technical experience to achieve a half-decent result. That's why the "hype" around 2007 didn't lead to widespread practice. DSLs are costly.

scottious 11y ago
I'm currently developing a DSL. I definitely agree with what you're saying. Domain and technical expertise are needed in abundance. Writing a spec is HARD. It takes a certain perseverance to learn good parser design (I've written a great parser generator and I still feel inadequate in this area sometimes). DSLs are costly. My concern is that the cost will outweigh the benefit and people won't like it and we'll be suck with it. My hope is that it'll stabilize and be expressive enough to fit everybody's needs. Did you ever write a DSL used in a production environment? How complex was the DSL, more declarative or closer to a general purpose language?
- sklogic 11y ago
  > Writing a spec is HARD. Chances are that you're severely over-engineering it. Just take the English specification for any particular problem your DSL is supposed to solve and slightly tweak it into looking like a formal language. That's it. Extremely easy with a bit of practice.
  - ExpiredLink 11y ago
    You have no idea.
    - sklogic 11y ago
      Yes, tell me more. I build eDSLs professionally. You can take a look at some of the stuff I'm building: https://github.com/combinatorylogic https://github.com/combinatorylogic
      - 11y ago

taeric 11y ago

If the language reflects the domain of the problem being solved, and it still grows; not sure there is really a better way. Artificially constraining it to a consistent API is just going to get you into the same problem many functional advocates are having. Which is the confusion of everything being covered by a few names.