5 ms·

> the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in

> the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (*specially if its not json/yaml* or just function chain based) I can confirm that having a DSL that is json/yaml helps a ton. Kind of like static type checking, it eliminates entire swaths of syntactical errors, allowing the LLM to focus on the semantics. > because if then you are going to ask the llm to "compile the dsl to implementation" we are back to square 1. I think this is an edge case; 99% of the time you (and/or the LLM) would have access to the implementor so it wouldn't need to do this.

codegladiator 3mo ago
> DSL that is json/yaml helps a ton it definitely does, and i would say json/yaml is not a dsl. this example of json/yaml keeps coming in the form of "DSL". i would say your configuration is not a dsl, it a declaration. llms are better at declarative stuff ? maybe but there are hardly that many of complex declarative frameworks. PlantUML is a real dsl. not just declarative yaml.
- billyp-rva 3mo ago
  > and i would say json/yaml is not a dsl But you can have DSLs that are json/yaml, is my point. > PlantUML is a real dsl. PlantUML is a DSL that isn't json/yaml. That doesn't make it better, and you can make the argument that it is worse because the tooling around it won't be as good.
  - codegladiator 3mo ago
    > you can have DSLs that are json/yaml, is my point i disagree on the semantics of "DSL" vs "config in json". these are different. and anyways both of them just pretend to be "reliable" by throwing the responsibility to an upper layer of validator/compiler/interpreter > it is worse because the tooling around it won't be as good plantuml is good because of the the tooling around it. not sure if we are agreeing/disagreeing there, confused by the wording
  - jazzypants 3mo ago
    That's not a domain-specific language. You can use JSON and YAML in other domains. A DSL is defined by its syntax.
- cestith 3mo ago
  I would say any JSON or YAML that carries semantics with the syntax is a language. It may be a configuration language rather than a programming language, but it’s a language rather than just generic data for some program to slurp.
  - codegladiator 3mo ago
    > any JSON or YAML that carries semantics with the syntax is a language semantics are defined by the converter/compiler/interpreter, and that is the process which is going to consume the said json/yaml. if the json/yaml is going to be consumed by any process then the semantics are inherently defined, so by your definition all jsons/yamls are in their own a "language" (or they are not being consumed at all), which just defeats the purpose of calling it a language at that point.
    - cestith 3mo ago
      At that point you can say C has no semantics, because the compiler defines them.
      - codegladiator 3mo ago
        yeah and so llm cannot write C reliably
ptx 3mo ago
> it eliminates entire swaths of syntactical errors, allowing the LLM to focus on the semantics As I understood it, the idea the article presents is basically the opposite: The LLM is completely clueless about the semantics, but by constraining the language so that semantic errors are syntax errors, we can catch the LLM's bullshit with a simple validation step and make it try again until it produces valid syntax, which limits how wrong it can get the semantics. It does apply to JSON in a way: An LLM doesn't always produce valid JSON, but it's easy to do a syntax check with a JSON parser and make the LLM try again until the JSON is valid. The JSON might still be semantically wrong on the level of the application domain (have the wrong values), but at least you can be sure that the generated data consists of lists, dictionaries, strings and numbers.
- tingletech 3mo ago
  I thought they got like a logit mask that removes illegal next tokens, at least in llama.cpp and GBNF.
  - ptx 3mo ago
    You're probably right about that. It looks like llama.cpp lets you specify a grammar file or a JSON schema - is that what you mean? If the syntax of your DSL can be expressed that way, I suppose you wouldn't have to validate it externally and retry.
    - tingletech 2mo ago
      the grammar support/json-schema applies to a whole response, but I was under the impression a similar logit guide rail can be applied when a tool call is opened.
cestith 3mo ago
Have you looked into YAMLScript?
nextaccountic 3mo ago
the best language by a mile for dsl-as-data-structure is kdl2 it reads like code and you can easily define a schema too well that's what i am using https://kdl.dev/ https://kdl.dev/ and https://crates.io/crates/knus/ https://crates.io/crates/knus/ to define the schema actually more than that. it would be awesome to have a programming language whose code is also valid as kdl 2. like lisp code being composed of s-expressions

codegladiator 3mo ago

> DSL that is json/yaml helps a ton it definitely does, and i would say json/yaml is not a dsl. this example of json/yaml keeps coming in the form of "DSL". i would say your configuration is not a dsl, it a declaration. llms are better at declarative stuff ? maybe but there are hardly that many of complex declarative frameworks. PlantUML is a real dsl. not just declarative yaml.

billyp-rva 3mo ago
> and i would say json/yaml is not a dsl But you can have DSLs that are json/yaml, is my point. > PlantUML is a real dsl. PlantUML is a DSL that isn't json/yaml. That doesn't make it better, and you can make the argument that it is worse because the tooling around it won't be as good.
- codegladiator 3mo ago
  > you can have DSLs that are json/yaml, is my point i disagree on the semantics of "DSL" vs "config in json". these are different. and anyways both of them just pretend to be "reliable" by throwing the responsibility to an upper layer of validator/compiler/interpreter > it is worse because the tooling around it won't be as good plantuml is good because of the the tooling around it. not sure if we are agreeing/disagreeing there, confused by the wording
- jazzypants 3mo ago
  That's not a domain-specific language. You can use JSON and YAML in other domains. A DSL is defined by its syntax.
cestith 3mo ago
I would say any JSON or YAML that carries semantics with the syntax is a language. It may be a configuration language rather than a programming language, but it’s a language rather than just generic data for some program to slurp.
- codegladiator 3mo ago
  > any JSON or YAML that carries semantics with the syntax is a language semantics are defined by the converter/compiler/interpreter, and that is the process which is going to consume the said json/yaml. if the json/yaml is going to be consumed by any process then the semantics are inherently defined, so by your definition all jsons/yamls are in their own a "language" (or they are not being consumed at all), which just defeats the purpose of calling it a language at that point.
  - cestith 3mo ago
    At that point you can say C has no semantics, because the compiler defines them.
    - codegladiator 3mo ago
      yeah and so llm cannot write C reliably

ptx 3mo ago

> it eliminates entire swaths of syntactical errors, allowing the LLM to focus on the semantics As I understood it, the idea the article presents is basically the opposite: The LLM is completely clueless about the semantics, but by constraining the language so that semantic errors are syntax errors, we can catch the LLM's bullshit with a simple validation step and make it try again until it produces valid syntax, which limits how wrong it can get the semantics. It does apply to JSON in a way: An LLM doesn't always produce valid JSON, but it's easy to do a syntax check with a JSON parser and make the LLM try again until the JSON is valid. The JSON might still be semantically wrong on the level of the application domain (have the wrong values), but at least you can be sure that the generated data consists of lists, dictionaries, strings and numbers.

tingletech 3mo ago
I thought they got like a logit mask that removes illegal next tokens, at least in llama.cpp and GBNF.
- ptx 3mo ago
  You're probably right about that. It looks like llama.cpp lets you specify a grammar file or a JSON schema - is that what you mean? If the syntax of your DSL can be expressed that way, I suppose you wouldn't have to validate it externally and retry.
  - tingletech 2mo ago
    the grammar support/json-schema applies to a whole response, but I was under the impression a similar logit guide rail can be applied when a tool call is opened.

cestith 3mo ago

Have you looked into YAMLScript?

nextaccountic 3mo ago

the best language by a mile for dsl-as-data-structure is kdl2 it reads like code and you can easily define a schema too well that's what i am using https://kdl.dev/ https://kdl.dev/ and https://crates.io/crates/knus/ https://crates.io/crates/knus/ to define the schema actually more than that. it would be awesome to have a programming language whose code is also valid as kdl 2. like lisp code being composed of s-expressions