4 ms·

The problem with writing a _general-purpose_ programming language (as a solo-developer), is the fact that writing the language (i.e defining the syntax+semantic

The problem with writing a _general-purpose_ programming language (as a solo-developer), is the fact that writing the language (i.e defining the syntax+semantics, writing the parser/compiler) is only half the battle - the rest of your time is spent on less technically interesting QoL features, like an IDE, standard library, debugger, syntax highlighting etc. which are nonetheless vital for making the programming language actually usable. (I still think writing your own PL is a useful exercise just to see how the sausage is made, but it should be made clear that there is a world of difference between a toy language and one that people would actually want to use) It's for this reason that nowadays I rarely write my own "new" programming languages from scratch, but rather embed them as DSLs inside existing programming languages using macros. This way you get the full ecosystem of the host language for free, leaving you more time to work on designing the language to suit your needs. If you combine macros with clever uses of GADTs you can even get the host language to "type-check" programs written in your DSL. OCaml and Lisp/Racket are my favourite host languages to use for this approach.

pjmlp 5y ago
Exactly, the days that a plain syntax+semantics, with parser/compiler were enough to win the hearts of the market are long gone, decades ago. It is still a fun exercise, though.
- ModernMech 5y ago
  Yeah it's pretty bad these days. If you tell someone you are writing a PL, they expect: - a complete parser/compiler and debugger toolchain - a complete language server implementation, and language modes for all popular IDEs including VSCode, VIM, Emacs, etc. - a package manager with a large, secure, and vibrant package ecosystem - a community with live chat support and a strong Stack Overflow presence - robust documentation with copious examples, tutorials, guides, and technical info. These days even a multi-hundred page book with professional editing available for free is expected. - That you will be responsive to bugs, provide patches promptly, and review/merge community feature requests asap. They will get very mad with you if you don't, and likely write a blog post about how much of a tyrant you are over your project. - your language is expected to have undergone a full independent security audit, and to be up-to-date on the latest security issues. It's expected to be stable with complete backwards compatibility. - you have to manage the community and deal with petty interpersonal conflicts or else word gets out that your language has a "toxic community" - support for all major operating systems and hardware architectures including Mac, Windows, and all distributions of Linux, as well as esoteric hardware. - support for the web through wasm, so you need experts at that as well as x86 and ARM platforms now. - And on top of all that, your project needs to be completely open source, with a free and open license that permits royalty free, patent free, commercial use. Oh, and your users both corporate and personal expect your work to be completely free as in beer. They won't pay anything for it. Not even a dollar. And if you ask them to pay something they don't weigh the value of your product versus the competition, they balk immediately and don't even consider it. It's been a non-starter to ask for money for a PL for decades. It used to be the case that you could write down a spec on paper and call it a PL. Back in the 60s you could call that a "programming language" without even writing an implementation of the parser. Today a programming language is a massive undertaking and the saddest part of it all is that it's been commoditized. I've even seen a sentiment out there that programming languages are done, that we've invented all we need to. All future languages are unnecessary, and all current languages are largely the same. Could you imagine? So as hard as programming languages are, DO write them. That's my message. Just do so responsibly and with clear eyes and expectations.
  - pjmlp 5y ago
    At very least, because the baton eventually needs to be passed on to younger generations. Regarding language innovation, I will oversimplify and assert that due the resistance to modernization (for whatever reasons), we have spent the last decades making Algol 68, Lisp and ML appealing to mainstream, smuggling their capabilities into more acceptable packaging.
pydry 5y ago
Ecosystem of packages and language interop (e.g. C bindings) are also criminally underrated.
mathgladiator 5y ago
We are in a golden age for new languages. You have https://microsoft.github.io/language-server-protocol/ https://microsoft.github.io/language-server-protocol/ for doing the integration between IDE and your parser. Also, https://highlightjs.org/ https://highlightjs.org/ which lets you broadcast your code in a pretty way. I do however feel that your QoL will depend on whether or not you make a compiler or a translator. The key benefit of a translator is that you can co-opt the host language's standard runtime and debugger.
- brundolf 5y ago
  Agree with all of this. Don't do any yak-shaving you don't want to do just for the fun of it; build your language and its tooling on existing systems so you can spend your time on what makes it special, instead of what doesn't.
ModernMech 5y ago
> It's for this reason that nowadays I rarely write my own "new" programming languages from scratch, but rather embed them as DSLs inside existing programming languages using macros. This way you get the full ecosystem of the host language for free I don't understand what you mean by this, so I'm hoping you can clarify. What's the advantage of doing this over the alternative way: writing a parser which would compile to code in the host language, or some primitives that you've arranged in the host language that would represent the bytecode of your language interpreter. From there you can leverage whatever in the host language you want, including its type system, packages, language semantics, FFI access to other languages. Really whatever you like. What you're suggesting is leveraging a macro system to help you write what is essentially a parser/compiler, which is a good win. But I don't see what that buys you past that point. If I do things the other way, by writing a parser that consumes a *.my_lang file and compiles it to primitives or actual source code in the host language, I can still leverage all the benefits of the host language just as you describe doing with a macro system. The advantage of doing things this way is you're not limited to languages with tolerable metaprogramming capabilities.

pjmlp 5y ago

Exactly, the days that a plain syntax+semantics, with parser/compiler were enough to win the hearts of the market are long gone, decades ago. It is still a fun exercise, though.

ModernMech 5y ago
Yeah it's pretty bad these days. If you tell someone you are writing a PL, they expect: - a complete parser/compiler and debugger toolchain - a complete language server implementation, and language modes for all popular IDEs including VSCode, VIM, Emacs, etc. - a package manager with a large, secure, and vibrant package ecosystem - a community with live chat support and a strong Stack Overflow presence - robust documentation with copious examples, tutorials, guides, and technical info. These days even a multi-hundred page book with professional editing available for free is expected. - That you will be responsive to bugs, provide patches promptly, and review/merge community feature requests asap. They will get very mad with you if you don't, and likely write a blog post about how much of a tyrant you are over your project. - your language is expected to have undergone a full independent security audit, and to be up-to-date on the latest security issues. It's expected to be stable with complete backwards compatibility. - you have to manage the community and deal with petty interpersonal conflicts or else word gets out that your language has a "toxic community" - support for all major operating systems and hardware architectures including Mac, Windows, and all distributions of Linux, as well as esoteric hardware. - support for the web through wasm, so you need experts at that as well as x86 and ARM platforms now. - And on top of all that, your project needs to be completely open source, with a free and open license that permits royalty free, patent free, commercial use. Oh, and your users both corporate and personal expect your work to be completely free as in beer. They won't pay anything for it. Not even a dollar. And if you ask them to pay something they don't weigh the value of your product versus the competition, they balk immediately and don't even consider it. It's been a non-starter to ask for money for a PL for decades. It used to be the case that you could write down a spec on paper and call it a PL. Back in the 60s you could call that a "programming language" without even writing an implementation of the parser. Today a programming language is a massive undertaking and the saddest part of it all is that it's been commoditized. I've even seen a sentiment out there that programming languages are done, that we've invented all we need to. All future languages are unnecessary, and all current languages are largely the same. Could you imagine? So as hard as programming languages are, DO write them. That's my message. Just do so responsibly and with clear eyes and expectations.
- pjmlp 5y ago
  At very least, because the baton eventually needs to be passed on to younger generations. Regarding language innovation, I will oversimplify and assert that due the resistance to modernization (for whatever reasons), we have spent the last decades making Algol 68, Lisp and ML appealing to mainstream, smuggling their capabilities into more acceptable packaging.

pydry 5y ago

Ecosystem of packages and language interop (e.g. C bindings) are also criminally underrated.

mathgladiator 5y ago

We are in a golden age for new languages. You have https://microsoft.github.io/language-server-protocol/ https://microsoft.github.io/language-server-protocol/ for doing the integration between IDE and your parser. Also, https://highlightjs.org/ https://highlightjs.org/ which lets you broadcast your code in a pretty way. I do however feel that your QoL will depend on whether or not you make a compiler or a translator. The key benefit of a translator is that you can co-opt the host language's standard runtime and debugger.

brundolf 5y ago
Agree with all of this. Don't do any yak-shaving you don't want to do just for the fun of it; build your language and its tooling on existing systems so you can spend your time on what makes it special, instead of what doesn't.

ModernMech 5y ago

> It's for this reason that nowadays I rarely write my own "new" programming languages from scratch, but rather embed them as DSLs inside existing programming languages using macros. This way you get the full ecosystem of the host language for free I don't understand what you mean by this, so I'm hoping you can clarify. What's the advantage of doing this over the alternative way: writing a parser which would compile to code in the host language, or some primitives that you've arranged in the host language that would represent the bytecode of your language interpreter. From there you can leverage whatever in the host language you want, including its type system, packages, language semantics, FFI access to other languages. Really whatever you like. What you're suggesting is leveraging a macro system to help you write what is essentially a parser/compiler, which is a good win. But I don't see what that buys you past that point. If I do things the other way, by writing a parser that consumes a *.my_lang file and compiles it to primitives or actual source code in the host language, I can still leverage all the benefits of the host language just as you describe doing with a macro system. The advantage of doing things this way is you're not limited to languages with tolerable metaprogramming capabilities.