6 ms·

Inventing your own programming language is something I can highly recommend. If you've never done it before it will dramatically broaden your programming horizo

by Udo 12y ago

Inventing your own programming language is something I can highly recommend. If you've never done it before it will dramatically broaden your programming horizon. It's a journey. About a year ago I decided to make a language mainly because I wanted to experiment with some programming paradigms. I had made little domain-specific "languages" many times before, but this was going to be different. I'd suggest that you try to figure out most things on your own, however. It's way more rewarding to think about the problems you're trying to solve, go ahead and implement them. That way you're also forced to have eye-opening meta thoughts about programming in general. Only after you've gained some appreciation for the issues involved, look them up and see how other people solved them. This worked very well for me. Lisps and Schemes are certainly easier to implement than most syntaxes, but consider doing something else so you can get a better understanding of the complexities of parsing a non-Lisp. In my case, I did my prototype in Java, that's the lexer, parser, runtime and standard library. While I wouldn't necessarily want to do it that way again, it was a great learning experience and I was pleased with the result. About two months ago I decided to give it another try, but this time in C, using the Lua codebase. So far this has been a very good choice for a second stab at the idea. And I was pleased that I could recognize a lot of the patterns in the Lua codebase where I had implemented equivalent parts in my earlier project. If I hadn't taken the hard route back then, I probably would have a harder time now.

kvalle 12y ago
It's definitely a journey! And I agree that trying to figure out how the concepts for yourself is by far the most rewarding. I hope this tutorial can help people realize that it's not difficult to get started, and inspire them to get started exploring on their own.
vidarh 12y ago
While I agree with you that it's worth learning to parse something "non-Lisp", I think that starting with an s-expression type syntax is worthwhile for a first-time even if you want to eventually implement a non-Lisp, for the simple reason that parsing is "easy" and very well documented compared to code-generation. Unless your language has a particularly hairy grammar (Ruby, I'm looking at you...), figuring out how to parse it into your desired structure afterwards is fairly straight-forward in comparison. That's why I took the approach of bypassing the parser entirely with my compiler series ( http://www.hokstad.com/compiler http://www.hokstad.com/compiler ), where the first several parts involved going straight to code-generation from code simply abusing the Ruby Array notation. Then later adding a simple s-expression parser. I didn't even decide to turn it into a Ruby compiler for quite some time (in retrospect, had I planned to do Ruby from the very beginning, there are a few things I'd likely do differently, but not that much; the biggest issues with writing the series have been learning a lot of new things about how writing about code influences the entire process)
- bunderbunder 12y ago
  The approach I'm taking for a current language project is to start with a minimalist syntax for the first iteration. This allows me to focus on the actual language implementation, which is the part I'm most interested in learning. It also allows me to rapidly iterate on language features without having to spend too much time fiddling with the parser. Once I've got a semantics I like, I'll probably do a second pass to come up with a new syntax if I haven't decided I'm done with the project. By then I'll have written plenty of test cases using the first syntax, so I should have a much better idea of what I'd like to get out of a more complex one.
tel 12y ago
I kind of favor the idea that you should use Lisp syntax for a while. Learning parsers is a nice technique, but it's really a small detail in language design. Eliding it for a while can help you focus on the more interesting issues of semantics.
the_watcher 12y ago
How far along as a programmer should one be before attempting building a language. I am a novice (very, very much so) Rubyist, and don't think I am anywhere near being able to attempt it, but I do know that, when I was still in school, the process most likely to make a complicated subject click with me was to understand it all the way to it's origin (the starkest example was actually the quadratic formula - I had tons of trouble understanding it and how to use it until I had a teacher who made us actually derive it, and all of a sudden I was one of the best in the class). Building a programming language seems like it might have the same kind of effect on me, and, being a firm believer of living outside your comfort zone, would like to know if it's possible for someone who is nowhere near being able to call himself a programmer to eventually build one.
- mcguire 12y ago
  This may not be a common opinion, but building a language is certainly within the reach of a novice programmer. The base ideas are very, very simple. You do have to be careful not to be overwhelmed by either the terminology ("recursive descent parsing" is a slightly more elaborate version of "reading a file") or some of the more elaborate things that you don't need for the basics but might want for a truly useful tool ("parsing" is a deep subject built mostly around making "reading a file" simpler, for arbitrarily complicated files). Building languages is also one of the most thoroughly studied topics, which is both a boon and a bane. On one hand, there are lots of good tutorials and excellent tools. On the other hand, there is lots of stuff elaborating on the basics and much of it seems almost deliberately complicated. Go for it! There are a lot of much worse ways of going into the rabbit hole.
  - the_watcher 12y ago
    Thanks! Any suggestions for resources to help me get started? I'm not afraid of complicated terminology as long as there is a way for me to easily figure out what it means (whether Wikipedia, a glossary, or simply a great example).
    - loup-vaillant 12y ago

by Udo 12y ago

kvalle 12y ago

It's definitely a journey! And I agree that trying to figure out how the concepts for yourself is by far the most rewarding. I hope this tutorial can help people realize that it's not difficult to get started, and inspire them to get started exploring on their own.

vidarh 12y ago

While I agree with you that it's worth learning to parse something "non-Lisp", I think that starting with an s-expression type syntax is worthwhile for a first-time even if you want to eventually implement a non-Lisp, for the simple reason that parsing is "easy" and very well documented compared to code-generation. Unless your language has a particularly hairy grammar (Ruby, I'm looking at you...), figuring out how to parse it into your desired structure afterwards is fairly straight-forward in comparison. That's why I took the approach of bypassing the parser entirely with my compiler series ( http://www.hokstad.com/compiler http://www.hokstad.com/compiler ), where the first several parts involved going straight to code-generation from code simply abusing the Ruby Array notation. Then later adding a simple s-expression parser. I didn't even decide to turn it into a Ruby compiler for quite some time (in retrospect, had I planned to do Ruby from the very beginning, there are a few things I'd likely do differently, but not that much; the biggest issues with writing the series have been learning a lot of new things about how writing about code influences the entire process)

bunderbunder 12y ago
The approach I'm taking for a current language project is to start with a minimalist syntax for the first iteration. This allows me to focus on the actual language implementation, which is the part I'm most interested in learning. It also allows me to rapidly iterate on language features without having to spend too much time fiddling with the parser. Once I've got a semantics I like, I'll probably do a second pass to come up with a new syntax if I haven't decided I'm done with the project. By then I'll have written plenty of test cases using the first syntax, so I should have a much better idea of what I'd like to get out of a more complex one.

tel 12y ago

I kind of favor the idea that you should use Lisp syntax for a while. Learning parsers is a nice technique, but it's really a small detail in language design. Eliding it for a while can help you focus on the more interesting issues of semantics.

the_watcher 12y ago

How far along as a programmer should one be before attempting building a language. I am a novice (very, very much so) Rubyist, and don't think I am anywhere near being able to attempt it, but I do know that, when I was still in school, the process most likely to make a complicated subject click with me was to understand it all the way to it's origin (the starkest example was actually the quadratic formula - I had tons of trouble understanding it and how to use it until I had a teacher who made us actually derive it, and all of a sudden I was one of the best in the class). Building a programming language seems like it might have the same kind of effect on me, and, being a firm believer of living outside your comfort zone, would like to know if it's possible for someone who is nowhere near being able to call himself a programmer to eventually build one.

mcguire 12y ago
This may not be a common opinion, but building a language is certainly within the reach of a novice programmer. The base ideas are very, very simple. You do have to be careful not to be overwhelmed by either the terminology ("recursive descent parsing" is a slightly more elaborate version of "reading a file") or some of the more elaborate things that you don't need for the basics but might want for a truly useful tool ("parsing" is a deep subject built mostly around making "reading a file" simpler, for arbitrarily complicated files). Building languages is also one of the most thoroughly studied topics, which is both a boon and a bane. On one hand, there are lots of good tutorials and excellent tools. On the other hand, there is lots of stuff elaborating on the basics and much of it seems almost deliberately complicated. Go for it! There are a lot of much worse ways of going into the rabbit hole.
- the_watcher 12y ago
  Thanks! Any suggestions for resources to help me get started? I'm not afraid of complicated terminology as long as there is a way for me to easily figure out what it means (whether Wikipedia, a glossary, or simply a great example).
  - loup-vaillant 12y ago