4 ms·

I have been building a new not-exactly-a-programming-language using Racket and it has been an absolute pleasure, and the community has been extremely friendly a

by hyperion2010 9y ago

I have been building a new not-exactly-a-programming-language using Racket and it has been an absolute pleasure, and the community has been extremely friendly and helpful. One existing stumbling block that I encountered on the way is that I could not find documentation for the full process for the setup for creating a new packaged language (I have been known to fail hard at finding the right documents so this may just be my problem). https://beautifulracket.com/ https://beautifulracket.com/ has excellent coverage of the general process, but the differences between #lang br and #lang racket/base are just large enough that I couldn't fill in the gaps without going and looking through multiple repos to see how others had implemented their repo layout and then could intuit which setup commands I needed to run. If I find time I intend to write up my findings and submit them to the documentation, the short version is here in case someone finds it useful. repo-base ├── .git ├── my-lang ; run `raco pkg install` here │ └── info.rkt ; meta package depends on my-lang-lib └── my-lang-lib ; run `raco pkg install` here ├── info.rkt ; include actual dependencies └── my-lang ; implementation goes here └── main.rkt ; in theory you can put your whole implementation here Once that set up is complete you should be able to use `#lang my-lang`.

araes 9y ago
Thank you for the link. Reading the example on the creation of stacker.rkt (a toy reverse polish language) was dramatically more enlightening than the main article. As I understand, Racket = source-to-source interpreter with a lot of helpful shorthand for defining syntax of {insert language here} or arbitrary language you make on the spot. Do they already have libraries for major languages like c++, JS, python? Would be nice to write in a syntax I like (perl) and turn it into c++ that compiled.
- hyperion2010 9y ago
  Racket isn't really a source to source compiler/interpreter (you could try to write a compiler that converted racket into your language of choice, but that can often be quite difficult). Everything ultimately has to pass through Racket's language semantics, so matching implementation details requires a lot of work if the underlying language spec is large. The #lang C example down below sidesteps these issues because it acts as a pass through to the system c compiler and then handles returning those results into racket, essentially it delegates the language semantics to the c compiler and only deals with how to map the results of calling c code back onto racket objects. Similar issue with the FFI if you want to translate from one language to another, you still have to go through the process of mapping language semantics. Two examples where languages/parsers have been implemented in racket: Algol60 is the best and most complete example https://docs.racket-lang.org/algol60/index.html https://docs.racket-lang.org/algol60/index.html. A python parser has been implemented https://github.com/dyoo/ragg/blob/master/ragg/examples/python-grammar.rkt https://github.com/dyoo/ragg/blob/master/ragg/examples/pytho...
  - araes 9y ago
    Thank you for the clarification. Even after two articles I obviously still didn't fully get it. Appreciate the help, and the example links.
    - fiddlerwoaroof 9y ago
      Racket uses macros to transform your language’s syntax into valid racket code, which is then interpreted.
      - skrishnamurthi 9y ago
        No, this is far too glib, and the last part is wrong. See https://www.hashcollision.org/brainfudge/ https://www.hashcollision.org/brainfudge/ instead.
    - skrishnamurthi 9y ago
      Try reading https://www.hashcollision.org/brainfudge/ https://www.hashcollision.org/brainfudge/ and see if that helps.
- rscho 9y ago
  I also found the stacker example to be the clearest, because it really tackles the basics. I am however also confused as to whether Racket can transform any type of language. Indeed, how about interpreted-only languages such as for example J? Would the Racket JIT not be an obstacle in such a case? (Shameless plug for my askHN on this exact question I have been wondering about: https://news.ycombinator.com/item?id=16456562 https://news.ycombinator.com/item?id=16456562)
  - shakna 9y ago
    Racket includes a lazy language/library, and you can see some of the side-effects from the compiler there: #lang lazy is incredibly slow compared to #lang racket. Racket probably can transform any kind of language, but some seem to require extra work to get them to run smoothly, by which I mean you may have to write your own compiler as well as the rest, before you see real performance. However, #lang datalog is pretty much a pure interpreter, and it does have acceptable performance. So it should depend a bit on the language's fundamentals. Speaking of J, there is some work on getting it to play nice with Racket [0], though it is incomplete, and I have no idea if the work is ongoing. [0] https://docs.racket-lang.org/j/index.html https://docs.racket-lang.org/j/index.html
- sriku 9y ago
  More than just general purpose languages, the article highlights how creating small languages that integrate well into the host language is a powerful way to layer your system. I've experienced this first hand in making a DSL for sophisticated video editing and "style authoring" [1] in the context of muvee, and it was all much inspired by Racket (then known as MzScheme). You can tell based on the fact that the docs were written yet another embedded language "scribble" that's available with Racket/MzScheme. The speed of iteration and ease of discovery and abstraction of patterns in lisp/scheme is hard to do justice to in a descriptive article. The Racket team has taken that to a whole new level that you'll be hard pressed to find in any other "batteries included" system. [1] https://srikumarks.github.io/muvee-style-authoring/ https://srikumarks.github.io/muvee-style-authoring/ (disclaimer: I used to work for muvee)
sillysaurus3 9y ago
You're missing a reader, I think.
- hyperion2010 9y ago
  That was the 'in theory' bit. You can create a reader module inside main.rkt using (module+ reader (provide read-syntax)) where read-syntax can be defined in main.rkt or required from elsewhere. Good point though since I actually don't know exactly where to put a reader.rkt file to get the equivalent behavior of defining it as a submodule inside of main.rkt.
  - samth 9y ago
    You put in in a `lang` subdirectory (so `my-lang/lang/reader.rkt`).

araes 9y ago

Thank you for the link. Reading the example on the creation of stacker.rkt (a toy reverse polish language) was dramatically more enlightening than the main article. As I understand, Racket = source-to-source interpreter with a lot of helpful shorthand for defining syntax of {insert language here} or arbitrary language you make on the spot. Do they already have libraries for major languages like c++, JS, python? Would be nice to write in a syntax I like (perl) and turn it into c++ that compiled.

hyperion2010 9y ago
Racket isn't really a source to source compiler/interpreter (you could try to write a compiler that converted racket into your language of choice, but that can often be quite difficult). Everything ultimately has to pass through Racket's language semantics, so matching implementation details requires a lot of work if the underlying language spec is large. The #lang C example down below sidesteps these issues because it acts as a pass through to the system c compiler and then handles returning those results into racket, essentially it delegates the language semantics to the c compiler and only deals with how to map the results of calling c code back onto racket objects. Similar issue with the FFI if you want to translate from one language to another, you still have to go through the process of mapping language semantics. Two examples where languages/parsers have been implemented in racket: Algol60 is the best and most complete example https://docs.racket-lang.org/algol60/index.html https://docs.racket-lang.org/algol60/index.html. A python parser has been implemented https://github.com/dyoo/ragg/blob/master/ragg/examples/python-grammar.rkt https://github.com/dyoo/ragg/blob/master/ragg/examples/pytho...
- araes 9y ago
  Thank you for the clarification. Even after two articles I obviously still didn't fully get it. Appreciate the help, and the example links.
  - fiddlerwoaroof 9y ago
    Racket uses macros to transform your language’s syntax into valid racket code, which is then interpreted.
    - skrishnamurthi 9y ago
      No, this is far too glib, and the last part is wrong. See https://www.hashcollision.org/brainfudge/ https://www.hashcollision.org/brainfudge/ instead.
  - skrishnamurthi 9y ago
    Try reading https://www.hashcollision.org/brainfudge/ https://www.hashcollision.org/brainfudge/ and see if that helps.
rscho 9y ago
I also found the stacker example to be the clearest, because it really tackles the basics. I am however also confused as to whether Racket can transform any type of language. Indeed, how about interpreted-only languages such as for example J? Would the Racket JIT not be an obstacle in such a case? (Shameless plug for my askHN on this exact question I have been wondering about: https://news.ycombinator.com/item?id=16456562 https://news.ycombinator.com/item?id=16456562)
- shakna 9y ago
  Racket includes a lazy language/library, and you can see some of the side-effects from the compiler there: #lang lazy is incredibly slow compared to #lang racket. Racket probably can transform any kind of language, but some seem to require extra work to get them to run smoothly, by which I mean you may have to write your own compiler as well as the rest, before you see real performance. However, #lang datalog is pretty much a pure interpreter, and it does have acceptable performance. So it should depend a bit on the language's fundamentals. Speaking of J, there is some work on getting it to play nice with Racket [0], though it is incomplete, and I have no idea if the work is ongoing. [0] https://docs.racket-lang.org/j/index.html https://docs.racket-lang.org/j/index.html
sriku 9y ago
More than just general purpose languages, the article highlights how creating small languages that integrate well into the host language is a powerful way to layer your system. I've experienced this first hand in making a DSL for sophisticated video editing and "style authoring" [1] in the context of muvee, and it was all much inspired by Racket (then known as MzScheme). You can tell based on the fact that the docs were written yet another embedded language "scribble" that's available with Racket/MzScheme. The speed of iteration and ease of discovery and abstraction of patterns in lisp/scheme is hard to do justice to in a descriptive article. The Racket team has taken that to a whole new level that you'll be hard pressed to find in any other "batteries included" system. [1] https://srikumarks.github.io/muvee-style-authoring/ https://srikumarks.github.io/muvee-style-authoring/ (disclaimer: I used to work for muvee)

sillysaurus3 9y ago

You're missing a reader, I think.

hyperion2010 9y ago
That was the 'in theory' bit. You can create a reader module inside main.rkt using (module+ reader (provide read-syntax)) where read-syntax can be defined in main.rkt or required from elsewhere. Good point though since I actually don't know exactly where to put a reader.rkt file to get the equivalent behavior of defining it as a submodule inside of main.rkt.
- samth 9y ago
  You put in in a `lang` subdirectory (so `my-lang/lang/reader.rkt`).