8 ms·

Cakelisp: A Programming Language for Games

178 points by makuto 6y ago

akhilcacharya 6y ago
I’m interested in this “comptime” aspect. Zig has it too, but are there any other languages that use this? I’m under the impression that there’s a distinction between comptime evaluation and typical lisp macros.
- makuto 6y ago
  The big distinction to me is that there is a final executable that eventually gets created, whereas Lisp programs can continue to generate code and self-modify at runtime. Cakelisp differs from Zig in that arbitrary code modification is supported, which is a step closer to the modifiable environment of Lisps. Very few languages besides lisp allow you to do things like "iterate over every function defined and change their bodies, and create a new function which calls them". Zig does not support that. It's extremely useful for some tasks (e.g. hot-reloading code modification, mentioned in the article) Jai isn't out yet, but I was inspired by many of the comptime ideas.
- david2ndaccount 6y ago
  I don’t know how comptime works in Zig, but D has nearly unrestricted compile time code execution.
  - moonchild 6y ago
    ? Lots of things don't work in d's CTFE. It's certainly very useful, and much better than e.g. c++ constexpr. But it's still worlds away from proper lisp metaprogramming.
makuto 6y ago
See also on /r/gamedev [0]. After years of dealing with points of frustration in C++ land, I've created my own programming language. It emphasizes compile-time code generation, seamless C/C++ interoperability, and easier 3rd-party dependency integration. It's like "C in S-expressions", but offers much more than just that. I had a hard time trimming this article down because of how excited I am about the language. Feel free to skim and read more in the sections that pique your interest. I don't expect everyone to love it and adopt it. I do hope that some of the ideas are interesting to fellow programmers. I found it eye-opening to realize how much better my development environment could become once I opened this door. [0] https://www.reddit.com/r/gamedev/comments/kh1p0a/cakelisp_a_programming_language_for_games/ https://www.reddit.com/r/gamedev/comments/kh1p0a/cakelisp_a_...
azhenley 6y ago
It looks like an interesting project. The syntax for pointers is what grabbed my attention at first. Would really need to spend some time with this to know how it would work in practice. const char* myString = "Blah"; (var my-string (* (const char)) "Blah")
- makuto 6y ago
  Thanks! It is a bit different, and does require more typing, but I like how unambiguous it is. It also simplifies code modification by making type parsing and changing easier, because you can recursively unwrap multiple pointers, or pointer-ify things easily.
  - MycroftJones 6y ago
    Have you looked at newlisp? It has pretty seamless interop with C pointers.
  - emmelaich 6y ago
    Looks fantastic. Does it handle typedefs too? So one could write (var my-string conststring "Blah") with the appropriate typedef.
    - makuto 6y ago
      Yes! You can typedef types and function signatures.
erlend_sh 6y ago
Would love to see a comparison with https://gamelisp.rs/ https://gamelisp.rs/
- makuto 6y ago
  It looks like Cakelisp for Rust! I'll have to look at it in more depth. Thanks for the link!
  - anentropic 6y ago
    I have not used it but I think is not written for performance in the same way that Cakelisp appears to be: https://gamelisp.rs/reference/performance-figures.html#benchmarks https://gamelisp.rs/reference/performance-figures.html#bench... It seems more intended for scripting glue code in a Rust project
    - fleabitdev 6y ago
      Hello! I'm the developer of GameLisp :) You're correct that I've chosen safety and convenience over performance. I might eventually consider integrating the Cranelift code generator to try to achieve LuaJIT-like performance, but it's definitely not on the short-term roadmap. In practice, I find that GameLisp is more than fast enough to script a busy 2D platformer, as long as you're sensible about using Rust for the more CPU-intensive parts of your game engine. Incidentally, I'm planning to release GameLisp 0.2 within the next few days - happy to field any other questions while I'm here!
elgatonegro 6y ago
> Lisp has extremely powerful code generation, but makes serious performance compromises This is a serious exaggeration. Common Lisp has extremely good compilers that can meet C performance. There are plenty of Scheme implementations (I use Chez) with very good performance characteristics too.
- makuto 6y ago
  I do wish the C compiler was as fast as e.g. SBCL's compiler in terms of compile time, because multiple rounds of dependent macro compilation eats up a lot of time. I think Lisps tend to optimize for throughput, but games have very strict latency requirements. Garbage collection pauses could cause frame pacing issues (not that C solves that completely, but it is at least not a built in disadvantage of idiomatic use of the language)
  - moonchild 6y ago
    Newer concurrent java GCs (shenandoah, zgc) provide consistent ≤1ms pause times independent of heap size. That's a reasonable latency for most soft real-time tasks.
    - AlchemistCamp 6y ago
      Some older VMs like Erlang's also do very well in avoiding GC pauses. In Erlang's, the key is that garbage collection is per process. (Note that Erlang processes are analogous to Golang green threads, not OS prcessses.)
      - dugmartin 6y ago
        And in the BEAM the garbage collector is only invoked when the heap and stack meet which means for most short-lived processes it never runs: https://erlang.org/doc/apps/erts/GarbageCollection.html#overview https://erlang.org/doc/apps/erts/GarbageCollection.html#over... (btw, I'm enjoying the Reactor podcast, keep it up)
tines 6y ago
Nice! I've always been a proponent of thinking about C++ as "C with templates" rather than "C with classes," and once you realize that templates are just AST expanders, it becomes apparent that C++ is just a bad lisp :) I made the same thing a while back, and one of the neat simple things you can do is implement function-overloading a la C++. All you need is to define a way to serialize types to strings that are valid identifiers; then you (1) append the string-forms of the types of each function parameter to the name of the function at the definition site, along with a normal function that will do the dispatching in the second part, and (2) do the same thing for the type of the arguments at each call site. Et voila! Function overloading! Not quite as powerful as C++, which takes conversions and stuff into account, but it's an interesting experiment nonetheless. You can see how I did it here: https://github.com/zc1036/ivy/blob/master/src/lib/std/overload.icl https://github.com/zc1036/ivy/blob/master/src/lib/std/overlo... (DEFUN2 is the version of DEFUN in my language that supports overloading.)
- makuto 6y ago
  I agree. Templates make some tasks easy (type specialization and generic containers) but become tangled messes with other tasks (function bindings, serialization). I used something similar to your technique for compile-time variable destruction. The compiler doesn't know the type, so a macro generates a callback which deletes a casted version. These callbacks are named with the type so they can be lazily added and reused.
billfruit 6y ago
How does it compare with Carp, another lisp with C interop? Why did the creator create this, wont Carp have served the purpose.?
- makuto 6y ago
  Carp requires writing bindings to call C functions or use types. Additionally, code modification isn't possible in Carp. When I say seamless, I'm going for as close as possible, I.e. it should feel even easier to use C from Cakelisp than from C itself. The build system especially makes this possible.
- timdeve 6y ago
  As someone who works on Carp, it seems we're going for one level of abstraction higher than what this is aiming for. It's pretty easy to do C interop but the goal isn't to write "C in S-expressions". Doesn't mean it doesn't perform well, currently writing a GBA game: https://github.com/TimDeve/radicorn https://github.com/TimDeve/radicorn
fmakunbound 6y ago
It looks how I’d code Lisp if I was C programmer.
- 59nadir 6y ago
  That seems to be the explicit intent of the creator; a way to output C/C++ and hit the same use case targets but with much more powerful code generation facilities and a syntax that is much more amenable to that.
  - makuto 6y ago
    That's correct. My goal wasn't to make a CL-compatible Lisp. I imagine the code will start to look less like C as more macros/generators get added.
steinuil 6y ago
This reminds me a lot of c-amplify. I didn't see it in the author's list of languages they tried so I thought I'd mention it, it has a lot of neat ideas for building a lispy C. https://voodoo-slide.blogspot.com/2010/01/amplifying-c.html https://voodoo-slide.blogspot.com/2010/01/amplifying-c.html https://github.com/deplinenoise/c-amplify https://github.com/deplinenoise/c-amplify
kazinator 6y ago
Some well-known existing work in this exact area: https://github.com/kiselgra/c-mera https://github.com/kiselgra/c-mera https://github.com/eudoxia0/cmacro https://github.com/eudoxia0/cmacro (Written in Common Lisp, doesn't use S-exp syntax) https://github.com/tomhrr/dale https://github.com/tomhrr/dale (Prev disc: https://news.ycombinator.com/item?id=14079573 https://news.ycombinator.com/item?id=14079573) Newer stuff: https://github.com/saman-pasha/lcc https://github.com/saman-pasha/lcc (No mention of meta-programming) Lesser known: https://github.com/deplinenoise/c-amplify https://github.com/deplinenoise/c-amplify (No docs, no update since 2010)
- makuto 6y ago
  Thanks! I have a list of similar projects, I will add these! https://github.com/makuto/cakelisp/blob/master/doc/VsOtherLanguages.org https://github.com/makuto/cakelisp/blob/master/doc/VsOtherLa...
  - X4lldux 6y ago
    Great to see Ferret there! I've used it in some Arduino projects. I'd love to see a comparison. p.s. Carp is still actively maintained, should be on the other list.
deleted 6y ago
[deleted]
kloud 6y ago
Very cool project, great to see more s-expression based languages. I maintain a list of lisp-flavored languages [0], so I will add Cakelisp there. How did you implement the macro expansion? Are you translating the macros to C/C++, then compile it with C/C++ compiler and execute the temporary binary or do you have an interpreter for that? I work on a somewhat similar project called Liz, which is basically a lisp-flavored dialect of Zig [1]. I did not implement user-defined macros yet, planning to learn more about comptime and its limitations first. But the compiler itself uses macro-expansion to implement many features. [0] https://github.com/dundalek/awesome-lisp-languages https://github.com/dundalek/awesome-lisp-languages [1] https://github.com/dundalek/liz https://github.com/dundalek/liz
- pietrofmaggi 6y ago
  Thanks for maintaining this list, it looks super interesting and it was exactly what I was looking for!
- makuto 6y ago
  Yes, they are compiled just like the final exe, only they become dynamic libraries that I load with libdl. On subsequent builds, they are loaded again, unless the macro changed.
kodoque2 6y ago
Those projects really need more game examples. The best poster child for lisp based gamedev is crash bandicoot and it is almost 20 years old now.
macjohnmcc 6y ago
So many articles on Lisps on the front page. Did I miss something? It is Lispmas?
qw3rty01 6y ago
> While languages like Rust offer benefits in terms of security and stability, they cost programmers in terms of productivity. It makes sense to value safety so highly if your code is safety-critical (operating systems, aerospace, automotive, etc.), but it's much less valuable when safety isn't as important (e.g. in games). Just want to note that there is a large benefit to this kind of safety even if you're not writing safety-critical code: lack of bugs! The biggest benefit I've seen from rust is that entire classes of bugs, some of which can be extremely difficult to root cause and fix, are removed by design. So you spend significantly less time on the later half of the project tracking down bugs, which is more than enough to offset the productivity loss at the beginning.
- cylon13 6y ago
  This has been my experience as well. I'm building an exploratory game project in Rust to test the waters for switching to it from C++ for a professional project. Initially I was hesitant because I didn't really care about memory safety, so I didn't want to pay some mental overhead for something I didn't care about. But after using Rust for a while, I'd hands-down choose it over C++, even for projects where I feel like I don't care about either memory safety or even maintainability like single player games. The ergonomics are just so much better, the package manager exists (and is good), and the constraints placed on the architecture really just result in the program being easier to grow. So while safety and maintainability are what Rust gets marketed for, the ergonimics and just overall productivity of the language is enough to sell me on it for game dev. Languages like Zig and Jai also seem interesting in this space, but they're far from being ready to do anything in production with. The Rust ecosystem is actually ready for production now, and the language is a pleasure to work with.
schemescape 6y ago
Is there any reason this couldn't/shouldn't be used for applications besides than games? Other than your motivation/goals coming from solving game development problems, I didn't see anything that makes this useful for _only_ games. Edit: The GitHub repository implies that this could be used for purposes other than games: > The goal is a metaprogrammable, hot-reloadable, non-garbage-collected language ideal for high performance, iteratively-developed programs (especially games).
- makuto 6y ago
  Yes, there's nothing in the language which is specific to games. Once I get pure C output supported, it should be suitable even for embedded C environments.
didibus 6y ago
This looks great. The S-expr syntax is my favourite, though I'd have preferred a Clojure-like one personally, a CL inspired one is still fine. Some of the downsides mentioned can easily be taken care of by a macro I believe. Like the path traversal can be flattened with a macro most likely. Same thing for the type definitions.

178 points by makuto 6y ago

I’m interested in this “comptime” aspect. Zig has it too, but are there any other languages that use this? I’m under the impression that there’s a distinction between comptime evaluation and typical lisp macros.

makuto 6y ago
The big distinction to me is that there is a final executable that eventually gets created, whereas Lisp programs can continue to generate code and self-modify at runtime. Cakelisp differs from Zig in that arbitrary code modification is supported, which is a step closer to the modifiable environment of Lisps. Very few languages besides lisp allow you to do things like "iterate over every function defined and change their bodies, and create a new function which calls them". Zig does not support that. It's extremely useful for some tasks (e.g. hot-reloading code modification, mentioned in the article) Jai isn't out yet, but I was inspired by many of the comptime ideas.
david2ndaccount 6y ago
I don’t know how comptime works in Zig, but D has nearly unrestricted compile time code execution.
- moonchild 6y ago
  ? Lots of things don't work in d's CTFE. It's certainly very useful, and much better than e.g. c++ constexpr. But it's still worlds away from proper lisp metaprogramming.

makuto 6y ago

See also on /r/gamedev [0]. After years of dealing with points of frustration in C++ land, I've created my own programming language. It emphasizes compile-time code generation, seamless C/C++ interoperability, and easier 3rd-party dependency integration. It's like "C in S-expressions", but offers much more than just that. I had a hard time trimming this article down because of how excited I am about the language. Feel free to skim and read more in the sections that pique your interest. I don't expect everyone to love it and adopt it. I do hope that some of the ideas are interesting to fellow programmers. I found it eye-opening to realize how much better my development environment could become once I opened this door. [0] https://www.reddit.com/r/gamedev/comments/kh1p0a/cakelisp_a_programming_language_for_games/ https://www.reddit.com/r/gamedev/comments/kh1p0a/cakelisp_a_...

azhenley 6y ago

It looks like an interesting project. The syntax for pointers is what grabbed my attention at first. Would really need to spend some time with this to know how it would work in practice. const char* myString = "Blah"; (var my-string (* (const char)) "Blah")

makuto 6y ago
Thanks! It is a bit different, and does require more typing, but I like how unambiguous it is. It also simplifies code modification by making type parsing and changing easier, because you can recursively unwrap multiple pointers, or pointer-ify things easily.
- MycroftJones 6y ago
  Have you looked at newlisp? It has pretty seamless interop with C pointers.
- emmelaich 6y ago
  Looks fantastic. Does it handle typedefs too? So one could write (var my-string conststring "Blah") with the appropriate typedef.
  - makuto 6y ago
    Yes! You can typedef types and function signatures.

erlend_sh 6y ago

Would love to see a comparison with https://gamelisp.rs/ https://gamelisp.rs/

makuto 6y ago
It looks like Cakelisp for Rust! I'll have to look at it in more depth. Thanks for the link!
- anentropic 6y ago
  I have not used it but I think is not written for performance in the same way that Cakelisp appears to be: https://gamelisp.rs/reference/performance-figures.html#benchmarks https://gamelisp.rs/reference/performance-figures.html#bench... It seems more intended for scripting glue code in a Rust project
  - fleabitdev 6y ago
    Hello! I'm the developer of GameLisp :) You're correct that I've chosen safety and convenience over performance. I might eventually consider integrating the Cranelift code generator to try to achieve LuaJIT-like performance, but it's definitely not on the short-term roadmap. In practice, I find that GameLisp is more than fast enough to script a busy 2D platformer, as long as you're sensible about using Rust for the more CPU-intensive parts of your game engine. Incidentally, I'm planning to release GameLisp 0.2 within the next few days - happy to field any other questions while I'm here!

elgatonegro 6y ago

> Lisp has extremely powerful code generation, but makes serious performance compromises This is a serious exaggeration. Common Lisp has extremely good compilers that can meet C performance. There are plenty of Scheme implementations (I use Chez) with very good performance characteristics too.

makuto 6y ago
I do wish the C compiler was as fast as e.g. SBCL's compiler in terms of compile time, because multiple rounds of dependent macro compilation eats up a lot of time. I think Lisps tend to optimize for throughput, but games have very strict latency requirements. Garbage collection pauses could cause frame pacing issues (not that C solves that completely, but it is at least not a built in disadvantage of idiomatic use of the language)
- moonchild 6y ago
  Newer concurrent java GCs (shenandoah, zgc) provide consistent ≤1ms pause times independent of heap size. That's a reasonable latency for most soft real-time tasks.
  - AlchemistCamp 6y ago
    Some older VMs like Erlang's also do very well in avoiding GC pauses. In Erlang's, the key is that garbage collection is per process. (Note that Erlang processes are analogous to Golang green threads, not OS prcessses.)
    - dugmartin 6y ago
      And in the BEAM the garbage collector is only invoked when the heap and stack meet which means for most short-lived processes it never runs: https://erlang.org/doc/apps/erts/GarbageCollection.html#overview https://erlang.org/doc/apps/erts/GarbageCollection.html#over... (btw, I'm enjoying the Reactor podcast, keep it up)

tines 6y ago

Nice! I've always been a proponent of thinking about C++ as "C with templates" rather than "C with classes," and once you realize that templates are just AST expanders, it becomes apparent that C++ is just a bad lisp :) I made the same thing a while back, and one of the neat simple things you can do is implement function-overloading a la C++. All you need is to define a way to serialize types to strings that are valid identifiers; then you (1) append the string-forms of the types of each function parameter to the name of the function at the definition site, along with a normal function that will do the dispatching in the second part, and (2) do the same thing for the type of the arguments at each call site. Et voila! Function overloading! Not quite as powerful as C++, which takes conversions and stuff into account, but it's an interesting experiment nonetheless. You can see how I did it here: https://github.com/zc1036/ivy/blob/master/src/lib/std/overload.icl https://github.com/zc1036/ivy/blob/master/src/lib/std/overlo... (DEFUN2 is the version of DEFUN in my language that supports overloading.)

makuto 6y ago
I agree. Templates make some tasks easy (type specialization and generic containers) but become tangled messes with other tasks (function bindings, serialization). I used something similar to your technique for compile-time variable destruction. The compiler doesn't know the type, so a macro generates a callback which deletes a casted version. These callbacks are named with the type so they can be lazily added and reused.

billfruit 6y ago

How does it compare with Carp, another lisp with C interop? Why did the creator create this, wont Carp have served the purpose.?

makuto 6y ago
Carp requires writing bindings to call C functions or use types. Additionally, code modification isn't possible in Carp. When I say seamless, I'm going for as close as possible, I.e. it should feel even easier to use C from Cakelisp than from C itself. The build system especially makes this possible.
timdeve 6y ago
As someone who works on Carp, it seems we're going for one level of abstraction higher than what this is aiming for. It's pretty easy to do C interop but the goal isn't to write "C in S-expressions". Doesn't mean it doesn't perform well, currently writing a GBA game: https://github.com/TimDeve/radicorn https://github.com/TimDeve/radicorn

fmakunbound 6y ago

It looks how I’d code Lisp if I was C programmer.

59nadir 6y ago
That seems to be the explicit intent of the creator; a way to output C/C++ and hit the same use case targets but with much more powerful code generation facilities and a syntax that is much more amenable to that.
- makuto 6y ago
  That's correct. My goal wasn't to make a CL-compatible Lisp. I imagine the code will start to look less like C as more macros/generators get added.

steinuil 6y ago

This reminds me a lot of c-amplify. I didn't see it in the author's list of languages they tried so I thought I'd mention it, it has a lot of neat ideas for building a lispy C. https://voodoo-slide.blogspot.com/2010/01/amplifying-c.html https://voodoo-slide.blogspot.com/2010/01/amplifying-c.html https://github.com/deplinenoise/c-amplify https://github.com/deplinenoise/c-amplify

kazinator 6y ago

Some well-known existing work in this exact area: https://github.com/kiselgra/c-mera https://github.com/kiselgra/c-mera https://github.com/eudoxia0/cmacro https://github.com/eudoxia0/cmacro (Written in Common Lisp, doesn't use S-exp syntax) https://github.com/tomhrr/dale https://github.com/tomhrr/dale (Prev disc: https://news.ycombinator.com/item?id=14079573 https://news.ycombinator.com/item?id=14079573) Newer stuff: https://github.com/saman-pasha/lcc https://github.com/saman-pasha/lcc (No mention of meta-programming) Lesser known: https://github.com/deplinenoise/c-amplify https://github.com/deplinenoise/c-amplify (No docs, no update since 2010)

makuto 6y ago
Thanks! I have a list of similar projects, I will add these! https://github.com/makuto/cakelisp/blob/master/doc/VsOtherLanguages.org https://github.com/makuto/cakelisp/blob/master/doc/VsOtherLa...
- X4lldux 6y ago
  Great to see Ferret there! I've used it in some Arduino projects. I'd love to see a comparison. p.s. Carp is still actively maintained, should be on the other list.

deleted 6y ago

[deleted]

kloud 6y ago

Very cool project, great to see more s-expression based languages. I maintain a list of lisp-flavored languages [0], so I will add Cakelisp there. How did you implement the macro expansion? Are you translating the macros to C/C++, then compile it with C/C++ compiler and execute the temporary binary or do you have an interpreter for that? I work on a somewhat similar project called Liz, which is basically a lisp-flavored dialect of Zig [1]. I did not implement user-defined macros yet, planning to learn more about comptime and its limitations first. But the compiler itself uses macro-expansion to implement many features. [0] https://github.com/dundalek/awesome-lisp-languages https://github.com/dundalek/awesome-lisp-languages [1] https://github.com/dundalek/liz https://github.com/dundalek/liz

pietrofmaggi 6y ago
Thanks for maintaining this list, it looks super interesting and it was exactly what I was looking for!
makuto 6y ago
Yes, they are compiled just like the final exe, only they become dynamic libraries that I load with libdl. On subsequent builds, they are loaded again, unless the macro changed.

kodoque2 6y ago

Those projects really need more game examples. The best poster child for lisp based gamedev is crash bandicoot and it is almost 20 years old now.

macjohnmcc 6y ago

So many articles on Lisps on the front page. Did I miss something? It is Lispmas?

qw3rty01 6y ago

> While languages like Rust offer benefits in terms of security and stability, they cost programmers in terms of productivity. It makes sense to value safety so highly if your code is safety-critical (operating systems, aerospace, automotive, etc.), but it's much less valuable when safety isn't as important (e.g. in games). Just want to note that there is a large benefit to this kind of safety even if you're not writing safety-critical code: lack of bugs! The biggest benefit I've seen from rust is that entire classes of bugs, some of which can be extremely difficult to root cause and fix, are removed by design. So you spend significantly less time on the later half of the project tracking down bugs, which is more than enough to offset the productivity loss at the beginning.

cylon13 6y ago
This has been my experience as well. I'm building an exploratory game project in Rust to test the waters for switching to it from C++ for a professional project. Initially I was hesitant because I didn't really care about memory safety, so I didn't want to pay some mental overhead for something I didn't care about. But after using Rust for a while, I'd hands-down choose it over C++, even for projects where I feel like I don't care about either memory safety or even maintainability like single player games. The ergonomics are just so much better, the package manager exists (and is good), and the constraints placed on the architecture really just result in the program being easier to grow. So while safety and maintainability are what Rust gets marketed for, the ergonimics and just overall productivity of the language is enough to sell me on it for game dev. Languages like Zig and Jai also seem interesting in this space, but they're far from being ready to do anything in production with. The Rust ecosystem is actually ready for production now, and the language is a pleasure to work with.

schemescape 6y ago

Is there any reason this couldn't/shouldn't be used for applications besides than games? Other than your motivation/goals coming from solving game development problems, I didn't see anything that makes this useful for _only_ games. Edit: The GitHub repository implies that this could be used for purposes other than games: > The goal is a metaprogrammable, hot-reloadable, non-garbage-collected language ideal for high performance, iteratively-developed programs (especially games).

makuto 6y ago
Yes, there's nothing in the language which is specific to games. Once I get pure C output supported, it should be suitable even for embedded C environments.

didibus 6y ago

This looks great. The S-expr syntax is my favourite, though I'd have preferred a Clojure-like one personally, a CL inspired one is still fine. Some of the downsides mentioned can easily be taken care of by a macro I believe. Like the path traversal can be flattened with a macro most likely. Same thing for the type definitions.