6 ms·

Using C++ as a scripting language, part 8

134 points by fwsgonzo 3y ago

mike_hock 3y ago
On the performance side, pretty much the entire game logic would have to be implemented on the "script" side for this to be worth it compared to just embedding Lua and calling it a day, right? I can see the benefit of typechecking the "script" code, though. The vulnerable parts are the "syscall" wrappers but they don't change all the time.
- jamesu 3y ago
  Is performance really going to be a deciding factor for using this? I'd guess it depends what you really want in your "scripting" language. For me I'd want some sort of runtime with an easily readable language that can tolerate bad code, so in that case I would prefer something like lua. Must admit though, things like this really make one think... what do people even want in a "scripting" language these days?
  - masklinn 3y ago
    Probably depends how much of the game is implemented via "scripting", some games just put scripting at the edges (UI, data definition) while others do everything up from the game engine via "scripting" e.g. https://forums.unrealengine.com/t/ive-created-an-infinite-voxel-world-similar-to-minecraft/57218 https://forums.unrealengine.com/t/ive-created-an-infinite-vo... > The whole project s 100% blueprint with no C++ being used. I think this project is a great example for the power of blueprint. And that's a post from 2016. These days you can probably create entire games via visual scripting (https://unity.com/features/unity-visual-scripting https://unity.com/features/unity-visual-scripting) as long as they're not too bespoke or demanding.
    - deaddodo 3y ago
      The example you use seems to be specifically a "because I can" project. We can't be sure (because the code isn't available, obviously), but I would highly doubt there are any AA or AAA games built entirely in the scripting engine. However, many 4X games get pretty close (Paradox series, Civilization, the Endless games, HoM&M, AoW, etc).
    - intelVISA 3y ago
      The limiting factor is not truly perf for visual scripting but everything around its ecosystem (diffing, architecture, spaghetti)
- fwsgonzo 3y ago
  I don't know about the entire script. The game and game engine has tens of thousands of lines of code that is not script. But, I feel like it opens creative doors to be able to make (lets just say) a magnitude more calls into the virtual machine compared to other options. That's not to say that I am done with this research I am doing. The idea is to be able enter and leave the script as fast as possible, while also call back into the game engine as fast as possible. Once this is achieved, game logic that once had to rely on finite options can have script callbacks. Also the game will be naturally moddable from the start. Here is an outtake from one of my game scripts: https://gist.github.com/fwsGonzo/d9116c46e5b7f8ed4743ab15f89c68c2 https://gist.github.com/fwsGonzo/d9116c46e5b7f8ed4743ab15f89... While it may not look like it, flow.cpp calls back into the game engine in a surprising amount of places. It's also something that regularly gets called 10k times per simulation step if there is a cave under the water, for example. For me right now, I feel like I have reached a sweet spot where I would no longer consider going back. Also, it's a fun project! Sometimes I am wondering if I am enjoying making the game engine more than the game. Oh well. Might as well tack on to this that you can also choose your language. I have emulated Go, Rust and Nim pretty successfully. Enough to know that they would be a possible choice for emulation. Go is perhaps weird in that you want to avoid the C API (cgo) and just figure out the assembly instead, but that could also lock you to a Go version.
  - fwsgonzo 3y ago
    Let me just add that perhaps we can sandbox python too? LPython is on the frontpage now and if I read it right, it will compile to C and C++ when targeting AOT. See: https://lpython.org/blog/2023/07/lpython-novel-fast-retargetable-python-compiler/ https://lpython.org/blog/2023/07/lpython-novel-fast-retarget...
  - bsenftner 3y ago
    You ought to meet and discuss your project with Arnaud Hervas, the original author of the Shake compositing software. Shake was a VFX industry compositor that used GCC C++ as the "scripting language" that non-developers wrote. It was not exposed as C, but macro modified to look simpler. Arnaud did several industry firsts, including using the GPU to render compositing layers back when all that type of work was in software. His company before Apple bought them was called Nothing Real, https://en.wikipedia.org/wiki/Nothing_Real https://en.wikipedia.org/wiki/Nothing_Real He's a nice guy, so don't be intimidated.
dgan 3y ago
part _8_ .. oh that doesn't start well
kwijibo123456 3y ago
If you want to see real C++ scripting: the (slightly insane) people of CERN use a 99% ANSI compliant, self-written C++ interpreter (!) for online data analytics. https://root.cern/manual/first_steps_with_root/ https://root.cern/manual/first_steps_with_root/
- andromeduck 3y ago
  That is so incredibly cool!!!
- noobermin 3y ago
  Unless things have changed, root is quite horrible actually.
  - moelf 3y ago
    yeah... basically... in fact they're leaning more into JIT + string programming now and seeing big JIT latency etc. (imo is kinda ~approaching Julia from C++ side)
  - amadsen 3y ago
    Root is absolutely, mind-blowingly, amazing. It gets a bad rap because it forces you to use primitives that were designed back in the early nineties. If you're "just" trying to analyze some data, your experience will indeed be "horrible" compared to what's offered by Python, R, Matlab, or Julia. But beyond that... Root adds fully working reflection to C++. Root gives you dynamic library loading and reloading - you can fix a bug or add a new feature, recompile parts of your program and keep working without restarting it. Root has a feature complete C++ interpreter, with scripting and a REPL loop. You can work with it completely interactively. After prototyping you can save your code as a script. After identifying performance critical parts of your code, you can compile them and get the full power of bare metal C++, without changing anything about the code. Yes this is technically possible with e.g. python + numba as well, but not as straight-forward. Root is fully interoperable with Python and R - you can mix scripts and REPLs between the languages and pass objects between them. Root can serialize any object, without requiring any custom code whatsoever (some serious dark magic needed for this). In fact you can pause your entire program and save it to disk or send it over the network to keep running somewhere else. Root has its own file format for efficiently storing massive amounts of data in arbitrarily complex structures. It can stream it over the network too, with probabilistic read ahead and caching for maximum efficiency. Root comes with libraries for physics/math/stats that rival those of the largest commercial and open source offerings. Each one of these is a massive technical achievement and Root has had most of them for decades now. Oh, and it has largely maintained backwards compatibility through all this time as well. Of course, very few people outside of CERN need all of this. Even within CERN, many projects don't. But for those who do, there are very few - if any - alternatives.
pyuser583 3y ago
The most eye-catching title I’ve seen.
Jeaye 3y ago
On the topic of using C++ for scripting, and related to the discussion of CERN's ROOT/Cling, I am developing a Clojure dialect on C++/LLVM called jank: https://jank-lang.org/ https://jank-lang.org/ jank is a true Clojure, meaning you get interactive, REPL-based development (thanks to Cling) and a whole stdlib of persistent, immutable data structures (thanks to immer) and functions to transform them. But it's also C++, so you can write inline C++ within your jank source, and interpolate jank expressions within that. You can link with existing native code using LLVM and you can embed jank into your existing native projects to use for scripting. jank is pre-alpha, right now, and I've only been showing it to Clojure devs so far, but there's a huge audience of C++ devs which may be interested in introducing Clojure to their native code.
- widdershins 3y ago
  Count me among them. I write C++ all day and I'd love to introduce Lisp/Clojure to orchestrate my more performant code. This is a very interesting project!
- naruhodo 3y ago
  I don't know too much about either Jank or Ferret, but would you care to compare the two?
  - Jeaye 3y ago
    Happy to. Ferret and jank have similar goals, but some key differences: 1. Ferret targets embedded platforms primarily; jank is more general and less constrained 2. Ferret doesn't have a JIT-based runtime, meaning no nREPL while you build your programs; jank provides the whole Clojure interactive programming experience, meaning you can start with a REPL and an empty file and end up with a whole server/GUI application/game without stopping to recompile/run again 3. Ferret is written in Clojure and requires the JVM to run, whereas jank's host is native, meaning its runtime and compiler and both native (C++) My overall sentiment for Ferret is that it's great and I'm excited that it exists. I have my eye on it so that we might be able to learn from some of the decisions made over there. In particular, its interop and memory management systems are of interest. Worth noting, when you look at all of the existing native Clojures (Ferret, Carp, jank, etc), only one of them aims to be Clojure source compatible while providing a true REPL-driven experience: jank.
corysama 3y ago
Visual Studio upgraded Edit And Continue to support live code updates. https://learn.microsoft.com/en-us/visualstudio/debugger/hot-reload?view=vs-2022 https://learn.microsoft.com/en-us/visualstudio/debugger/hot-... If that’s not enough, there’s https://liveplusplus.tech/index.html https://liveplusplus.tech/index.html
hgs3 3y ago
Worth mentioning that Quake 3 used C as a scripting language back in 1999. The engine used a modified LCC compiler [1] to generate byte code for execution in its own virtual machine. The same could be achieved today by embedding a WASM VM and compiling your C/Zig/Rust whatever to web assembly. As an aside, I find the idea of writing game code in unmanaged C++ perplexing. Even the original Quake, which targeted DOS-era hardware, relied on a scripting language running in a plain vanilla interpreter (no JIT here). In the [current year] I would think advances in computing power, parallelism, and JIT technology would be more than adequate to accommodate an interpreted scripting language. [1] https://en.wikipedia.org/wiki/LCC_(compiler)#Projects_incorporating_LCC https://en.wikipedia.org/wiki/LCC_(compiler)#Projects_incorp...
- __s 3y ago
  Using an interpreter on old hardware was often to save memory https://lwn.net/Articles/412024 https://lwn.net/Articles/412024
- ramesh31 3y ago
  >As an aside, I find the idea of writing game code in unmanaged C++ perplexing. https://github.com/id-Software/DOOM-3 https://github.com/id-Software/DOOM-3
- kevingadd 3y ago
  "advances in computing power, parallelism, and JIT technology would be more than adequate to accommodate an interpreted scripting language." While this is true, game consoles + iOS/iPadOS prohibit JIT and in some cases prohibit script interpreters, so people resort to stuff like 'we wrote a compiler for C# that turns it all into C++ source files' in order to pass review and collect Fortnite Money*. * I'm using Fortnite Money here to represent 'the kind of money you can make if you're willing to sell lootboxes to teens on the App Store'
- Tozen 3y ago
  Add Vlang to that list, as another that can compile to web assembly. In addition to having a C2V translator (and some other languages have similar). A number of safer and easier/convenient options are out there for those that care to look.
bullen 3y ago
I'm making a Action MMO engine where the game will be coded in C and hot-reloaded with this API: http://edit.rupy.se?host=move.rupy.se&path=/file/game.cpp&space http://edit.rupy.se?host=move.rupy.se&path=/file/game.cpp&sp... So far the API is really small and the turnaroud is 100 millisec.
_448 3y ago
ChaiScript is another option: https://github.com/ChaiScript/ChaiScript https://github.com/ChaiScript/ChaiScript
nathants 3y ago
this is interesting. i’m working on a high performance game in c++17 with clion and wickedengine on linux. compiles are 20 seconds, which is fine. 1 second would be better. blender exports are about 20 seconds too. 1 second would be better.
LiamCooper234 3y ago
[dead]

134 points by fwsgonzo 3y ago

On the performance side, pretty much the entire game logic would have to be implemented on the "script" side for this to be worth it compared to just embedding Lua and calling it a day, right? I can see the benefit of typechecking the "script" code, though. The vulnerable parts are the "syscall" wrappers but they don't change all the time.

jamesu 3y ago
Is performance really going to be a deciding factor for using this? I'd guess it depends what you really want in your "scripting" language. For me I'd want some sort of runtime with an easily readable language that can tolerate bad code, so in that case I would prefer something like lua. Must admit though, things like this really make one think... what do people even want in a "scripting" language these days?
- masklinn 3y ago
  Probably depends how much of the game is implemented via "scripting", some games just put scripting at the edges (UI, data definition) while others do everything up from the game engine via "scripting" e.g. https://forums.unrealengine.com/t/ive-created-an-infinite-voxel-world-similar-to-minecraft/57218 https://forums.unrealengine.com/t/ive-created-an-infinite-vo... > The whole project s 100% blueprint with no C++ being used. I think this project is a great example for the power of blueprint. And that's a post from 2016. These days you can probably create entire games via visual scripting (https://unity.com/features/unity-visual-scripting https://unity.com/features/unity-visual-scripting) as long as they're not too bespoke or demanding.
  - deaddodo 3y ago
    The example you use seems to be specifically a "because I can" project. We can't be sure (because the code isn't available, obviously), but I would highly doubt there are any AA or AAA games built entirely in the scripting engine. However, many 4X games get pretty close (Paradox series, Civilization, the Endless games, HoM&M, AoW, etc).
  - intelVISA 3y ago
    The limiting factor is not truly perf for visual scripting but everything around its ecosystem (diffing, architecture, spaghetti)
fwsgonzo 3y ago
I don't know about the entire script. The game and game engine has tens of thousands of lines of code that is not script. But, I feel like it opens creative doors to be able to make (lets just say) a magnitude more calls into the virtual machine compared to other options. That's not to say that I am done with this research I am doing. The idea is to be able enter and leave the script as fast as possible, while also call back into the game engine as fast as possible. Once this is achieved, game logic that once had to rely on finite options can have script callbacks. Also the game will be naturally moddable from the start. Here is an outtake from one of my game scripts: https://gist.github.com/fwsGonzo/d9116c46e5b7f8ed4743ab15f89c68c2 https://gist.github.com/fwsGonzo/d9116c46e5b7f8ed4743ab15f89... While it may not look like it, flow.cpp calls back into the game engine in a surprising amount of places. It's also something that regularly gets called 10k times per simulation step if there is a cave under the water, for example. For me right now, I feel like I have reached a sweet spot where I would no longer consider going back. Also, it's a fun project! Sometimes I am wondering if I am enjoying making the game engine more than the game. Oh well. Might as well tack on to this that you can also choose your language. I have emulated Go, Rust and Nim pretty successfully. Enough to know that they would be a possible choice for emulation. Go is perhaps weird in that you want to avoid the C API (cgo) and just figure out the assembly instead, but that could also lock you to a Go version.
- fwsgonzo 3y ago
  Let me just add that perhaps we can sandbox python too? LPython is on the frontpage now and if I read it right, it will compile to C and C++ when targeting AOT. See: https://lpython.org/blog/2023/07/lpython-novel-fast-retargetable-python-compiler/ https://lpython.org/blog/2023/07/lpython-novel-fast-retarget...
- bsenftner 3y ago
  You ought to meet and discuss your project with Arnaud Hervas, the original author of the Shake compositing software. Shake was a VFX industry compositor that used GCC C++ as the "scripting language" that non-developers wrote. It was not exposed as C, but macro modified to look simpler. Arnaud did several industry firsts, including using the GPU to render compositing layers back when all that type of work was in software. His company before Apple bought them was called Nothing Real, https://en.wikipedia.org/wiki/Nothing_Real https://en.wikipedia.org/wiki/Nothing_Real He's a nice guy, so don't be intimidated.

dgan 3y ago

part _8_ .. oh that doesn't start well

kwijibo123456 3y ago

If you want to see real C++ scripting: the (slightly insane) people of CERN use a 99% ANSI compliant, self-written C++ interpreter (!) for online data analytics. https://root.cern/manual/first_steps_with_root/ https://root.cern/manual/first_steps_with_root/

andromeduck 3y ago
That is so incredibly cool!!!
noobermin 3y ago
Unless things have changed, root is quite horrible actually.
- moelf 3y ago
  yeah... basically... in fact they're leaning more into JIT + string programming now and seeing big JIT latency etc. (imo is kinda ~approaching Julia from C++ side)
- amadsen 3y ago
  Root is absolutely, mind-blowingly, amazing. It gets a bad rap because it forces you to use primitives that were designed back in the early nineties. If you're "just" trying to analyze some data, your experience will indeed be "horrible" compared to what's offered by Python, R, Matlab, or Julia. But beyond that... Root adds fully working reflection to C++. Root gives you dynamic library loading and reloading - you can fix a bug or add a new feature, recompile parts of your program and keep working without restarting it. Root has a feature complete C++ interpreter, with scripting and a REPL loop. You can work with it completely interactively. After prototyping you can save your code as a script. After identifying performance critical parts of your code, you can compile them and get the full power of bare metal C++, without changing anything about the code. Yes this is technically possible with e.g. python + numba as well, but not as straight-forward. Root is fully interoperable with Python and R - you can mix scripts and REPLs between the languages and pass objects between them. Root can serialize any object, without requiring any custom code whatsoever (some serious dark magic needed for this). In fact you can pause your entire program and save it to disk or send it over the network to keep running somewhere else. Root has its own file format for efficiently storing massive amounts of data in arbitrarily complex structures. It can stream it over the network too, with probabilistic read ahead and caching for maximum efficiency. Root comes with libraries for physics/math/stats that rival those of the largest commercial and open source offerings. Each one of these is a massive technical achievement and Root has had most of them for decades now. Oh, and it has largely maintained backwards compatibility through all this time as well. Of course, very few people outside of CERN need all of this. Even within CERN, many projects don't. But for those who do, there are very few - if any - alternatives.

pyuser583 3y ago

The most eye-catching title I’ve seen.

Jeaye 3y ago

On the topic of using C++ for scripting, and related to the discussion of CERN's ROOT/Cling, I am developing a Clojure dialect on C++/LLVM called jank: https://jank-lang.org/ https://jank-lang.org/ jank is a true Clojure, meaning you get interactive, REPL-based development (thanks to Cling) and a whole stdlib of persistent, immutable data structures (thanks to immer) and functions to transform them. But it's also C++, so you can write inline C++ within your jank source, and interpolate jank expressions within that. You can link with existing native code using LLVM and you can embed jank into your existing native projects to use for scripting. jank is pre-alpha, right now, and I've only been showing it to Clojure devs so far, but there's a huge audience of C++ devs which may be interested in introducing Clojure to their native code.

widdershins 3y ago
Count me among them. I write C++ all day and I'd love to introduce Lisp/Clojure to orchestrate my more performant code. This is a very interesting project!
naruhodo 3y ago
I don't know too much about either Jank or Ferret, but would you care to compare the two?
- Jeaye 3y ago
  Happy to. Ferret and jank have similar goals, but some key differences: 1. Ferret targets embedded platforms primarily; jank is more general and less constrained 2. Ferret doesn't have a JIT-based runtime, meaning no nREPL while you build your programs; jank provides the whole Clojure interactive programming experience, meaning you can start with a REPL and an empty file and end up with a whole server/GUI application/game without stopping to recompile/run again 3. Ferret is written in Clojure and requires the JVM to run, whereas jank's host is native, meaning its runtime and compiler and both native (C++) My overall sentiment for Ferret is that it's great and I'm excited that it exists. I have my eye on it so that we might be able to learn from some of the decisions made over there. In particular, its interop and memory management systems are of interest. Worth noting, when you look at all of the existing native Clojures (Ferret, Carp, jank, etc), only one of them aims to be Clojure source compatible while providing a true REPL-driven experience: jank.

corysama 3y ago

Visual Studio upgraded Edit And Continue to support live code updates. https://learn.microsoft.com/en-us/visualstudio/debugger/hot-reload?view=vs-2022 https://learn.microsoft.com/en-us/visualstudio/debugger/hot-... If that’s not enough, there’s https://liveplusplus.tech/index.html https://liveplusplus.tech/index.html

hgs3 3y ago

Worth mentioning that Quake 3 used C as a scripting language back in 1999. The engine used a modified LCC compiler [1] to generate byte code for execution in its own virtual machine. The same could be achieved today by embedding a WASM VM and compiling your C/Zig/Rust whatever to web assembly. As an aside, I find the idea of writing game code in unmanaged C++ perplexing. Even the original Quake, which targeted DOS-era hardware, relied on a scripting language running in a plain vanilla interpreter (no JIT here). In the [current year] I would think advances in computing power, parallelism, and JIT technology would be more than adequate to accommodate an interpreted scripting language. [1] https://en.wikipedia.org/wiki/LCC_(compiler)#Projects_incorporating_LCC https://en.wikipedia.org/wiki/LCC_(compiler)#Projects_incorp...

__s 3y ago
Using an interpreter on old hardware was often to save memory https://lwn.net/Articles/412024 https://lwn.net/Articles/412024
ramesh31 3y ago
>As an aside, I find the idea of writing game code in unmanaged C++ perplexing. https://github.com/id-Software/DOOM-3 https://github.com/id-Software/DOOM-3
kevingadd 3y ago
"advances in computing power, parallelism, and JIT technology would be more than adequate to accommodate an interpreted scripting language." While this is true, game consoles + iOS/iPadOS prohibit JIT and in some cases prohibit script interpreters, so people resort to stuff like 'we wrote a compiler for C# that turns it all into C++ source files' in order to pass review and collect Fortnite Money*. * I'm using Fortnite Money here to represent 'the kind of money you can make if you're willing to sell lootboxes to teens on the App Store'
Tozen 3y ago
Add Vlang to that list, as another that can compile to web assembly. In addition to having a C2V translator (and some other languages have similar). A number of safer and easier/convenient options are out there for those that care to look.

bullen 3y ago

I'm making a Action MMO engine where the game will be coded in C and hot-reloaded with this API: http://edit.rupy.se?host=move.rupy.se&path=/file/game.cpp&space http://edit.rupy.se?host=move.rupy.se&path=/file/game.cpp&sp... So far the API is really small and the turnaroud is 100 millisec.

_448 3y ago

ChaiScript is another option: https://github.com/ChaiScript/ChaiScript https://github.com/ChaiScript/ChaiScript

nathants 3y ago

this is interesting. i’m working on a high performance game in c++17 with clion and wickedengine on linux. compiles are 20 seconds, which is fine. 1 second would be better. blender exports are about 20 seconds too. 1 second would be better.

LiamCooper234 3y ago

[dead]