4 ms·

Lua is great to embed, but for Redis I picked it as a scripting language with many regrets because I don't like the language itself: it looks almost like design

Lua is great to embed, but for Redis I picked it as a scripting language with many regrets because I don't like the language itself: it looks almost like designed to have some kind of friction compared to what you expect to write in a language that is at the same level of abstractions. Little design decisions here and there that cumulate into a language that is somewhat hostile (for me, at least). Yet, it is so fast, easy to integrate, small footprint and reliable, that many times it is still the way to go. Btw, the hostility is present even at C-API level, with the stack approach: it's not rocket science but more cognitive load than needed. Even if I was exposed in the past to stack languages like FORTH, I still had to do some mental gym while writing the bindings. Why?

helpfulContrib 2y ago
[dead]
ck45 2y ago
Did you consider any other languages, like TCL? If yes, do you remember the pros and cons of that set of languages?
- progre 2y ago
  Was the first version of Redis not in fact written in TCL?
  - ck45 2y ago
    Oh, interesting, I didn’t know. ChatGPT “confirms” that a prototype was written in TCL
matheusmoreira 2y ago
> the hostility is present even at C-API level, with the stack approach > Why? I suspect it's because implementing control mechanisms is much easier when you reify the language's stack. Especially advanced ones such as generators and continuations which require copying the stack into an object and restoring it later. Making that work with the native stack is really hard and many languages don't even try. It also makes garbage collection precise. Lua values in C variables would be placed in registers or the native stack. In order to trace those values, Lua would require a conservative garbage collector that spills the registers and scans the entire native stack. By managing their own stack, they can avoid doing all that.
peutetre 2y ago
WebAssembly is a nice way to go for such things in future. You provide the interface, their wasm code runs in a sandbox, and your users get to write their plugins in whatever language they like.
- deleted 2y ago
  [deleted]
- unleaded 2y ago
  This gets mentioned a lot but has it been used in practice? How would you pass objects and methods etc to the scripting language? Autogenerated C headers?
  - brabel 2y ago
    To be able to use WASM-compiled languages for embedding you first need a small WASM runtime you can depend on (and things like ref types, GC etc which have been standardized only in the last couple of years). That didn't exist until recently, but now you can use https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/doc/embed_wamr.md https://github.com/bytecodealliance/wasm-micro-runtime/blob/... I am not aware of anyone using this yet, but I hope to see that become common in the next few years.
  - peutetre 2y ago
    The WebAssembly component model is aimed at having composable components that can call each other. The components can be written in any language, compiled to WebAssembly, and interoperate: https://github.com/WebAssembly/component-model/ https://github.com/WebAssembly/component-model/ Extism is a plugin framework for WebAssembly: https://extism.org/ https://extism.org/ https://github.com/extism/extism https://github.com/extism/extism Postgres and Extism: https://dylibso.com/blog/pg-extism/ https://dylibso.com/blog/pg-extism/ Wasm plugins for Zellij: https://zellij.dev/documentation/plugins https://zellij.dev/documentation/plugins Visual Studio Code can run WebAssembly extensions: https://code.visualstudio.com/blogs/2024/05/08/wasm https://code.visualstudio.com/blogs/2024/05/08/wasm A project to bring WebAssembly plugins to Godot: https://github.com/ashtonmeuser/godot-wasm https://github.com/ashtonmeuser/godot-wasm WasmEdge can be embedded in applications: https://wasmedge.org/docs/embed/overview https://wasmedge.org/docs/embed/overview Wasmer can be embedded in applications: https://blog.wasmer.io/executing-webassembly-in-your-rust-application-d5cd32e8ce46 https://blog.wasmer.io/executing-webassembly-in-your-rust-ap... Wasmtime can be embedded in applications: https://docs.wasmtime.dev/lang.html https://docs.wasmtime.dev/lang.html
spacechild1 2y ago
> Why? See https://news.ycombinator.com/item?id=42519033 https://news.ycombinator.com/item?id=42519033
astrobe_ 2y ago
A historical thing: > Traditionally, most virtual machines intended for actual execution are stack based, a trend that started with Pascal’s P-machine and continues today with Java’s JVM and Microsoft’s .Net environment. Currently, however, there has been a growing interest in register-based virtual machines (for instance, the planned new virtual machine for Perl 6 (Parrot) will be register based). As far as we know, the virtual machine of Lua 5.0 is the first register-based virtual machine to have a wide use. The API directly reflected the (previous) internals of the VM, I guess [1]. [1] (pdf) https://www.lua.org/doc/jucs05.pdf https://www.lua.org/doc/jucs05.pdf
- ufo 2y ago
  The new register-based virtual machine still uses the same stack under the hood. The only difference is how instructions are encoded in the bytecode. By default they can read and write to any position in the stack as opposed to pushing and popping from the top. The Lua stack API already does some of this, because several operations take stack indices as arguments.
ufo 2y ago
Have you seen the Passing a Language Through the Eye of a Needle article, from the Lua authors? It talks about how some of unusual the design decisions such as pcall error handling and the stack are tradeoffs in favor of "embeddability". https://queue.acm.org/detail.cfm?id=1983083 https://queue.acm.org/detail.cfm?id=1983083 The main purpose of the stack is to facilitate garbage collection. The GC can find out what Lua objects are currently being manipulated by the C client. The price is that the Lua API can never expose a "Lua object pointer" to C. Many other scripting languages expose such pointers, but then must also expose ways to manage their memory. For example, in Python the user of the API must explicitly increment and decrement reference counts.
- ranger_danger 2y ago
  The one time I tried to embed Lua in a C project (which included C-based functions I could call from Lua and setting up variables from C that were visible in Lua), I constantly struggled with trying to figure out when I needed to push/pop things from the stack and it just seemed very error-prone and easy to leak memory with. Different functions seem to have different requirements for when/if you should push/pop things and the documentation was not always clear to me. Has this changed any?
  - ufo 2y ago
    The stack's still there. I agree that reference manual's notation for how many values are pushed and popped can take a while to get used to, but at least it's straight to the point. One trap with the stack is that it baits you into carefully sequencing the operations so that the output of one feeds into the input the the next. Sometimes values are popped far from the places that pushed them... It can be easier to reason about code that liberally copies the temporary values. Keep one stack slot for each "local variable" you want to work with. Then to work on them you copy that slot to the top, call the stack operation, and then write the result back to the appropriate stack slot. Essentially, favor positive stack indices over negative indices, because the latter are more sensitive to the sequencing of your operations. Also, consider giving names to the stack indices instead of hardcoded numbers.
jakupovic 2y ago
You can barely do anything before having to extend the language in a way. It's super simple and as such gets in the way as even the simplest stuff needs some extending/(much attention). Admittedly I have little of experience with Lua, did the work and moved on.

helpfulContrib 2y ago

[dead]

ck45 2y ago

Did you consider any other languages, like TCL? If yes, do you remember the pros and cons of that set of languages?

progre 2y ago
Was the first version of Redis not in fact written in TCL?
- ck45 2y ago
  Oh, interesting, I didn’t know. ChatGPT “confirms” that a prototype was written in TCL

matheusmoreira 2y ago

> the hostility is present even at C-API level, with the stack approach > Why? I suspect it's because implementing control mechanisms is much easier when you reify the language's stack. Especially advanced ones such as generators and continuations which require copying the stack into an object and restoring it later. Making that work with the native stack is really hard and many languages don't even try. It also makes garbage collection precise. Lua values in C variables would be placed in registers or the native stack. In order to trace those values, Lua would require a conservative garbage collector that spills the registers and scans the entire native stack. By managing their own stack, they can avoid doing all that.

peutetre 2y ago

WebAssembly is a nice way to go for such things in future. You provide the interface, their wasm code runs in a sandbox, and your users get to write their plugins in whatever language they like.

deleted 2y ago
[deleted]
unleaded 2y ago
This gets mentioned a lot but has it been used in practice? How would you pass objects and methods etc to the scripting language? Autogenerated C headers?
- brabel 2y ago
  To be able to use WASM-compiled languages for embedding you first need a small WASM runtime you can depend on (and things like ref types, GC etc which have been standardized only in the last couple of years). That didn't exist until recently, but now you can use https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/doc/embed_wamr.md https://github.com/bytecodealliance/wasm-micro-runtime/blob/... I am not aware of anyone using this yet, but I hope to see that become common in the next few years.
- peutetre 2y ago
  The WebAssembly component model is aimed at having composable components that can call each other. The components can be written in any language, compiled to WebAssembly, and interoperate: https://github.com/WebAssembly/component-model/ https://github.com/WebAssembly/component-model/ Extism is a plugin framework for WebAssembly: https://extism.org/ https://extism.org/ https://github.com/extism/extism https://github.com/extism/extism Postgres and Extism: https://dylibso.com/blog/pg-extism/ https://dylibso.com/blog/pg-extism/ Wasm plugins for Zellij: https://zellij.dev/documentation/plugins https://zellij.dev/documentation/plugins Visual Studio Code can run WebAssembly extensions: https://code.visualstudio.com/blogs/2024/05/08/wasm https://code.visualstudio.com/blogs/2024/05/08/wasm A project to bring WebAssembly plugins to Godot: https://github.com/ashtonmeuser/godot-wasm https://github.com/ashtonmeuser/godot-wasm WasmEdge can be embedded in applications: https://wasmedge.org/docs/embed/overview https://wasmedge.org/docs/embed/overview Wasmer can be embedded in applications: https://blog.wasmer.io/executing-webassembly-in-your-rust-application-d5cd32e8ce46 https://blog.wasmer.io/executing-webassembly-in-your-rust-ap... Wasmtime can be embedded in applications: https://docs.wasmtime.dev/lang.html https://docs.wasmtime.dev/lang.html

spacechild1 2y ago

> Why? See https://news.ycombinator.com/item?id=42519033 https://news.ycombinator.com/item?id=42519033

astrobe_ 2y ago

A historical thing: > Traditionally, most virtual machines intended for actual execution are stack based, a trend that started with Pascal’s P-machine and continues today with Java’s JVM and Microsoft’s .Net environment. Currently, however, there has been a growing interest in register-based virtual machines (for instance, the planned new virtual machine for Perl 6 (Parrot) will be register based). As far as we know, the virtual machine of Lua 5.0 is the first register-based virtual machine to have a wide use. The API directly reflected the (previous) internals of the VM, I guess [1]. [1] (pdf) https://www.lua.org/doc/jucs05.pdf https://www.lua.org/doc/jucs05.pdf

ufo 2y ago
The new register-based virtual machine still uses the same stack under the hood. The only difference is how instructions are encoded in the bytecode. By default they can read and write to any position in the stack as opposed to pushing and popping from the top. The Lua stack API already does some of this, because several operations take stack indices as arguments.

ufo 2y ago

Have you seen the Passing a Language Through the Eye of a Needle article, from the Lua authors? It talks about how some of unusual the design decisions such as pcall error handling and the stack are tradeoffs in favor of "embeddability". https://queue.acm.org/detail.cfm?id=1983083 https://queue.acm.org/detail.cfm?id=1983083 The main purpose of the stack is to facilitate garbage collection. The GC can find out what Lua objects are currently being manipulated by the C client. The price is that the Lua API can never expose a "Lua object pointer" to C. Many other scripting languages expose such pointers, but then must also expose ways to manage their memory. For example, in Python the user of the API must explicitly increment and decrement reference counts.

ranger_danger 2y ago
The one time I tried to embed Lua in a C project (which included C-based functions I could call from Lua and setting up variables from C that were visible in Lua), I constantly struggled with trying to figure out when I needed to push/pop things from the stack and it just seemed very error-prone and easy to leak memory with. Different functions seem to have different requirements for when/if you should push/pop things and the documentation was not always clear to me. Has this changed any?
- ufo 2y ago
  The stack's still there. I agree that reference manual's notation for how many values are pushed and popped can take a while to get used to, but at least it's straight to the point. One trap with the stack is that it baits you into carefully sequencing the operations so that the output of one feeds into the input the the next. Sometimes values are popped far from the places that pushed them... It can be easier to reason about code that liberally copies the temporary values. Keep one stack slot for each "local variable" you want to work with. Then to work on them you copy that slot to the top, call the stack operation, and then write the result back to the appropriate stack slot. Essentially, favor positive stack indices over negative indices, because the latter are more sensitive to the sequencing of your operations. Also, consider giving names to the stack indices instead of hardcoded numbers.

jakupovic 2y ago

You can barely do anything before having to extend the language in a way. It's super simple and as such gets in the way as even the simplest stuff needs some extending/(much attention). Admittedly I have little of experience with Lua, did the work and moved on.