4 ms·

Honestly for your use case, what if you made the interpreter executable a ZIP archive instead? Then any zip archiver could add new files/programs.

matheusmoreira 2y ago
Many people suggested that I do that. I believe it's the standard way to embed files into executables. Open the executable and read backwards from the end until the zip archive metadata is found. My goal was to do it in such a way that the Linux kernel memory mapped the embedded file section before the process had even started running. I explicitly wanted to avoid doing things like opening /proc/self/exe and reading its contents into memory. My interpreter is freestanding and has zero dependencies. This solution fits the minimalist spirit of the thing. I wrote an article detailing how it works: https://www.matheusmoreira.com/articles/self-contained-lone-lisp-applications https://www.matheusmoreira.com/articles/self-contained-lone-... Long story short: Linux passes a pointer to the ELF program header table to the process via the auxiliary vector which means the program segments are already reachable from inside the program, and I took advantage of that by adding custom load segments and making the interpreter search for them at runtime. This little mechanism allows embedding arbitrary data, including zip files. I made a little lisp S-expression file format which the interpreter simply parses but it could have been anything. Zip archivers should be able to edit the data in place, it should work fine provided the segments are updated with the new size of the zip archive. The practical application of this is to enable the creation of a freestanding, self-executing application by making a copy of an existing interpreter and adding the desired code and data to it, even though the interpreter has already been compiled and linked: cp lone my-app lone-embed my-app my-app.ln ./my-app The interpreter introspects, discovers there's an embedded segment inside itself and evaluates it. Unfortunately I had to write a tool to append those segments. Never figured out how to do it with linker script.

matheusmoreira 2y ago

Many people suggested that I do that. I believe it's the standard way to embed files into executables. Open the executable and read backwards from the end until the zip archive metadata is found. My goal was to do it in such a way that the Linux kernel memory mapped the embedded file section before the process had even started running. I explicitly wanted to avoid doing things like opening /proc/self/exe and reading its contents into memory. My interpreter is freestanding and has zero dependencies. This solution fits the minimalist spirit of the thing. I wrote an article detailing how it works: https://www.matheusmoreira.com/articles/self-contained-lone-lisp-applications https://www.matheusmoreira.com/articles/self-contained-lone-... Long story short: Linux passes a pointer to the ELF program header table to the process via the auxiliary vector which means the program segments are already reachable from inside the program, and I took advantage of that by adding custom load segments and making the interpreter search for them at runtime. This little mechanism allows embedding arbitrary data, including zip files. I made a little lisp S-expression file format which the interpreter simply parses but it could have been anything. Zip archivers should be able to edit the data in place, it should work fine provided the segments are updated with the new size of the zip archive. The practical application of this is to enable the creation of a freestanding, self-executing application by making a copy of an existing interpreter and adding the desired code and data to it, even though the interpreter has already been compiled and linked: cp lone my-app lone-embed my-app my-app.ln ./my-app The interpreter introspects, discovers there's an embedded segment inside itself and evaluates it. Unfortunately I had to write a tool to append those segments. Never figured out how to do it with linker script.