4 ms·

Tangentially, I've been exploring something kind of fascinating... So, in compiled binary executables it was not uncommon to include encoded binary resources,

Tangentially, I've been exploring something kind of fascinating... So, in compiled binary executables it was not uncommon to include encoded binary resources, like files or images. Not tons of them, because they would explode the size of the executable. For example, for C or C++ https://github.com/graphitemaster/incbin https://github.com/graphitemaster/incbin So here's the interesting part - we decided to build executables in a way that data in those executables could not be changed (makes sense since it's compiled and the compiled code runs on the machine, and for security reasons). That said, think about what a container (e.g. docker) is. A running container is kind of like a packaged executable, except it also has a filesystem. So if you pack data (which can be modified and runtime!) into a container, it's a similar concept to an embedded resource in an executable, except it can change. Now, in a container, any changes made to data at runtime inside the container won't persist unless the container is given persistent storage. What I've been wondering lately is why we didn't invent some kind of single "executable plus volatile data space embedded within the executable", so that programs and data (say, a database) could couple together into a single file. Just a musing but tangentially related to your "baked data" - basically, embedded resources in executables just embeds the encoded data right into an executable. For scripting languages, of course, we can just make a script file that contains a variable with the encoded data as base64 directly. For the latter 2, it's only good for relatively small static data of course, but it would be interesting to build tech that somehow lifted that constraint of executables.

zX41ZdbW 3y ago
Thanks for the link. We also make static websites by direct inclusion their HTML code in the C++ binary of ClickHouse. I've tried to use this library but found a bug: https://github.com/graphitemaster/incbin/issues/61 https://github.com/graphitemaster/incbin/issues/61
simonw 3y ago
Have you seen redbean? It's exploring ideas that are in that kind of space: https://redbean.dev/ https://redbean.dev/
- sixdimensional 3y ago
  I had not, thanks for sharing this is really interesting! I see "Self-Modifying PKZIP Object Store" and this is very much what I had in mind. Neat, will have a look!
IIsi50MHz 3y ago
> That said, think about what a container (e.g. docker) is. A running container is kind of like a packaged executable, except it also has a filesystem. > So if you pack data (which can be modified and runtime!) into a container, it's a similar concept to an embedded resource in an executable, except it can change. > Now, in a container, any changes made to data at runtime inside the container won't persist unless the container is given persistent storage. > What I've been wondering lately is why we didn't invent some kind of single "executable plus volatile data space embedded within the executable", so that programs and data (say, a database) could couple together into a single file. Things like this have existed before, and I presume still exist. For instance, before MacOS X, applications for Macintosh for (vaguely) similar to "application bundles. Leaving aside the technical differences of implementation, classic Macintosh applications had a "resource" store that included code and 'static' data (pictures, text, whatever) but could be modified by running program. Changes could be persisted without using any other files. In practice, changes were often persisted elsewhere, be cause if a fault occurred during modification, the executable could be corrupted, and because this made things like resetting to defaults easier (by deleting or moving an external preferences file, for example).

zX41ZdbW 3y ago

Thanks for the link. We also make static websites by direct inclusion their HTML code in the C++ binary of ClickHouse. I've tried to use this library but found a bug: https://github.com/graphitemaster/incbin/issues/61 https://github.com/graphitemaster/incbin/issues/61

simonw 3y ago

Have you seen redbean? It's exploring ideas that are in that kind of space: https://redbean.dev/ https://redbean.dev/

sixdimensional 3y ago
I had not, thanks for sharing this is really interesting! I see "Self-Modifying PKZIP Object Store" and this is very much what I had in mind. Neat, will have a look!

IIsi50MHz 3y ago

> That said, think about what a container (e.g. docker) is. A running container is kind of like a packaged executable, except it also has a filesystem. > So if you pack data (which can be modified and runtime!) into a container, it's a similar concept to an embedded resource in an executable, except it can change. > Now, in a container, any changes made to data at runtime inside the container won't persist unless the container is given persistent storage. > What I've been wondering lately is why we didn't invent some kind of single "executable plus volatile data space embedded within the executable", so that programs and data (say, a database) could couple together into a single file. Things like this have existed before, and I presume still exist. For instance, before MacOS X, applications for Macintosh for (vaguely) similar to "application bundles. Leaving aside the technical differences of implementation, classic Macintosh applications had a "resource" store that included code and 'static' data (pictures, text, whatever) but could be modified by running program. Changes could be persisted without using any other files. In practice, changes were often persisted elsewhere, be cause if a fault occurred during modification, the executable could be corrupted, and because this made things like resetting to defaults easier (by deleting or moving an external preferences file, for example).