6 ms·

Being able to include arbitrary data inside a static binary is definitely a nice side-effect of rust and golang being able to produce static binaries. If you'r

Being able to include arbitrary data inside a static binary is definitely a nice side-effect of rust and golang being able to produce static binaries. If you're interested in stuff like this for go, I've used https://github.com/GeertJohan/go.rice https://github.com/GeertJohan/go.rice with great success. [EDIT] "feature" -> "side effect" "rust and golang" -> "rust and golang being able to produce static binaries"

hpkuarg 9y ago
How does go.rice compare with go-bindata[0]? I only have experience with the latter. [0] https://github.com/jteeuwen/go-bindata https://github.com/jteeuwen/go-bindata
- hardwaresofton 9y ago
  I haven't compared them (as I haven't used go-bindata), but go.rice has a pretty ergonomic API -- I once took a look at go-bindata, but I much prefer go.rice. go.rice also seems more active in general.
anderspitman 9y ago
This looks really cool. I'd love to have a library for Rust to make this more seamless. In particular it would be nice to have a "development mode" where you just host the HTML/JS files normally while you're working on them, then flip a production switch when you're ready to have them compiled into the binary. Does go.rice have something like this?
- steveklabnik 9y ago
  I’ve come halfway to building this a few times; it’d be extremely helpful for mdbook, for example.
- hardwaresofton 9y ago
  I actually thought that's what Rouille (https://github.com/tomaka/rouille https://github.com/tomaka/rouille) was, which was mentioned in the blog post, but it looks like just uses the macro from that package (to include the strings), not the actual library even, since at the end of the day he opts for rocket (https://rocket.rs/ https://rocket.rs/) [EDIT] - I didn't address your question about go.rice -- from what I know, it doesn't have this kind of thing built in, and I don't know that it even should... "flipping a production switch" is a pretty application-specific endeavor. Also this sounds like something you should be doing with your build tools..., can't include something in the binary at runtime. Maybe I misunderstood what you were asking. To clarify, the description for go.rice is: > go.rice is a Go package that makes working with resources such as html,js,css,images,templates, etc very easy. During development go.rice will load required files directly from disk. Upon deployment it is easy to add all resource files to a executable using the rice tool, without changing the source code for your package. go.rice provides several methods to add resources to a binary.
  - anderspitman 9y ago
    Actually the include_str! macro is shipped with Rust. The router! macro for setting up the endpoints comes from Rouille. My question is really geared towards this pain point: the React/JS stuff can be quickly rebuilt for almost instant feedback. But even for a simple app like this example the Rust compilation takes a couple seconds on my system. You have to pay that cost even if you're just updating the JS because it has to be built into the binary. It would be nice if you're not changing the actual Rust code to be able to easily reload your JS without compiling.
    - hardwaresofton 9y ago
      Thanks for the clarification! I assumed it was from Rouille... I went back and re-read the area, and he's using Rouille, NOT Rocket -- I didn't notice this the first time through, didn't read closely enough. Oh I think you absolutely can have that kind of flow -- go.rice DOES support that. Also, I still don't really understand because this problem seems to be easily solved by just changing your build script, or detecting environment at runtime. You could even check for the data, and if it's not present, fall back to disk. I don't know a library that does it off the top of my head for rust though, since I'm not that familiar with rust dev.
- notheguyouthink 9y ago
  Assuming I understand you, it would be easy via an interface. Build an interface for the go.rice (or go-binddata/etc) such that, depending on the environment, it either pulls the file from disk or from memory. Super easy. I seem to recall seeing a Go library that actually has this functionality built in, but I've got no idea which it was.
- scottlamb 9y ago
  That feature's on my (currently just mental) TODO list for my http-serve crate (https://github.com/scottlamb/http-serve https://github.com/scottlamb/http-serve). I want to use it in my moonfire-nvr app. I'm thinking something a little fancier than include_str! because I want to include a whole subdir of resources, and I probably want to embed the gzipped version (and uncompress into RAM for when there's no "Accept-Encoding: gzip") rather than the reverse.
AlphaSite 9y ago
Are these basically statically linked jars?
- hardwaresofton 9y ago
  Rust can create static binaries -- AFAIK there isn't even such a thing as a "statically linked" jar -- if you're relying on a VM (the JVM in this case), your program isn't statically compiled. It's not even "compiled" in the strict traditional sense, since the target isn't the machine, it's an VM (JVM). But yes, it is just like including files in your classpath (and making sure they end up in the jar), then serving them from a Netty endpoint or something.
  - pjmlp 9y ago
    Yes it can be compiled into pure VM free Assembly. It is only a matter of buying one of the commercial JDKs, or if doing GPL stuff use the open source license from ExcelsiorJET. Those not willing to pay and already on Java 9/10, running on Linux x64, can play with the initial AOT support.
    - hardwaresofton 9y ago
      > Are these basically statically linked jars? A JAR is meant to run on the JVM. There is no such thing as a "statically linked jar". The process you're referring to is more like compiling bytecode to assembly, you're converting a JAR, a thing meant to run on the JVM, to native machine code (with assembly as the step in between). > it's only a matter of ... Yeah, it sounds simple in theory, but somehow I don't find many projects these days that use these commercial JDKs to generate VM free assembly. In fact, I don't even know one big tech company that does so -- maybe you could enlighten me. > Those not willing to pay and already on Java 9/10, running on Linux x64, can play with the initial AOT support. So just about the newest version of Java just got initial support? Cool. Well this brand new community-led thoroughly open-source language has this as a core tenet, and supports compiling to many platforms very easily. Kudos to Java for getting better, embracing a more open development methodology (I think this has been the case since 8), listening to the community more about which features to include, but when compared to projects that took a fresh look at all these concepts, and started with the open community-based approach, Java doesn't compare (except in terms of speed, JVM is pretty heckin' fast).
planteen 9y ago
Yeah its definitely nice to avoid an annoying "binary" to source step and then have to waste a bunch of cycles to parse/compile said source. I do embedded work and run into this all the time with C and C++. Its toolchain dependent on how to import binary blobs unfortunately in C/C++. You very quickly run into a wall on how large of a source file can be compiled around 100 MB or so. gcc is the best at handling huge source files, but its still slow. For example, you can do the import with gcc this way: https://balau82.wordpress.com/2012/02/19/linking-a-binary-blob-with-gcc/ https://balau82.wordpress.com/2012/02/19/linking-a-binary-bl... With Green Hills, there is a .rawimport directive.
pjmlp 9y ago
Maybe for younger generations. Apple and Microsoft systems have been supporting this on their native toolchains since the mid-90's.
- IshKebab 9y ago
  There still isn't a good built-in way to do this portably in C++.
  - deleted 9y ago
    [deleted]
  - pknopf 9y ago
    Qt has the moc/resource compiler that can do this for you. Not that you would/should shoehorn qmake into your project for it though, but they do it nonetheless. Make you can use qmake as a prebuild step and include the generated moc files in your build? I wonder how many qt headers are in the generated source files.
    - blattimwind 9y ago
      qmake is not the only way to build Qt projects; CMake is quite straightforward and qbs works extremely well, too, though is rarely used in the wild (pity). I'm sure other build systems have sugar for Qt as well; CMake and qmake are just wildly popular.
- hardwaresofton 9y ago
  Yes, including arbitrary data in binaries is probably as old as making binaries itself. I think what is new is that in the mid 90s you didn't have an open source, memory safe, ergonomic, strongly typed, type inferenced language that targets multiple platforms with ease, with a decent standard lib. Rust is delivering that. Golang is delivering that. I think those two are novel in that sense. From where I'm sitting the last few decades have been ruled by VMs and the explosion of scripting languages, because no one wanted to use those Apple/Microsoft toolchains that have been around in the 90s. While those toolchains rested on their laurels (or didn't, can't tell the difference now), Java became the enterprise workhorse.

hpkuarg 9y ago

How does go.rice compare with go-bindata[0]? I only have experience with the latter. [0] https://github.com/jteeuwen/go-bindata https://github.com/jteeuwen/go-bindata

hardwaresofton 9y ago
I haven't compared them (as I haven't used go-bindata), but go.rice has a pretty ergonomic API -- I once took a look at go-bindata, but I much prefer go.rice. go.rice also seems more active in general.

anderspitman 9y ago

This looks really cool. I'd love to have a library for Rust to make this more seamless. In particular it would be nice to have a "development mode" where you just host the HTML/JS files normally while you're working on them, then flip a production switch when you're ready to have them compiled into the binary. Does go.rice have something like this?

steveklabnik 9y ago
I’ve come halfway to building this a few times; it’d be extremely helpful for mdbook, for example.
hardwaresofton 9y ago
I actually thought that's what Rouille (https://github.com/tomaka/rouille https://github.com/tomaka/rouille) was, which was mentioned in the blog post, but it looks like just uses the macro from that package (to include the strings), not the actual library even, since at the end of the day he opts for rocket (https://rocket.rs/ https://rocket.rs/) [EDIT] - I didn't address your question about go.rice -- from what I know, it doesn't have this kind of thing built in, and I don't know that it even should... "flipping a production switch" is a pretty application-specific endeavor. Also this sounds like something you should be doing with your build tools..., can't include something in the binary at runtime. Maybe I misunderstood what you were asking. To clarify, the description for go.rice is: > go.rice is a Go package that makes working with resources such as html,js,css,images,templates, etc very easy. During development go.rice will load required files directly from disk. Upon deployment it is easy to add all resource files to a executable using the rice tool, without changing the source code for your package. go.rice provides several methods to add resources to a binary.
- anderspitman 9y ago
  Actually the include_str! macro is shipped with Rust. The router! macro for setting up the endpoints comes from Rouille. My question is really geared towards this pain point: the React/JS stuff can be quickly rebuilt for almost instant feedback. But even for a simple app like this example the Rust compilation takes a couple seconds on my system. You have to pay that cost even if you're just updating the JS because it has to be built into the binary. It would be nice if you're not changing the actual Rust code to be able to easily reload your JS without compiling.
  - hardwaresofton 9y ago
    Thanks for the clarification! I assumed it was from Rouille... I went back and re-read the area, and he's using Rouille, NOT Rocket -- I didn't notice this the first time through, didn't read closely enough. Oh I think you absolutely can have that kind of flow -- go.rice DOES support that. Also, I still don't really understand because this problem seems to be easily solved by just changing your build script, or detecting environment at runtime. You could even check for the data, and if it's not present, fall back to disk. I don't know a library that does it off the top of my head for rust though, since I'm not that familiar with rust dev.
notheguyouthink 9y ago
Assuming I understand you, it would be easy via an interface. Build an interface for the go.rice (or go-binddata/etc) such that, depending on the environment, it either pulls the file from disk or from memory. Super easy. I seem to recall seeing a Go library that actually has this functionality built in, but I've got no idea which it was.
scottlamb 9y ago
That feature's on my (currently just mental) TODO list for my http-serve crate (https://github.com/scottlamb/http-serve https://github.com/scottlamb/http-serve). I want to use it in my moonfire-nvr app. I'm thinking something a little fancier than include_str! because I want to include a whole subdir of resources, and I probably want to embed the gzipped version (and uncompress into RAM for when there's no "Accept-Encoding: gzip") rather than the reverse.

AlphaSite 9y ago

Are these basically statically linked jars?

hardwaresofton 9y ago
Rust can create static binaries -- AFAIK there isn't even such a thing as a "statically linked" jar -- if you're relying on a VM (the JVM in this case), your program isn't statically compiled. It's not even "compiled" in the strict traditional sense, since the target isn't the machine, it's an VM (JVM). But yes, it is just like including files in your classpath (and making sure they end up in the jar), then serving them from a Netty endpoint or something.
- pjmlp 9y ago
  Yes it can be compiled into pure VM free Assembly. It is only a matter of buying one of the commercial JDKs, or if doing GPL stuff use the open source license from ExcelsiorJET. Those not willing to pay and already on Java 9/10, running on Linux x64, can play with the initial AOT support.
  - hardwaresofton 9y ago
    > Are these basically statically linked jars? A JAR is meant to run on the JVM. There is no such thing as a "statically linked jar". The process you're referring to is more like compiling bytecode to assembly, you're converting a JAR, a thing meant to run on the JVM, to native machine code (with assembly as the step in between). > it's only a matter of ... Yeah, it sounds simple in theory, but somehow I don't find many projects these days that use these commercial JDKs to generate VM free assembly. In fact, I don't even know one big tech company that does so -- maybe you could enlighten me. > Those not willing to pay and already on Java 9/10, running on Linux x64, can play with the initial AOT support. So just about the newest version of Java just got initial support? Cool. Well this brand new community-led thoroughly open-source language has this as a core tenet, and supports compiling to many platforms very easily. Kudos to Java for getting better, embracing a more open development methodology (I think this has been the case since 8), listening to the community more about which features to include, but when compared to projects that took a fresh look at all these concepts, and started with the open community-based approach, Java doesn't compare (except in terms of speed, JVM is pretty heckin' fast).

planteen 9y ago

Yeah its definitely nice to avoid an annoying "binary" to source step and then have to waste a bunch of cycles to parse/compile said source. I do embedded work and run into this all the time with C and C++. Its toolchain dependent on how to import binary blobs unfortunately in C/C++. You very quickly run into a wall on how large of a source file can be compiled around 100 MB or so. gcc is the best at handling huge source files, but its still slow. For example, you can do the import with gcc this way: https://balau82.wordpress.com/2012/02/19/linking-a-binary-blob-with-gcc/ https://balau82.wordpress.com/2012/02/19/linking-a-binary-bl... With Green Hills, there is a .rawimport directive.

pjmlp 9y ago

Maybe for younger generations. Apple and Microsoft systems have been supporting this on their native toolchains since the mid-90's.

IshKebab 9y ago
There still isn't a good built-in way to do this portably in C++.
- deleted 9y ago
  [deleted]
- pknopf 9y ago
  Qt has the moc/resource compiler that can do this for you. Not that you would/should shoehorn qmake into your project for it though, but they do it nonetheless. Make you can use qmake as a prebuild step and include the generated moc files in your build? I wonder how many qt headers are in the generated source files.
  - blattimwind 9y ago
    qmake is not the only way to build Qt projects; CMake is quite straightforward and qbs works extremely well, too, though is rarely used in the wild (pity). I'm sure other build systems have sugar for Qt as well; CMake and qmake are just wildly popular.
hardwaresofton 9y ago
Yes, including arbitrary data in binaries is probably as old as making binaries itself. I think what is new is that in the mid 90s you didn't have an open source, memory safe, ergonomic, strongly typed, type inferenced language that targets multiple platforms with ease, with a decent standard lib. Rust is delivering that. Golang is delivering that. I think those two are novel in that sense. From where I'm sitting the last few decades have been ruled by VMs and the explosion of scripting languages, because no one wanted to use those Apple/Microsoft toolchains that have been around in the 90s. While those toolchains rested on their laurels (or didn't, can't tell the difference now), Java became the enterprise workhorse.