5 ms·

I've never used rust before, but have been meaning to look into it and play around some. OP mentions the include_str! function(maybe its a function, not sure wh

I've never used rust before, but have been meaning to look into it and play around some. OP mentions the include_str! function(maybe its a function, not sure what '!' signifies here) and it blew my mind. I'm betting that almost every language that I've used has something like this, but the succinctness of it and how it's used to keep the source clean, but still compile in the text is really cool.

empath75 9y ago
It's a macro.
thezoq2 9y ago
the ! indicates that it is a macro, so include_string! is run at compile time.
anderspitman 9y ago
I know what you mean! I first learned about `include_str!` more than a year ago and I've wanted to try this ever since. The '!' means `include_str!` is a macro in Rust, similar to a C macro in function, but Rust has a much better system for macros and ships with many useful ones.
hie2neiL 9y ago
Normally webservers serve files with the sendfile system call for zero copy IO. If the string is statically included in the binary then it first has to copy it into kernel space to send the data. So this is actually less efficient. In theory this could be re-optimized with vmsplice, but I wonder whether that webserver actually does it because vmsplice's lifetime requirements are... complex (although a &'static str would meet those requirements). Or the binary could sendfile itself if it knew the position of the string in its binary image, but that information is not preserved by a &str.
- NiceGuy_Ty 9y ago
  Putting the string in the binary is really only useful for trying to do ultra-portable deployments like this imo. The overhead of copying the string into memory is pretty meaningless in the use case where you only expect a single, home-desktop user. Now if this was a production server having to service thousands of clients than the sendfile optimization becomes much more important.
  - XorNot 9y ago
    Even in production though, you can just have a step on startup which copies built-in resources to a /tmp location and then sendfile's from there.
- geofft 9y ago
  > Normally webservers serve files with the sendfile system call for zero copy IO. If the string is statically included in the binary then it first has to copy it into kernel space to send the data. So this is actually less efficient. This doesn't sound right to me - sendfile's "zero copy" is about eliminating the extra copy when you read() a file into a userspace buffer and then write() that file into a socket. If you have a memory-mapped file (which is what your executable itself is), and you pass an address inside that mapped region to write(), I would hope that the kernel will just access the buffers of that file directly - i.e., the fact that it's memory mapped means there is no separate copy of the file, there's a virtual page that references the single kernel buffer for that file. So at that point write() should behave just like sendfile(). If you need to process the file in the kernel itself, there will still be a copy, but you'd have that in either case. If you don't, I would again hope that both write() from a memory-mapped region and sendfile() know how to DMA the file from the disk to the network card. I am not confident about this and it seems worth benchmarking. (Also, your production server probably uses HTTPS, at which point this is all irrelevant if your TLS encryption happens in userspace. If you're using in-kernel TLS, then you're definitely not doing DMA, and I would strongly hope that sendfile() and write() from a mapped buffer involve the same number of copies - one read from disk, at which point the kernel encrypts it into a writable buffer.)
- always_good 9y ago
  > Normally webservers serve files with the sendfile system call for zero copy IO Not if they are doing their own tls encryption which hopefully one day is the new normal.
tmzt 9y ago
The ! indicates the object you are "calling" is a macro, which essentially replaces that call with the equivalent syntax which is produced by applying the macro. In addition to the older hygenic macros (which have their own syntax to define them), Rust also supports proc_macros, which are actually code that is executed by the compiler and can preprocess a struct or function. For more information, see the Rust Book (second edition) section on macros: https://doc.rust-lang.org/book/second-edition/appendix-04-macros.html https://doc.rust-lang.org/book/second-edition/appendix-04-ma...
phyllostachys 9y ago
Others have mentioned that include_str! is a macro, which I suppose it is. But if you look at the source it says that it is a compiler built-in[0]. If you dig into that, you'll see that it happens inside the compiler[1] (well duh). I feel that that makes it slightly different than a normal macro in that it doesn't expand to a bunch of Rust code. I found that pretty neat. [0] - https://github.com/rust-lang/rust/blob/master/src/libcore/macros.rs#L730-L741 https://github.com/rust-lang/rust/blob/master/src/libcore/ma... [1] - https://github.com/rust-lang/rust/blob/e8af0f4c1f121263e55da29854208db0ae1fea54/src/libsyntax/ext/source_util.rs#L132-L166 https://github.com/rust-lang/rust/blob/e8af0f4c1f121263e55da...

empath75 9y ago

It's a macro.

thezoq2 9y ago

the ! indicates that it is a macro, so include_string! is run at compile time.

anderspitman 9y ago

I know what you mean! I first learned about `include_str!` more than a year ago and I've wanted to try this ever since. The '!' means `include_str!` is a macro in Rust, similar to a C macro in function, but Rust has a much better system for macros and ships with many useful ones.

hie2neiL 9y ago

Normally webservers serve files with the sendfile system call for zero copy IO. If the string is statically included in the binary then it first has to copy it into kernel space to send the data. So this is actually less efficient. In theory this could be re-optimized with vmsplice, but I wonder whether that webserver actually does it because vmsplice's lifetime requirements are... complex (although a &'static str would meet those requirements). Or the binary could sendfile itself if it knew the position of the string in its binary image, but that information is not preserved by a &str.

NiceGuy_Ty 9y ago
Putting the string in the binary is really only useful for trying to do ultra-portable deployments like this imo. The overhead of copying the string into memory is pretty meaningless in the use case where you only expect a single, home-desktop user. Now if this was a production server having to service thousands of clients than the sendfile optimization becomes much more important.
- XorNot 9y ago
  Even in production though, you can just have a step on startup which copies built-in resources to a /tmp location and then sendfile's from there.
geofft 9y ago
> Normally webservers serve files with the sendfile system call for zero copy IO. If the string is statically included in the binary then it first has to copy it into kernel space to send the data. So this is actually less efficient. This doesn't sound right to me - sendfile's "zero copy" is about eliminating the extra copy when you read() a file into a userspace buffer and then write() that file into a socket. If you have a memory-mapped file (which is what your executable itself is), and you pass an address inside that mapped region to write(), I would hope that the kernel will just access the buffers of that file directly - i.e., the fact that it's memory mapped means there is no separate copy of the file, there's a virtual page that references the single kernel buffer for that file. So at that point write() should behave just like sendfile(). If you need to process the file in the kernel itself, there will still be a copy, but you'd have that in either case. If you don't, I would again hope that both write() from a memory-mapped region and sendfile() know how to DMA the file from the disk to the network card. I am not confident about this and it seems worth benchmarking. (Also, your production server probably uses HTTPS, at which point this is all irrelevant if your TLS encryption happens in userspace. If you're using in-kernel TLS, then you're definitely not doing DMA, and I would strongly hope that sendfile() and write() from a mapped buffer involve the same number of copies - one read from disk, at which point the kernel encrypts it into a writable buffer.)
always_good 9y ago
> Normally webservers serve files with the sendfile system call for zero copy IO Not if they are doing their own tls encryption which hopefully one day is the new normal.

tmzt 9y ago

The ! indicates the object you are "calling" is a macro, which essentially replaces that call with the equivalent syntax which is produced by applying the macro. In addition to the older hygenic macros (which have their own syntax to define them), Rust also supports proc_macros, which are actually code that is executed by the compiler and can preprocess a struct or function. For more information, see the Rust Book (second edition) section on macros: https://doc.rust-lang.org/book/second-edition/appendix-04-macros.html https://doc.rust-lang.org/book/second-edition/appendix-04-ma...

phyllostachys 9y ago

Others have mentioned that include_str! is a macro, which I suppose it is. But if you look at the source it says that it is a compiler built-in[0]. If you dig into that, you'll see that it happens inside the compiler[1] (well duh). I feel that that makes it slightly different than a normal macro in that it doesn't expand to a bunch of Rust code. I found that pretty neat. [0] - https://github.com/rust-lang/rust/blob/master/src/libcore/macros.rs#L730-L741 https://github.com/rust-lang/rust/blob/master/src/libcore/ma... [1] - https://github.com/rust-lang/rust/blob/e8af0f4c1f121263e55da29854208db0ae1fea54/src/libsyntax/ext/source_util.rs#L132-L166 https://github.com/rust-lang/rust/blob/e8af0f4c1f121263e55da...