14 ms·

Go command support for embedded static assets (files)

211 points by eberkund 6y ago

nicoburns 6y ago
Seems pretty complicated compared to Rust's `include_bytes!("path/to/file")`. Sounds like it might have a compile time advantage though, seeing as it produces a seperate package.
- opqpo 6y ago
  Everything in Golang seems pretty complicated and nonintuitive compared to any other modern language. But "it's all well thought out"!
  - jimktrains2 6y ago
    Every nil panic and manual type assertion holds a potentially very nice language back. They just seem like relics we should abandon.
    - opqpo 6y ago
      Accessing nil pointers is well thought out. Not enforcing error checking is well thought out. Reflection and code generators instead of generics and traits is well thought out. Lack of conditional compilation is well thought out. No compilation with unused variables instead of warnings is well thought out. Doing your own sort, map, all essential array operations from scratch everytime is well thought out. Spamming err != nil instead of some operator like `?` is well thought out. iota instead of type-safe Enums is well thought out. No file-scope variables is well thought out. go mod is well thought out compared to npm and cargo.
      - leetrout 6y ago
        Go doesn’t have to be bad for all those things you like in Scala/Rust/etc to be true. It’s a simple language with simple, explicit, verbose, tedious, pedantic patterns. The communication resulting from those patterns is where the value lies for a lot of developers and teams.
        opqpo 6y ago
        Mediocrity, ugliness and verbosity is more well thought out than engineering.
        leetrout 6y ago
        Sure - I know you disagree - I was just pointing out a Ferrari and a school bus can both exist and be good for different purposes.
        opqpo 6y ago
        disagreeing is a pretty strong word. Nobody can disagree these days. I was flagged and labeled as a troll within a couple of minutes once I criticized a programming language. 30 years from now, people will remember this pathetic mob tyranny era when one couldn't even offend a programming language feelings!
        hu3 6y ago
        If you allow me an advice, I'd rather not waste precious time with trolls. That user has multiple flagged messages in this month alone.
        jimktrains2 6y ago
        Before using it I thought the if err != Nil would be annoying, but it actually isn't. I think a sum type would be better, but the pattern is common and honestly not very different from a match on a sum type. In general I agree that it does lead to code that's more familiar even if it was written by someone else. Having only used it a few months now, I can definitely see where the value in go lies.
- Thaxll 6y ago
  It seems much more powerful than the Rust version. The Rust version basicaly embed the file into a byte array that's it or I'm missing something?
  - nightfly 6y ago
    If you need more check out the rust-embed crate.
  - euank 6y ago
    This proposal is more powerful than that one rust macro, but rust's abilities around embedding files are much more powerful than go's approach. This proposal allows "go build" to embed things in a very specific way, but it's not meant to be extensible. Rust's 'include_bytes!' macro on the other hand is a macro in the stdlib that can be emulated in an external library. I'm fairly sure every feature of go's embed proposal could be implemented via a rust macro outside the stdlib. For a specific example, I had a project where I wanted to serve the project's source code as a tarball (to comply with the AGPL license of the project). I was able to write a rust macro that made this as easy as effectively "SOURCE_TARBALL = include_repo!()" [0] to embed the output of 'git archive' in my binary. Of course, there's a very conscious tradeoff being made here. In rust, "cargo build" allows arbitrary execution of code for any dependency (trivially via build.rs), while in go, "go build" is meant to be a safe operation with minimal extensibility, side effects, or slowdowns. [0]: https://github.com/euank/include-repo https://github.com/euank/include-repo
    - zenhack 6y ago
      > Of course, there's a very conscious tradeoff being made here. In rust, "cargo build" allows arbitrary execution of code for any dependency (trivially via build.rs), while in go, "go build" is meant to be a safe operation with minimal extensibility, side effects, or slowdowns. I've been working off and on on a language that tries to get the best of both worlds to some extent. The whole language is built around making sandboxing code natural and composable. Like Rust, it has a macro system, so lots of compile time logic is possible without adding complexity to the build system, but macros don't have access to anything but the AST you give them, so they are safe to execute. There's a built in embed keyword that works like Rust's include_bytes, which runs before macro expansion, which you can use to feed the contents external files to macros for processing. At some point I'll probably add a variant that lets you pass whole directory trees.
- leetrout 6y ago
  Also note in this proposal: > The path separator is a forward slash, even on Windows systems. Which is much more ergonomic for developers than rusts treatment of os specific paths.
  - Arnavion 6y ago
    Forward slashes work on Windows systems for most [1] paths. You can write `include_bytes!("foo/bar.txt")` in Rust and be very confident it'll compile correctly for most of your users. [1]: "Most" because they don't work for namespaced paths, by design. But note that the number of times you'll encounter those is limited, the Rust standard library doesn't handle them correctly in general, and a lot of other non-Rust software breaks when given them.
    - leetrout 6y ago
      Oh, TIL! Well that is good to know- so it is only that windows backslashes fail on linux.
      - josefx 6y ago
        Windows path handling is a horrid mess. It basically takes your input and tries to turn it into a valid NT file path, that also includes handling special device names like COM1. At some point this was limited to 256 chars, so you had to input a valid NT path yourself if you wanted it longer, however I think Microsoft removed that limit a few years ago.
- justasitsounds 6y ago
  I know it's been mentioned before, but I really feel there is a need to codify an 'internet lore law' about how many comments can appear on a Go themed HN post before someone brings up how superior Rust is Something like: Pikes law of unfavourable comparison OR HN Golang oxidation rate
  - dodobirdlord 6y ago
    Oxidation Factor: Inversely proportional to the amount of time before a comment brings up the superiority of Rust. Go has a pretty high oxidation factor.
    - justasitsounds 6y ago
      Oxidation (noun): the act or process of shoehorning Rust into a comment thread
  - papaf 6y ago
    I suspect that, programming in Rust is so onerous, that Rust programmers must take lots of breaks and troll Go threads so they feel less bad about themselves. Or maybe they're just waiting for something to compile?
    - no_wizard 6y ago
      To be fair, I remember thinking the same thing when Go was new. HN was pretty dominated by everyone apparently switching to Go (if you were to believe the general attitude anyway) and how Go was the clear path in the future Not that it is or isn’t and clearly Go does things really well in some problem spaces but yeah, this seems to be a common theme for all young languages I do think some members of the Rust community Take it waaay too personally, which I never saw in general when Go was the hip thing. That I agree can be very obnoxious
      - pantulis 6y ago
        Same with Nodejs, and same with Ruby when the Rails craze. That's us devs.
axaxs 6y ago
Can someone explain please the point of embedding bin data into a binary, vs just bundling those assets and opening as a file at runtime?
- jhardy54 6y ago
  Convenience. It's easier and more foolproof to give someone one file than it is to give them a directory.
  - mickaelkerjean 6y ago
    Yes! I can't count how many people have tried to use my software (https://github.com/mickael-kerjean/filestash https://github.com/mickael-kerjean/filestash), flushing some entire directories with docker volume and creating support tickets and emails as things got broken. The landscape of tools supposed to solve that problem isn't great. Things are changing way too fast with projects being archived or deprecated with mention to migrate to another solution which after another year is deprecated again. Because of those issue, I've went with go generate which isn't ideal but is at least stable. That proposal would be a game changer and hopefully it will become a reality as Brad is quite a prominent figure in the Golang world.
    - throwaway29103 6y ago
      Sorry... what do you mean by "flushing" (thanks) BTW. Fantastic software, filestash - really really good.
      - jhardy54 6y ago
        My guess: deleting.
        throwaway29103 6y ago
        Thanks, first I thought... it cannot be, because when you delete the container, the named volumes are still there; but I guess that they might have deleted contents using the app itself, without realizing they were sharing the directory...
jhardy54 6y ago
Can you run one of these files as an executable?
- leetrout 6y ago
  You mean load the packed payload at runtime and execute it?
  - stblack 6y ago
    Interesting.
- zelly 6y ago
  It looks like this is just for static data files. To run a piece of memory as executable you'd have to explictly mark the page as executable with e.g. mprotect(2) on Linux. I guess there's nothing stopping you from doing that from Go also.
  - q3k 6y ago
    You still need an ELF loader, unless you're shipping shellcode. As far as I know, there's currently no great way to load an arbitrary segment of memory as an ELF, without doing some sort of copy (eg. memfd_create, write, exec).
leetrout 6y ago
That’s showing some real support to the community IMO. There are plenty of purists that argue both for and against the concept but one cannot deny it fits very well in the sales pitch for Go. You can have a simple tool chain to ship a binary for any number of systems and that support for embedded assets would be a first class citizen. I would also love to see them address more robust plugin options or officially adopt / endorse the pattern HashiCorp uses as they’re doing here with the bindata prior art.
- api 6y ago
  Go is not a purist language. It's pretty relentlessly designed for maximum productivity with minimum complexity. This sort of feature is right in line with its mission.
  - Filligree 6y ago
    I keep hearing that, but Go is one of the languages I'm least productive in, and I know I'm not alone in that. You can pick it up in a week, sure. That's not the same as productivity.
    - mhh__ 6y ago
      Apart from the, I feel, quite pretentious and lazy Rob Pike quote. The productivity in this case is probably referring to the initial period rather than the steady state afterwards (so to speak). Go is a fairly horrible language by today's standards, but it does what it's supposed to. I like generating code automatically at compile time based on, but I can appreciate the appeal of something that forces you to actually do work rather than being clever.
      - throwaway894345 6y ago
        There’s also a lot outside of the language—tooling, ecosystem, etc that Go absolutely nails; however, these things are tremendously undervalued. You have people in this thread talking about how awful Go is for using “//go:” instead of #pragma as though this sort of concern dominated the software development process.
lsllc 6y ago
I think it's a well thought out design, particularly the implementation of some common Go interfaces (such as fs.FS) that will allow this to be transparently used with existing Go code. The only thing I'd want is to allow environment variables in the "go:embed" statements, while my assets might be in the same repo and thus relative, they may also be in a different asset repo (if I'm using git I could use git submodule, but if I'm using something else that might not be possible).
- yiyus 6y ago
  The best way to do this with the current proposal is to write a simple go package in your assets repository and import it. Something like: package assets // import "github.com/me/myproject/assets" //go:embed * var FS embed.Files package mypackage import "github.com/me/myproject/assets" //use assets.FS Would that cover your use case?
  - lsllc 6y ago
    Yeah! (as long as the UI/UX folks don't mind a bit of Go code in their repo) I was kind of hoping for: package foo //go:embed $ASSETS/* var assets embed.Files But one issue with this is that it might make a dogs breakfast of the filenames within the `assets` object -- it sort of needs something like this: //go:embed * from $ASSETS (and it would not include the $ASSETS path in the object).
    - yiyus 6y ago
      An interesting option that I've not seen proposed yet would be to accept go:embed comments before importing a package (obviating the need to write the assets package in my previous example). So, assuming your assets are in the repository github.com/me/myproject/assets, you could just do: package foo //go:embed */* import "github.com/me/myproject/assets" And the go tool would take charge of generating an assets package with assets.Files. Then we don't even need a magic embed package at all. And generating a package would give us more flexibility, for example: package foo // import "foo.io/foo" //go:embed Name string "name.txt" //go:embed Data []byte "bin.dat" //go:embed "images" "templates" "html/index.html" import embedded "foo.io/foo" //use embedded.Name, with type string //use embedded.Data, with type []byte //use embedded.Files, with type fs.FS
atombender 6y ago
Video presentation: https://golang.org/s/draft-embed-video https://golang.org/s/draft-embed-video
- brown9-2 6y ago
  I hope this catches on as a trend, it’s really nice to see the process being so open to the community to give feedback.
ceocoder 6y ago
My former coworker Miki Tebeka wrote nrcs[0], this was specifically for static files - css, JS, png etc - for shipping a self contained web server binary. I still use it now and then because it is well designed and does one thing and that one thing really well. [0] https://github.com/tebeka/nrsc https://github.com/tebeka/nrsc
- joice 6y ago
  I am using stuffbin to embed files. github.com/knadh/stuffbin
banana_giraffe 6y ago
I'm happy to see this, I've always liked using embedded assets in executables to make single file distributions for tools. That said, I really don't like the further overloading of comments. Comments are for the human, not the compiler.
- kstenerud 6y ago
  It's a consequence of go's simplicity by design. You can't actually destroy natural complexity; only move it around. And if you're unwilling to design a proper container around that slowly increasing complexity, it just squeezes out between the cracks in odd ways. In go's case, it's manifest primarily through magic (magic files, directories, comments, names, env vars, switches, etc that you just have to know how to invoke). Another warning sign is the need for an external build tool like makefiles in order to build-in-one-command because you need extra steps beyond "go build" (such as "go generate").
- Cthulhu_ 6y ago
  I kinda agree; I get that they don't want to change the language definition itself, and by putting things in comments they don't need to do any change in the compiler, but it makes things feel unsafe, or "stringly typed" if you will. IIRC Java eventually "solved" it by adding annotations as a language feature. I'm not saying Go should add annotations, but just sharing some language history. And C / C++ has had compiler directives (e.g. #define) since forever, although there too (I believe) # is "just" a comment.
bww 6y ago
I’m aware there is demand for this sort of thing but I’d suggest that almost nobody actually needs it. This strikes me as a feature in search of a problem – especially considering how many of the examples in this document relate to static web assets. If you’re considering bundling static assets into your binary for a service, you would almost certainly be better served by containerizing your service and copying those assets into the image.
- justaguy88 6y ago
  Sometimes the point is to not use a container
- anonfunction 6y ago
  I disagree, one amazing aspect of Go is the static binary that can easily be distributed. I embed templates and other text files in my binaries and it makes the installation so simple and without extra dependencies or steps. Even though I'm a huge proponent of containers I don't think it is needed for things like CLI tools.
  - pjmlp 6y ago
    I was quite amazed doing static compilation with Turbo Basic, Turbo Pascal, Turbo C, Turbo C++,....
    - anonfunction 6y ago
      When I see all the CLI tools that require NPM / PIP to install hundreds of dependencies I am quite amazed when one requires a single binary. I'm not saying it's a unique feature with Go but that it is nonetheless a feature.
      - pjmlp 6y ago
        What amazes me is that installing hundreds of dependencies is acceptable, and that modern generations don't get that static compilation goes back to the first compiler, while dynamic compilation only went mainstream around mid-90's.
mjibson 6y ago
I wrote one of the listed tools (github.com/mjibson/esc) and am thrilled about this proposal. I think it's great and solves all the problems in a great way.
- stevekemp 6y ago
  I wrote an unlisted tool too, and I am also a fan of this proposal. The fact that there are so many of these "embed file in binary" tools suggests that it really is a problem that could be usefully solved once, in a consistent and reliable fashion.
- BillinghamJ 6y ago
  It does seem like it'd benefit from a "Must" method though - so you can use it easily to populate variables at init-time
marcrosoft 6y ago
No mention of go-bindata which was one of the originals :( I think this problem is best solved by a library. Leave the damn language alone!
- cespare 6y ago
  go-bindata is mentioned first, before the bullet list.
- zshev 6y ago
  It's right there > One of the earliest and most popular was github.com/jteeuwen/go-bindata and its forks
- coder543 6y ago
  > No mention of go-bindata which was one of the originals :( There’s literally an entire section in the appendix dedicated to `go-bindata` and explaining what it generates. It was also first in the list of libraries mentioned. I also disagree with you entirely. The document does a great job explaining the problems with having this “solved by a library”. I’m excited about this draft design.
- uluyol 6y ago
  This isn't a language change, it's a library + tooling one.
- mseepgood 6y ago
  > Leave the damn language alone! This proposal does not change the language. "//go:" comments are comments because they are not part of the language.
tmaly 6y ago
This would be huge. I have had to deal with 3rd party libraries to try to embed assets for a web server.
- marcrosoft 6y ago
  Which is the way it should be. Not everything needs to be in std lib.
  - donatj 6y ago
    While I agree that "not everything needs to be in std lib", having worked with other languages that support embedding out of the box, let me tell you it's handy. It also seems like an area where it'd be nice for there to be a single uniform way to do it.
    - Gibbon1 6y ago
      Of all things C is missing the ability to import binary assets. Which is kinda deranged when you think about it.
      - shakna 6y ago
        For runtime, you can certainly read any arbitrary binary file in C without problems. For compile time, I make heavy use of xxd -i to generate header files in a range of projects. It is no longer installed by default on most +nix systems, but it is generally still in your package manager (often installed alongside vim), and has been around since 1990.
        anitil 6y ago
        Oh I wish I'd known about 'xxd -i' ! When I saw it in the video linked above I was so annoyed that I hadn't used it before
      - pjmlp 6y ago
        ISO C surely, C based SDKs on macOS and Windows do support embedding resources since 16 bit days.
ocdtrekkie 6y ago
This is funny to me to see on HN tonight because I spent a bunch of time packaging a Go app on Sandstorm this evening, and one of the listed packages apparently was pitching a fit about some static assets not being pulled into the Sandstorm package correctly... I resolved it, checked HN, and saw this was a peeve they wanted to solve. Hah.
AnonC 6y ago
I love this idea of making executables contain assets, and it fits well with the Go ethos of a single (statically built) executable. This makes distribution simpler for applications that do need to deal with assets of different kinds. A tangential comment: is anyone else concerned a bit that discussions and debates happen on platforms like GitHub and reddit? These two platforms are large enough to be around for quite sometime, but are we making it easier to lose historical context because platform creators/designers are choosing third party platforms that they don’t host or control (or in some instances don’t or can’t pay for)?
- aspenmayer 6y ago
  It’s concerning that the debate that formed and forms open source is not itself available on open source platforms and/or archived and licensed and available for access in the same manner and license as the code itself. Good spot, quite apropos!
- Cthulhu_ 6y ago
  Yes, but at the same time, it'll reach a much bigger audience than mailing lists (which IIRC is still the official communications platform for Go), which are still painful to work with, even with using Google's own tooling. That said, mailing lists are much more open and easily archived for posterity, moreso than Reddit.
malandrew 6y ago
I would love if gob encoded files were supported natively with some special sugar syntax. I have a project I work on where I need to package a bunch of data that won’t be available locally. Given that I’m loading data, it seems most optimal to just package it as encoded go data from the get-go. Right now I’m using go-bindata to embed gob files.
breakingcups 6y ago
Again, more magic comments. The proposed feature is great, but the unwillingness of the Go team to use a separate, clearly defined project file or at the very least a separate syntax in your code file leads them to stuff every additional feature into comments, a space shared by human notetaking. Let's have a look: * Build constraints (// +build linux) * Code generation (//go:generate <command> <arguments>) * Cgo, you can even stuff entire C programs in the comments (// #include <stdio.h>) * Cgo flags (// #cgo CFLAGS: -DPNG_DEBUG=1) * and now this, file embedding (//go:embed html/index.html) Most novices would assume the commented out code does nothing, and rightly so in my opinion. Half of these features aren't even code-file specific but project-wide, making deciding which file to put them in hard and looking them up even harder.
- gwd 6y ago
  You forgot documentation (which actually annoys me more, because I can't put comments to developers of the function right before the function, because it will be interpreted as documentation to users of the function). I mean yeah, I think the comment thing is a bit wonky, and it's not the way I'd do it. But once you've got "//go:generate", adding "//go:embed" isn't really any worse. EDIT: Just noticed this line in the proposal: > Another explicit goal is to avoid a language change. To us, embedding static assets seems like a tooling issue, not a language issue. Avoiding a language change also means we avoid the need to update the many tools that process Go code, among them goimports, gopls, and staticcheck. So this partly explains the idea of putting things in comments: things classified as "tooling issues" are put there so that "language tools" aren't affected. I do agree it probably would have been better to invent a specific language construct for, "This is a tooling issue, please ignore"; a bit like the # prefix in C.
  - masklinn 6y ago
    > But once you've got "//go:generate", adding "//go:embed" isn't really any worse. There's way more than that: https://golang.org/src/cmd/compile/internal/gc/lex.go#L53 https://golang.org/src/cmd/compile/internal/gc/lex.go#L53 And I would argue that it's bull. Once they realised that they were introducing pragmas / attributes into the language, they should have bitten the bullet and actually introduced a clean way to annotate language items. The first and second ones are excusable as "we'll only need one" and "well it's not worth the hassle" but at the third one it's not a special case it's a pattern. Especially as the excuse that "other implementations can ignore those" gets less and less true: a compiler which ignores go:embed can not be considered working.
skocznymroczny 6y ago
D has the import statement for that: https://dlang.org/spec/expression.html#import_expressions https://dlang.org/spec/expression.html#import_expressions
jedisct1 6y ago
Something I'd love to see in Zig.
lenkite 6y ago
Go needs proper typed annotations like Java
- pjmlp 6y ago
  Go started being a Java 1.0, instead of learning what a modern language should be like, so every missing feature gets eventually added with a couple of hacks. Go 15 is going to be a pleasure to use.....
  - jjice 6y ago
    Is 15 the version that generics are going to be implemented in? Or are generics still in the proposal stage?
    - pjmlp 6y ago
      Hopefully by then they should be available, we will be discussing how they are unsound and how it was a mistake to add them in like that, while everyone will be discussing this new language that is going to be so much better and fix all the problems.
AlexMax 6y ago
Regardless of the bike-shedding over the exact syntax, this would be really cool to see. I have no clue how C/C++ made it into 2020 without this language feature - and yes, I know about incbin and xxd, and Go having it ahead of those languages would be ironic given how new and conservative of a language it is.

211 points by eberkund 6y ago

Seems pretty complicated compared to Rust's `include_bytes!("path/to/file")`. Sounds like it might have a compile time advantage though, seeing as it produces a seperate package.

opqpo 6y ago
Everything in Golang seems pretty complicated and nonintuitive compared to any other modern language. But "it's all well thought out"!
- jimktrains2 6y ago
  Every nil panic and manual type assertion holds a potentially very nice language back. They just seem like relics we should abandon.
  - opqpo 6y ago
    Accessing nil pointers is well thought out. Not enforcing error checking is well thought out. Reflection and code generators instead of generics and traits is well thought out. Lack of conditional compilation is well thought out. No compilation with unused variables instead of warnings is well thought out. Doing your own sort, map, all essential array operations from scratch everytime is well thought out. Spamming err != nil instead of some operator like `?` is well thought out. iota instead of type-safe Enums is well thought out. No file-scope variables is well thought out. go mod is well thought out compared to npm and cargo.
    - leetrout 6y ago
      Go doesn’t have to be bad for all those things you like in Scala/Rust/etc to be true. It’s a simple language with simple, explicit, verbose, tedious, pedantic patterns. The communication resulting from those patterns is where the value lies for a lot of developers and teams.
      - opqpo 6y ago
        Mediocrity, ugliness and verbosity is more well thought out than engineering.
        leetrout 6y ago
        Sure - I know you disagree - I was just pointing out a Ferrari and a school bus can both exist and be good for different purposes.
        opqpo 6y ago
        disagreeing is a pretty strong word. Nobody can disagree these days. I was flagged and labeled as a troll within a couple of minutes once I criticized a programming language. 30 years from now, people will remember this pathetic mob tyranny era when one couldn't even offend a programming language feelings!
      - hu3 6y ago
        If you allow me an advice, I'd rather not waste precious time with trolls. That user has multiple flagged messages in this month alone.
      - jimktrains2 6y ago
        Before using it I thought the if err != Nil would be annoying, but it actually isn't. I think a sum type would be better, but the pattern is common and honestly not very different from a match on a sum type. In general I agree that it does lead to code that's more familiar even if it was written by someone else. Having only used it a few months now, I can definitely see where the value in go lies.
Thaxll 6y ago
It seems much more powerful than the Rust version. The Rust version basicaly embed the file into a byte array that's it or I'm missing something?
- nightfly 6y ago
  If you need more check out the rust-embed crate.
- euank 6y ago
  This proposal is more powerful than that one rust macro, but rust's abilities around embedding files are much more powerful than go's approach. This proposal allows "go build" to embed things in a very specific way, but it's not meant to be extensible. Rust's 'include_bytes!' macro on the other hand is a macro in the stdlib that can be emulated in an external library. I'm fairly sure every feature of go's embed proposal could be implemented via a rust macro outside the stdlib. For a specific example, I had a project where I wanted to serve the project's source code as a tarball (to comply with the AGPL license of the project). I was able to write a rust macro that made this as easy as effectively "SOURCE_TARBALL = include_repo!()" [0] to embed the output of 'git archive' in my binary. Of course, there's a very conscious tradeoff being made here. In rust, "cargo build" allows arbitrary execution of code for any dependency (trivially via build.rs), while in go, "go build" is meant to be a safe operation with minimal extensibility, side effects, or slowdowns. [0]: https://github.com/euank/include-repo https://github.com/euank/include-repo
  - zenhack 6y ago
    > Of course, there's a very conscious tradeoff being made here. In rust, "cargo build" allows arbitrary execution of code for any dependency (trivially via build.rs), while in go, "go build" is meant to be a safe operation with minimal extensibility, side effects, or slowdowns. I've been working off and on on a language that tries to get the best of both worlds to some extent. The whole language is built around making sandboxing code natural and composable. Like Rust, it has a macro system, so lots of compile time logic is possible without adding complexity to the build system, but macros don't have access to anything but the AST you give them, so they are safe to execute. There's a built in embed keyword that works like Rust's include_bytes, which runs before macro expansion, which you can use to feed the contents external files to macros for processing. At some point I'll probably add a variant that lets you pass whole directory trees.
leetrout 6y ago
Also note in this proposal: > The path separator is a forward slash, even on Windows systems. Which is much more ergonomic for developers than rusts treatment of os specific paths.
- Arnavion 6y ago
  Forward slashes work on Windows systems for most [1] paths. You can write `include_bytes!("foo/bar.txt")` in Rust and be very confident it'll compile correctly for most of your users. [1]: "Most" because they don't work for namespaced paths, by design. But note that the number of times you'll encounter those is limited, the Rust standard library doesn't handle them correctly in general, and a lot of other non-Rust software breaks when given them.
  - leetrout 6y ago
    Oh, TIL! Well that is good to know- so it is only that windows backslashes fail on linux.
    - josefx 6y ago
      Windows path handling is a horrid mess. It basically takes your input and tries to turn it into a valid NT file path, that also includes handling special device names like COM1. At some point this was limited to 256 chars, so you had to input a valid NT path yourself if you wanted it longer, however I think Microsoft removed that limit a few years ago.
justasitsounds 6y ago
I know it's been mentioned before, but I really feel there is a need to codify an 'internet lore law' about how many comments can appear on a Go themed HN post before someone brings up how superior Rust is Something like: Pikes law of unfavourable comparison OR HN Golang oxidation rate
- dodobirdlord 6y ago
  Oxidation Factor: Inversely proportional to the amount of time before a comment brings up the superiority of Rust. Go has a pretty high oxidation factor.
  - justasitsounds 6y ago
    Oxidation (noun): the act or process of shoehorning Rust into a comment thread
- papaf 6y ago
  I suspect that, programming in Rust is so onerous, that Rust programmers must take lots of breaks and troll Go threads so they feel less bad about themselves. Or maybe they're just waiting for something to compile?
  - no_wizard 6y ago
    To be fair, I remember thinking the same thing when Go was new. HN was pretty dominated by everyone apparently switching to Go (if you were to believe the general attitude anyway) and how Go was the clear path in the future Not that it is or isn’t and clearly Go does things really well in some problem spaces but yeah, this seems to be a common theme for all young languages I do think some members of the Rust community Take it waaay too personally, which I never saw in general when Go was the hip thing. That I agree can be very obnoxious
    - pantulis 6y ago
      Same with Nodejs, and same with Ruby when the Rails craze. That's us devs.

axaxs 6y ago

Can someone explain please the point of embedding bin data into a binary, vs just bundling those assets and opening as a file at runtime?

jhardy54 6y ago
Convenience. It's easier and more foolproof to give someone one file than it is to give them a directory.
- mickaelkerjean 6y ago
  Yes! I can't count how many people have tried to use my software (https://github.com/mickael-kerjean/filestash https://github.com/mickael-kerjean/filestash), flushing some entire directories with docker volume and creating support tickets and emails as things got broken. The landscape of tools supposed to solve that problem isn't great. Things are changing way too fast with projects being archived or deprecated with mention to migrate to another solution which after another year is deprecated again. Because of those issue, I've went with go generate which isn't ideal but is at least stable. That proposal would be a game changer and hopefully it will become a reality as Brad is quite a prominent figure in the Golang world.
  - throwaway29103 6y ago
    Sorry... what do you mean by "flushing" (thanks) BTW. Fantastic software, filestash - really really good.
    - jhardy54 6y ago
      My guess: deleting.
      - throwaway29103 6y ago
        Thanks, first I thought... it cannot be, because when you delete the container, the named volumes are still there; but I guess that they might have deleted contents using the app itself, without realizing they were sharing the directory...

jhardy54 6y ago

Can you run one of these files as an executable?

leetrout 6y ago
You mean load the packed payload at runtime and execute it?
- stblack 6y ago
  Interesting.
zelly 6y ago
It looks like this is just for static data files. To run a piece of memory as executable you'd have to explictly mark the page as executable with e.g. mprotect(2) on Linux. I guess there's nothing stopping you from doing that from Go also.
- q3k 6y ago
  You still need an ELF loader, unless you're shipping shellcode. As far as I know, there's currently no great way to load an arbitrary segment of memory as an ELF, without doing some sort of copy (eg. memfd_create, write, exec).

leetrout 6y ago

That’s showing some real support to the community IMO. There are plenty of purists that argue both for and against the concept but one cannot deny it fits very well in the sales pitch for Go. You can have a simple tool chain to ship a binary for any number of systems and that support for embedded assets would be a first class citizen. I would also love to see them address more robust plugin options or officially adopt / endorse the pattern HashiCorp uses as they’re doing here with the bindata prior art.

api 6y ago
Go is not a purist language. It's pretty relentlessly designed for maximum productivity with minimum complexity. This sort of feature is right in line with its mission.
- Filligree 6y ago
  I keep hearing that, but Go is one of the languages I'm least productive in, and I know I'm not alone in that. You can pick it up in a week, sure. That's not the same as productivity.
  - mhh__ 6y ago
    Apart from the, I feel, quite pretentious and lazy Rob Pike quote. The productivity in this case is probably referring to the initial period rather than the steady state afterwards (so to speak). Go is a fairly horrible language by today's standards, but it does what it's supposed to. I like generating code automatically at compile time based on, but I can appreciate the appeal of something that forces you to actually do work rather than being clever.
    - throwaway894345 6y ago
      There’s also a lot outside of the language—tooling, ecosystem, etc that Go absolutely nails; however, these things are tremendously undervalued. You have people in this thread talking about how awful Go is for using “//go:” instead of #pragma as though this sort of concern dominated the software development process.

lsllc 6y ago

I think it's a well thought out design, particularly the implementation of some common Go interfaces (such as fs.FS) that will allow this to be transparently used with existing Go code. The only thing I'd want is to allow environment variables in the "go:embed" statements, while my assets might be in the same repo and thus relative, they may also be in a different asset repo (if I'm using git I could use git submodule, but if I'm using something else that might not be possible).

yiyus 6y ago
The best way to do this with the current proposal is to write a simple go package in your assets repository and import it. Something like: package assets // import "github.com/me/myproject/assets" //go:embed * var FS embed.Files package mypackage import "github.com/me/myproject/assets" //use assets.FS Would that cover your use case?
- lsllc 6y ago
  Yeah! (as long as the UI/UX folks don't mind a bit of Go code in their repo) I was kind of hoping for: package foo //go:embed $ASSETS/* var assets embed.Files But one issue with this is that it might make a dogs breakfast of the filenames within the `assets` object -- it sort of needs something like this: //go:embed * from $ASSETS (and it would not include the $ASSETS path in the object).
  - yiyus 6y ago
    An interesting option that I've not seen proposed yet would be to accept go:embed comments before importing a package (obviating the need to write the assets package in my previous example). So, assuming your assets are in the repository github.com/me/myproject/assets, you could just do: package foo //go:embed */* import "github.com/me/myproject/assets" And the go tool would take charge of generating an assets package with assets.Files. Then we don't even need a magic embed package at all. And generating a package would give us more flexibility, for example: package foo // import "foo.io/foo" //go:embed Name string "name.txt" //go:embed Data []byte "bin.dat" //go:embed "images" "templates" "html/index.html" import embedded "foo.io/foo" //use embedded.Name, with type string //use embedded.Data, with type []byte //use embedded.Files, with type fs.FS

atombender 6y ago

Video presentation: https://golang.org/s/draft-embed-video https://golang.org/s/draft-embed-video

brown9-2 6y ago
I hope this catches on as a trend, it’s really nice to see the process being so open to the community to give feedback.

ceocoder 6y ago

My former coworker Miki Tebeka wrote nrcs[0], this was specifically for static files - css, JS, png etc - for shipping a self contained web server binary. I still use it now and then because it is well designed and does one thing and that one thing really well. [0] https://github.com/tebeka/nrsc https://github.com/tebeka/nrsc

joice 6y ago
I am using stuffbin to embed files. github.com/knadh/stuffbin

banana_giraffe 6y ago

I'm happy to see this, I've always liked using embedded assets in executables to make single file distributions for tools. That said, I really don't like the further overloading of comments. Comments are for the human, not the compiler.

kstenerud 6y ago
It's a consequence of go's simplicity by design. You can't actually destroy natural complexity; only move it around. And if you're unwilling to design a proper container around that slowly increasing complexity, it just squeezes out between the cracks in odd ways. In go's case, it's manifest primarily through magic (magic files, directories, comments, names, env vars, switches, etc that you just have to know how to invoke). Another warning sign is the need for an external build tool like makefiles in order to build-in-one-command because you need extra steps beyond "go build" (such as "go generate").
Cthulhu_ 6y ago
I kinda agree; I get that they don't want to change the language definition itself, and by putting things in comments they don't need to do any change in the compiler, but it makes things feel unsafe, or "stringly typed" if you will. IIRC Java eventually "solved" it by adding annotations as a language feature. I'm not saying Go should add annotations, but just sharing some language history. And C / C++ has had compiler directives (e.g. #define) since forever, although there too (I believe) # is "just" a comment.

bww 6y ago

I’m aware there is demand for this sort of thing but I’d suggest that almost nobody actually needs it. This strikes me as a feature in search of a problem – especially considering how many of the examples in this document relate to static web assets. If you’re considering bundling static assets into your binary for a service, you would almost certainly be better served by containerizing your service and copying those assets into the image.

justaguy88 6y ago
Sometimes the point is to not use a container
anonfunction 6y ago
I disagree, one amazing aspect of Go is the static binary that can easily be distributed. I embed templates and other text files in my binaries and it makes the installation so simple and without extra dependencies or steps. Even though I'm a huge proponent of containers I don't think it is needed for things like CLI tools.
- pjmlp 6y ago
  I was quite amazed doing static compilation with Turbo Basic, Turbo Pascal, Turbo C, Turbo C++,....
  - anonfunction 6y ago
    When I see all the CLI tools that require NPM / PIP to install hundreds of dependencies I am quite amazed when one requires a single binary. I'm not saying it's a unique feature with Go but that it is nonetheless a feature.
    - pjmlp 6y ago
      What amazes me is that installing hundreds of dependencies is acceptable, and that modern generations don't get that static compilation goes back to the first compiler, while dynamic compilation only went mainstream around mid-90's.

mjibson 6y ago

I wrote one of the listed tools (github.com/mjibson/esc) and am thrilled about this proposal. I think it's great and solves all the problems in a great way.

stevekemp 6y ago
I wrote an unlisted tool too, and I am also a fan of this proposal. The fact that there are so many of these "embed file in binary" tools suggests that it really is a problem that could be usefully solved once, in a consistent and reliable fashion.
BillinghamJ 6y ago
It does seem like it'd benefit from a "Must" method though - so you can use it easily to populate variables at init-time

marcrosoft 6y ago

No mention of go-bindata which was one of the originals :( I think this problem is best solved by a library. Leave the damn language alone!

cespare 6y ago
go-bindata is mentioned first, before the bullet list.
zshev 6y ago
It's right there > One of the earliest and most popular was github.com/jteeuwen/go-bindata and its forks
coder543 6y ago
> No mention of go-bindata which was one of the originals :( There’s literally an entire section in the appendix dedicated to `go-bindata` and explaining what it generates. It was also first in the list of libraries mentioned. I also disagree with you entirely. The document does a great job explaining the problems with having this “solved by a library”. I’m excited about this draft design.
uluyol 6y ago
This isn't a language change, it's a library + tooling one.
mseepgood 6y ago
> Leave the damn language alone! This proposal does not change the language. "//go:" comments are comments because they are not part of the language.

tmaly 6y ago

This would be huge. I have had to deal with 3rd party libraries to try to embed assets for a web server.

marcrosoft 6y ago
Which is the way it should be. Not everything needs to be in std lib.
- donatj 6y ago
  While I agree that "not everything needs to be in std lib", having worked with other languages that support embedding out of the box, let me tell you it's handy. It also seems like an area where it'd be nice for there to be a single uniform way to do it.
  - Gibbon1 6y ago
    Of all things C is missing the ability to import binary assets. Which is kinda deranged when you think about it.
    - shakna 6y ago
      For runtime, you can certainly read any arbitrary binary file in C without problems. For compile time, I make heavy use of xxd -i to generate header files in a range of projects. It is no longer installed by default on most +nix systems, but it is generally still in your package manager (often installed alongside vim), and has been around since 1990.
      - anitil 6y ago
        Oh I wish I'd known about 'xxd -i' ! When I saw it in the video linked above I was so annoyed that I hadn't used it before
    - pjmlp 6y ago
      ISO C surely, C based SDKs on macOS and Windows do support embedding resources since 16 bit days.

ocdtrekkie 6y ago

This is funny to me to see on HN tonight because I spent a bunch of time packaging a Go app on Sandstorm this evening, and one of the listed packages apparently was pitching a fit about some static assets not being pulled into the Sandstorm package correctly... I resolved it, checked HN, and saw this was a peeve they wanted to solve. Hah.

AnonC 6y ago

I love this idea of making executables contain assets, and it fits well with the Go ethos of a single (statically built) executable. This makes distribution simpler for applications that do need to deal with assets of different kinds. A tangential comment: is anyone else concerned a bit that discussions and debates happen on platforms like GitHub and reddit? These two platforms are large enough to be around for quite sometime, but are we making it easier to lose historical context because platform creators/designers are choosing third party platforms that they don’t host or control (or in some instances don’t or can’t pay for)?

aspenmayer 6y ago
It’s concerning that the debate that formed and forms open source is not itself available on open source platforms and/or archived and licensed and available for access in the same manner and license as the code itself. Good spot, quite apropos!
Cthulhu_ 6y ago
Yes, but at the same time, it'll reach a much bigger audience than mailing lists (which IIRC is still the official communications platform for Go), which are still painful to work with, even with using Google's own tooling. That said, mailing lists are much more open and easily archived for posterity, moreso than Reddit.

malandrew 6y ago

I would love if gob encoded files were supported natively with some special sugar syntax. I have a project I work on where I need to package a bunch of data that won’t be available locally. Given that I’m loading data, it seems most optimal to just package it as encoded go data from the get-go. Right now I’m using go-bindata to embed gob files.

breakingcups 6y ago

Again, more magic comments. The proposed feature is great, but the unwillingness of the Go team to use a separate, clearly defined project file or at the very least a separate syntax in your code file leads them to stuff every additional feature into comments, a space shared by human notetaking. Let's have a look: * Build constraints (// +build linux) * Code generation (//go:generate <command> <arguments>) * Cgo, you can even stuff entire C programs in the comments (// #include <stdio.h>) * Cgo flags (// #cgo CFLAGS: -DPNG_DEBUG=1) * and now this, file embedding (//go:embed html/index.html) Most novices would assume the commented out code does nothing, and rightly so in my opinion. Half of these features aren't even code-file specific but project-wide, making deciding which file to put them in hard and looking them up even harder.

gwd 6y ago
You forgot documentation (which actually annoys me more, because I can't put comments to developers of the function right before the function, because it will be interpreted as documentation to users of the function). I mean yeah, I think the comment thing is a bit wonky, and it's not the way I'd do it. But once you've got "//go:generate", adding "//go:embed" isn't really any worse. EDIT: Just noticed this line in the proposal: > Another explicit goal is to avoid a language change. To us, embedding static assets seems like a tooling issue, not a language issue. Avoiding a language change also means we avoid the need to update the many tools that process Go code, among them goimports, gopls, and staticcheck. So this partly explains the idea of putting things in comments: things classified as "tooling issues" are put there so that "language tools" aren't affected. I do agree it probably would have been better to invent a specific language construct for, "This is a tooling issue, please ignore"; a bit like the # prefix in C.
- masklinn 6y ago
  > But once you've got "//go:generate", adding "//go:embed" isn't really any worse. There's way more than that: https://golang.org/src/cmd/compile/internal/gc/lex.go#L53 https://golang.org/src/cmd/compile/internal/gc/lex.go#L53 And I would argue that it's bull. Once they realised that they were introducing pragmas / attributes into the language, they should have bitten the bullet and actually introduced a clean way to annotate language items. The first and second ones are excusable as "we'll only need one" and "well it's not worth the hassle" but at the third one it's not a special case it's a pattern. Especially as the excuse that "other implementations can ignore those" gets less and less true: a compiler which ignores go:embed can not be considered working.

skocznymroczny 6y ago

D has the import statement for that: https://dlang.org/spec/expression.html#import_expressions https://dlang.org/spec/expression.html#import_expressions

jedisct1 6y ago

Something I'd love to see in Zig.

lenkite 6y ago

Go needs proper typed annotations like Java

pjmlp 6y ago
Go started being a Java 1.0, instead of learning what a modern language should be like, so every missing feature gets eventually added with a couple of hacks. Go 15 is going to be a pleasure to use.....
- jjice 6y ago
  Is 15 the version that generics are going to be implemented in? Or are generics still in the proposal stage?
  - pjmlp 6y ago
    Hopefully by then they should be available, we will be discussing how they are unsound and how it was a mistake to add them in like that, while everyone will be discussing this new language that is going to be so much better and fix all the problems.

AlexMax 6y ago

Regardless of the bike-shedding over the exact syntax, this would be really cool to see. I have no clue how C/C++ made it into 2020 without this language feature - and yes, I know about incbin and xxd, and Go having it ahead of those languages would be ironic given how new and conservative of a language it is.