14 ms·
Tell HN: People forget that you can stick any data at the end of a bash script
This is a neat trick I've used to write self-extracting software or scripts that extract files from archives by just using
tail -c <number of bytes for the binary> $0
All you have to do is make sure you append an explicit 'exit' to the end of your program before your new 'data section', so that bash won't parse any of the 'data section'.
One thing to bear in mind is that if you append binary data, it will be corrupted if you save it in most text editors so when I want to make changes I just delete all the binary and reappend it.
- Gabrys1 3y ago"$0" otherwise it won't work for paths with spaces
- jjgreen 3y agoRuby (and earlier, Perl) formalised this with the __END__ section: https://www.honeybadger.io/blog/data-and-end-in-ruby/ https://www.honeybadger.io/blog/data-and-end-in-ruby/
- _rwo 3y agoWhile applying for new job once I made a self-unpacking CV in ruby using this trick; whole binary content was compressed with zlib, I also added some blank padding to get some 'nice' numbers, so I could stuff like DATA.seek(1337 * 32, IO::SEEK_CUR); needless to say either nobody appreciated the idea, then again, what I was expecting? :D
- x86x87 3y agoyup. after that you can use the global var DATA to access the data injected after the __END__
- jjgreen 3y agoWhen I was first learning Ruby knowing Perl rather well, it was discovering that it supported DATA/__END__ that made me feel like I was at home ...
- francislavoie 3y agoPHP also has __halt_compiler() https://www.php.net/manual/en/function.halt-compiler.php https://www.php.net/manual/en/function.halt-compiler.php
- brasic 3y agoAre you sure that Perl took it from ruby and not the other way around? (edit: a subsequent correction has obsoleted this comment)
- vram22 3y agoThey said "earlier, Perl".
- deleted 3y ago[deleted]
- throwawaaarrgh 3y agoBack in the day I made a Perl script that would use an inline encryption algorithm to decode a payload and execute it in memory so it would never hit the disk. Data after the __DATA__ line.
- cocodill 3y agoI can vaguely remember that many programs used to install themselves this way under Linux.
- nerdponx 3y agoI've seen it recently with the Conda and Mamba package managers.
- teddyh 3y agoIt was used on Unix systems even before that.
- dekhn 3y agodefinitely used something similar on VAX/VMS called VMS_SHARE (https://www.glaver.org/ftp/multinet-contributed-software/vms_share/vms_share.user_guide https://www.glaver.org/ftp/multinet-contributed-software/vms...) circa '90-91 in fact I found an old archive of mine floating around on usenet and wrote a python script to unpack it. Looking at the original, it was using a scripting. language bootstrap to make a COM script unpack embedded the original code.
- a2tech 3y agoLots of commercial Linux software use this still for installing their stuff. It’s a neat trick
- deleted 3y ago[deleted]
- not2b 3y agoThat's what uuencode / uudecode were once used for.
- eadler 3y agoSee https://man.freebsd.org/cgi/man.cgi?query=shar&sektion=1&format=html https://man.freebsd.org/cgi/man.cgi?query=shar&sektion=1&for... for a tool to generate these types of archives.
- teddyh 3y agoAlso on GNU/Linux systems: <https://manpages.debian.org/stable/sharutils/shar.1.en.html https://manpages.debian.org/stable/sharutils/shar.1.en.html>
- nottorp 3y agoShell archive it was called? There used to be a lot of installers like that.
- NovemberWhiskey 3y agoYup; "shar" https://en.wikipedia.org/wiki/Shar https://en.wikipedia.org/wiki/Shar
- gattilorenz 3y agoExactly. Very popular way of distributing software e.g. on alt.sources, back in the day.
- linsomniac 3y agoMinor nit: every "shar" I've seen (from distant memory) used a "here document" rather than appending (possibly binary) data to the end of a shell script. https://en.wikipedia.org/wiki/Here_document https://en.wikipedia.org/wiki/Here_document
- nottorp 3y agoHmm i have a yocto generated sdk installer handy and it seems to do just this: payload_offset=$(($(grep -na -m1 "^MARKER:$" "$0"|cut -d':' -f1) + 1)) and at the end of the code it just has: exit 0 MARKER:
- kkfx 3y agoMakeself archives are a classic self-extracting tarball who do exactly that...
- xg15 3y agoIf you care less about space efficiency and more about maintainability of the script, you can also encode the binary as base64 and put an echo '...base64 data...' | base64 -d > somefile in your script. Or add compression to reclaim at least some of the wasted space: echo '...base64 gzipped data...' | base64 -d | gunzip > somefile Also note that bash accepts line breaks in quoted strings and the base64 utility has an "ignore garbage" option that lets it skip over e.g. whitespace in its input. You can use those to break up the base64 over multiple lines: echo ' ...base64 gzipped data... ...more data... ...even more data... ' | base64 -di | gunzip > somefile
- dheera 3y agoIs there an encoding that is less wasteful that base64 but not vulnerable to text editor corruption issues? I think avoiding 0x0 to 0x20 should be enough to not get corrupted by text editors, though base64 avoids a lot more than that.
- ElectricalUnion 3y agoIf you can count on every printable ascii character being not-mangled, you can use ascii85/base85/Z85 (5 "ascii characters" to 4 bytes) instead of base64.
- raverbashing 3y agoThere's probably a base(bigger number) with Unicode chars today
- bashinator 3y agobase65536, and look who the author is :-D https://github.com/qntm/base65536 https://github.com/qntm/base65536
- cassianoleal 3y agoWho is the author?
- deleted 3y ago[deleted]
- dietrichepp 3y agoThis trick is used in the demoscene. Instead of using -c, I use -n, tail -n +2 $0 The -n +2 option means “starting at line 2”, which is what you want if you cram your script into one line. You can make an executable packed with lzma this way, a=`mktemp`;tail -n+2 $0|unxz>$a;chmod +x $a;$a;rm $a;exit This is the polite way to do it, using mktemp. You can save some bytes if you don’t care about that stuff.
- londons_explore 3y agoThere must be a way to run something without needing a temp file...
- BasedAnon 3y agoBasically everything in Linux will create a temp file one way or another even pipes. For you to take a binary and run it directly it has to have an inode. At best you can use Python & the ctypes module to write a program into part of Python's memory and trick it into continuing execution from there.
- dietrichepp 3y agoPipes are not temp files, they’re more like kernel buffers associated with no file. When I think of temp file, I think of something that is at least associated with a filesystem. The reason that running a binary needs a file is because execve() takes a path as an argument. But, as you said, there are other ways to load code into memory.
- Denvercoder9 3y ago> Basically everything in Linux will create a temp file one way or another even pipes. pipe(2) doesn't create a file, at least not in the sense that I usually think about files (something that's accessible through the filesystem).
- dietrichepp 3y agoYes. You can load the decompressed code into memory, mprotect() it to give it execution permissions, then jump in. Hard to do in a shell, though.
- INTPenis 3y agoThat's how I made a bash backdoor once. It was just a script somewhere on the FS, until it unpacked itself and executed the rest of the rootkit. Long story but trust me that I had good intentions.
- vram22 3y agoBASIC and Perl had or have something like that too. IIRC, Perl copied it from BASIC, because BASIC came much before Perl. And, again, IIRC, I've read about the shar (shell archive) method that someone else commented about in this thread (and which even has a Wikipedia entry), in either the classic Kernighan and Pike book, The Unix Programming Environment (which I've recommended here multiple times before), or in some Unix man pages, long ago. So it's quite an old method.
- JohnFen 3y agoThis is my default approach to writing installers for the Unices. The program is compressed and added to the end of the script, and the script does the unpacking and any needed setup/configuration for the specific platform it's getting installed on. I don't append it in binary form, though. I uuencode it. That way, there is no danger in using text editors.
- themerone 3y agoWhy uuencode? Base64 is the defacto standard these days.
- vram22 3y agoI've used both, but only briefly. I think I used uuencode when using uucp. And Base64 in one of my Python programs. What are their pros and cons, in your opinion?
- JohnFen 3y agoSorry, I did mean base64. I have a bad habit of calling all "binary as text" encodings "uuencode". I usually catch myself before I put it in writing, though.
- onion2k 3y agoThis reminds me of ZX Spectrum Basic where all the graphics, sound, and level layouts were defined using DATA lines at the end of the program.
- allarm 3y agoYou could also put the binary data in the first line of the Basic program after the ‘rem’ command, change the line number to 0 using the poke command, so that it’s not possible to edit this line. The second line would run the code using ‘randomize usr’. There were also fun tricks with control sequences, that would hide the ‘rem’ command and the line number, and put something like “Cracked by Bill Gilbert (c) 1982” instead. Gosh, why I still remember all this nonsense after all these years…
- antod 3y agoOr any machine code routines you wanted to POKE into memory. A suppressed obscure part of my lizard brain secretly wishes I could just code for 8bit computers from the 80s, just with all the modern niceties like text editors, assemblers and emulators etc.
- kamma4434 3y agoMe too!…
- davidw 3y agoI seem to recall that you can do the opposite as well: stash some extra data at the end of a binary file. The 'tclkit' system used this to package up an executable with the scripts you wanted to ship.
- giantrobot 3y agoThe Löve game development framework packages games like this. The games are nominally Alia scripts but if you cat a zip file to the end of the Lua script the framework lets you access assets in that zip as if they were in the game's cwd. It's a cool abstraction, during development you can have assets just living on the file system and then "deploy" a flat file that accesses those assets the same way.
- 2OEH8eoCRo0 3y agoI think this is how GOG ships the Linux version of Battletech.
- aquova 3y agoI believe this is how GOG ships all of its Linux titles, all of the installs I've used from them are downloaded as a single *.sh file. I just checked an example game, and it looks to be using this method.
- twic 3y agoSince zip files use a directory at the end, you can make a kind of mullet file - script at the front, archive at the back. I generated single-file runnable Java binaries like that at once point.
- ndsipa_pomu 3y ago> mullet file That's a great expression
- BonoboIO 3y agoIn German that would be VokuHila: Vorne Kurz, Hinten Lang Front Short, Back Long https://de.wikipedia.org/wiki/Vokuhila https://de.wikipedia.org/wiki/Vokuhila
- twic 3y agoI note that in Danish it is called "Svenskerhår" [1], which i think means "Swedish hair", and in Polish, "Czeski piłkarz" [2], or "Czech footballer"! Meanwhile in Swedish, it is "hockeyfrilla", the "Hockey frill". [1] https://da.wikipedia.org/wiki/Frisure https://da.wikipedia.org/wiki/Frisure [2] https://pl.wikipedia.org/wiki/Czeski_pi%C5%82karz https://pl.wikipedia.org/wiki/Czeski_pi%C5%82karz [3] https://sv.wikipedia.org/wiki/Hockeyfrilla https://sv.wikipedia.org/wiki/Hockeyfrilla
- BonoboIO 3y agoCzech Footballer HAHAHA Amazing
- n8henrie 3y agoSimilar trick to bundle a python script with self-contained dependencies (with some limitations, and doesn't include python itself): https://n8henrie.com/2022/08/easily-create-almost-standalone-python-executables-with-the-builtin-zipapp-module/ https://n8henrie.com/2022/08/easily-create-almost-standalone...
- ShowalkKama 3y agoportswigger does that for the burpsuite installers. https://portswigger-cdn.net/burp/releases/download?product=community&version=2023.6.2&type=Linux https://portswigger-cdn.net/burp/releases/download?product=c...
- vram22 3y ago>portswigger does that for the burpsuite installers. Wow, that triggered my wordplay radar, which I'm working on as a fun side line these days, thanks :) port, suite (sweet) swig, burp Heh.
- VMGunKelly 3y ago[flagged]
- sumosudo 3y agoI use a fun little hack, a la awk: ``` #!/usr/local/bin/bash echo "HELLO" TAIL_REMOTE_MARKER=`awk '/^__THE_REMOTE_PART__/{flag=1;next}/^__END_THE_REMOTE_PART__/{flag=0;exit}flag' ${0}` eval "$TAIL_REMOTE_MARKER" exit 0 __THE_REMOTE_PART__ echo "WORLD" __END_THE_REMOTE_PART__ ```
- heresie-dabord 3y agoIn Perl, __DATA__ indicates the beginning of the data section of the file. A portable way to provide test data or sample data. https://perldoc.perl.org/functions/__DATA__ https://perldoc.perl.org/functions/__DATA__
- hey00 3y ago[flagged]
- hey00 3y agoI dont understand this website it is too hard and i dont understand anything. Anyone help me with this?
- nobody9999 3y agoI suggest starting here: https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html
- norir 3y agoThis is a great trick, but no one should ever run someone else's script that does this unless they have verified the script line by line beforehand.
- ygjb 3y agoSure, but that's turtles all the way down... any time you run untrusted code, you are making a risk based decision, usually based on the provenance of the code.
- michaelcampbell 3y agoMaybe? People run all manner of binaries/installers without checking them; I'm not sure why these sorts of things require any EXTRA scrutiny.
- fargle 3y agoEXACTLY! I can't stand the overblown security posturing with regards to things like bash and git, even python. When everyone in the world INCLUDING cyber-security professionals seems to think that it's perfectly OK to download setup.exe or install.msi from sketchy driver mfgs. and run them. But SHARs and curl | sh is too scary! It's not that any of these are a good idea, it's the blind-spot to traditional binary installers that's annoying. How do you audit them? Why are they OK (because they have to be)
- oefrha 3y agocurl | sh is actually bad because a completely innocuous script can become dangerous with a network hiccup, e.g. rm -rf /path/to/directory becomes rm -rf /path if the connection accidentally drops half way. Or something less dramatic, but still leaving you in a broken state. Some people took the “don’t curl | sh” advice as “you have to inspect every line of a shell script installer you download”, which is of course absurd.
- fargle 3y ago
- thinkmusic2000 3y agoI used to do something similar for Windows executable files. Append a large file to the end as necessary.
- Fudgel 3y agoI vaguely remember this is what Ocaml does for one format of its executable.
- mbreese 3y agoJava JAR files are similar, but reversed. You can add anything you want to the beginning of the JAR file (or is it any ZIP file?) so long as it doesn't include the Zip file header "PK". So, I use this to prepend a bash script that ultimately calls java -jar $0 It makes it very easy to setup and use Java based command line programs on a server.
- zmmmmm 3y agothis sounds incredibly useful but I couldn't get it to work. I just get java.util.zip.ZipException: invalid CEN header (bad signature) at java.base/java.util.zip.ZipFile$Source.zerror(ZipFile.java:1623) if I try to do anything with a JAR file that has leading text. I'm creating it just using echo 'java -jar $0' | cat - test.jar > test.run.jar Is there more to it?
- SeanA208 3y agohttps://github.com/puniverse/capsule/blob/master/capsule-util/src/main/resources/capsule/execheader.sh https://github.com/puniverse/capsule/blob/master/capsule-uti... This is what the (now defunkt) Capsule project prepends to get this same effect
- nneonneo 3y agoTechnically, you should update the offset to the central directory in the Zip footer, along with the offsets to each file header in each central directory entry. If you don’t, the zip file reader has to apply some heuristics to locate the central directory; not all readers implement these heuristics, and those that do won’t always be robust. The “unzip” utility can be useful as a sanity check; run “unzip -t” to test the integrity of the file.
- mbreese 3y agoDid you add a shebang line? I have a small bash stub that I use. It’s roughly (I also add JAVA_OPTS, etc…): #!/bin/bash java -jar $0 Then… cat stub.sh myproject.jar > myexec See: https://github.com/compgen-io/ngsutilsj/blob/master/src/scripts/stub.sh https://github.com/compgen-io/ngsutilsj/blob/master/src/scri...
- karmicthreat 3y agoI did a similar thing for a lowish volume embedded product. The update files are just bash scripts with a tar file cat'd on them. The unit just looks for a particular file on an external flash drive to run and the bash script runs, copies off a tar and checks that it has the right hash. Super simple and flexible when customers need me to do something special. Like extract some specific log onto a flash drive.
- RajT88 3y agoThis is a malware technique. I am not saying don't do it. But that is mostly where I see this type of trick.
- rubicks 3y agoSee also: https://makeself.io/ https://makeself.io/ https://manpages.debian.org/bookworm/sharutils/shar.1.en.html https://manpages.debian.org/bookworm/sharutils/shar.1.en.htm...
- zeroonetwothree 3y agoVon Neumann architecture to the extreme :)
- jmclnx 3y agoThis for any sh type script, not just bash :) Will work with sh, ksh and even [t]csh
- doktorhladnjak 3y agoThis reminds me of a job I had 15+ years ago where we did code reviews by emailing files to one another with our changes. It worked like this with the first part of the file being a script and the end of the file being a base64 encoded zip of the changed files. We had tooling that would pack them, but unpacking was done by execution. What could possibly go wrong with emailing executable scripts?
- habibur 3y ago> What could possibly go wrong with emailing executable scripts? Server side malware filters will strip the attachment.
- mogwire 3y agoA very large Electronic Medical Records company shipped an extremely large shell script to us for an install. Upon examination it contained binary data and a command to extract it to a file and then installed the application. This was the “efficient” way to ship and install the binary.
- lakomen 3y agoIt's also good for signed bash scripts.
- febed 3y agoIt’s better explained here: https://www.xmodulo.com/embed-binary-file-bash-script.html https://www.xmodulo.com/embed-binary-file-bash-script.html
- speg 3y agoThank you. As a shell amateur I was having trouble wrapping my head around the original description.
- OnlyMortal 3y agoOne “naughty” thing you can do is write invisible data into the last block of a file… - truncate the file to extend it to the end of the last block - write data to that area - truncate the file back to its original size An edit of that file will likely lose you data though.
- lloeki 3y agoHa, turns out I just wrote this helper function a few weeks ago, inspired by Perl and Ruby: #!/usr/bin/env bash # read data starting from the provided section marker up to the next one or EOF function section() { local section="$1" local source="${BASH_SOURCE[0]}" awk '/^__[A-Z0-9]+__$/{f=0} f{print} /^'"${section}"'$/{f=1}' "${source}" } section __JSON__ | jq section __YAML__ | ruby -ryaml -e 'p YAML.load(STDIN.read)' exit __JSON__ { "a": 1 } __YAML__ b: - 1 - 2 - 3 My only wish is that shellcheck had a directive to stop yelling at me starting at a certain line. Usually I augment it with such functions for clarity: # whatever raw data function data() { section __DATA__ } # man/perldoc like function doc() { section __DOC__ } # command line help function help() { section __HELP__ }
- TacticalCoder 3y ago> My only wish is that shellcheck had a directive to stop yelling at me starting at a certain line. But it does no? Shellcheck can be configured to ignore SC by putting a "disable" directive in a config file or even as a comment in your Bash source file. It would work if it's always the same annoying warning / SCxxx message you get.
- lloeki 3y agoDirectives apply line by line. The only ones that apply to a bigger scope are mandated to be the next line right after shebang and apply to the whole file. https://github.com/koalaman/shellcheck/wiki/Directive https://github.com/koalaman/shellcheck/wiki/Directive > Directives that replace or are immediately after the shebang apply to the entire script. Otherwise, they are scoped to the command that follows it https://github.com/koalaman/shellcheck/wiki/Ignore https://github.com/koalaman/shellcheck/wiki/Ignore > Note that the directive must be on the first line after the shebang with versions before 0.4.6. As of 0.4.6 comments and whitespace are allowed before file-wide directives.
- lloeki 3y ago
- acc_297 3y agoI use this at work for batch scripts which call R code for some of their functionality it’s very handy providing somebody who’s not very technology literate a solution which is a single .bat file which windows is happy to run by double clicking than a directory of files which must be stored together in order to work
- ilyt 3y ago....that's horrid. Why would you do that to your fellow humans ? just use cat >outfile <<EOF some data EOF add base64 if binary edit: after looking thru the thread I am deeply disappointed so little people know of that feature.
- zulban 3y ago>deeply disappointed I am deeply disappointed you didn't know this at some point in the past. An alternative oulook: https://xkcd.com/1053/ https://xkcd.com/1053/
- ilyt 3y agoI expected in a forum with far more than 10k users for this to be common enough that it would be first suggestion or the post's content. Not apparently some rare unknown knowledge that I only thought it was normal because corpo I work for used it often in CM. And to clarify I'm not really disappointed in people, just the fact it is unknown
- pornel 3y agoSadly, it won't work with my favourite curl | sh.
- 1vuio0pswjnm7 3y ago"All you have to do is make sure you append an explicit 'exit' to the end of your program before your new 'data section', so that bash won't parse any of the 'data section'." Or just use exec. exec tail -c [number of bytes for the binary] $0