45 ms·
Import – A Simple and Fast Module System for Bash and Other Unix Shells
- meddlepal 6y agoNever change below your feet unless your machines are ephemeral, constantly changing, and thus do not have a permanent cache. Do not use this in production folks.
- SrslyJosh 6y agoThe bare minimum for saying "never change" involves a cryptographically secure checksum of the content, and even then it's not guaranteed.
- SrslyJosh 6y ago> import "https://git.io/fAWiz" https://git.io/fAWiz" Hoo boy.
- yjftsjthsd-h 6y ago...huh. I would have expected to pin to a commit (by hash), but that shortener link has nothing like enough address space to be doing that, so yeah that's a poor arrangement. But you could point to a commit, which I would think is quite good.
- deleted 6y ago[deleted]
- coolreader18 6y agoTo be fair, it is an immutable shortened url, but it could still point to a random site that then redirects to a git url (when it's being benevolent). Definitely better to point directly to a git commit hash, though still a bit sketchy regardless.
- price 6y agoExciting idea! For me the application of this that I'm most interested in personally is for dotfiles, configuring the interactive shell. I have a fair amount of useful stuff in my own bashrc that I've developed over the years, which people who see me work often want to use too (and I happily share). I've wished for a good way to share each logically-independent customization in its own separate chunk -- so that people can pick and choose, and also combine what they like from me with their own existing config and chunks they've picked up elsewhere. For zsh, oh-my-zsh seems to have been extremely successful, and IIUC this describes a key technical contribution at the core of how it's worked so well. So I'm hoping that this thing can play part of the role of oh-my-zsh, for bash! (And for other shells too, sure.) I'll definitely be interested to read more of the details of how it works, and to play around with it.
- j1elo 6y agoYou might be able to make your own "chunk modules" system, by sourcing from .profile something like ".shell_modules//.sh", and then use GNU Stow [0] to selectively install, from a centralized directory, whatever modules you're interested in. Each module would have to go to its own subdir (the '*' part in my example), but that's a good thing as it means that each module could contain several files. [0]: https://www.gnu.org/software/stow/ https://www.gnu.org/software/stow/
- price 6y ago(Before using this for real, I'd certainly want to be able to track the exact version I'm getting of a dependency, with a hash, in a "lockfile" sort of file that I can keep in version control. That's not mentioned in the demo on the front page - but if it doesn't already have that feature, I hope it will grow it.)
- JoshTriplett 6y agoIf you're pinning a shell script to a specific hash, why not just keep the script itself in version control?
- 6y ago
- JoshTriplett 6y agoSeems to be failing at the moment: https://imgur.com/a/UEbJBKA https://imgur.com/a/UEbJBKA
- boogies 6y agoIt was the same for me the first time I tried, but the "View on GitHub" link at the bottom still worked (and now the main page works too).
- kazinator 6y ago# The URL is downloaded once, cached forever, and then sourced import "https://git.io/fAWiz" Where is this program declaring the above content's expected SHA256 to check that the expected content is being received, and not some altered version? Think about it. You write the program with the above #import, and at the time, it fetches a certain piece of code. You see in your cache that it's the right one. Then you deploy the script to a fellow user. They run it, and the import fetches something different from the same URL. The script breaks. You're scratching your head: hey, it works on my machine, why not yours? Or worse: the replacement code is malicious not simply breaking. The user's files are gone or whatever.
- Spivak 6y agoDon't get me wrong, that's would be a nice feature, but I think it's a little optimistic that people would actually use it and ask for more security outside of TLS. I mean the most popular (n)vim plug-in managers just clone arbitrary git repos with no other verification. Code signing or hashing is vanishingly rare and a huge burden on the end user who simply, unfortunately, not care in the slightest. The "good enough" root of trust is that the server hosting the code hasn't been compromised.
- jiggunjer 6y agoIt's not so much about security as about reproducibility. What if this is being used in scientific research?
- kazinator 6y agoWhat if it's being used by your bank, government site, or embedded system in your TCP/IP router, or automobile?
- kazinator 6y agoFar from this being rare, every major GNU/Linux distro checks the hashes of downloaded material it depends on. Failing to do this threatens integrity/stability of the system and provides an open door for malware. Vim package managers pulling git repos can easily check out a specific hash; if they are just blindly following branch heads by name, that's just dumb and nobody should emulate their behavior. TLS is neither here nor there because there isn't much of a privacy concern. Plaintext HTTP works fine for fetching some piece of FOSS, if you check the digest to verify that you're getting what you're expecting to be getting. On the contrary, the assurance that the HTTPS host key has not changed is not relevant. It does not amount to (is something other than) the relevant assurance that the URL content has not changed. Moreover, there may be numerous hosts which provide that exact same content. If you have the digest, and the original host stops providing the content reliably, you can confidently switch to a different mirror, which could be a HTTPS host you have never contacted, or even a HTTP one. The trust that the server hosting the code has not been compromised is misplaced and foolish. The person in charge of that content could be someone who thinks it is a good idea to change the content of a software release (to sneak in one more bugfix) without changing anything in the URL. Thus two of your own users who think they are using the exact same version of your software will fetch the same foolib-1.2.tar.gz URL a day apart, and get different files, which breaks the integrity of your own release. Without hashes, this goes undetected. There is no "huge burden" here. All the module tool has to do is to make it mandatory for the fetching commmand to have a digest argument. If the digest doesn't match the content, all it has to do is fail that command, and issue a diagnostic. The diagnostic can include the actual hash that would work, and point to the file and line number where the failing import is located. The user relies on hundreds of imports that they didn't write themselves, and which require no attention at all in the happy case that they fetch what they are supposed to fetch.
- ary 6y ago# The URL is downloaded once, cached forever, and then sourced import "https://git.io/fAWiz" Call me old and cranky, but I thought this naive fad of piping the unchecked contents of URLs to the shell would die off eventually once people got clued in to how dangerous it is. At a bare minimum the contents of any shell script coming from a remote source should be manually reviewed the first time, and then verified to have not changed by comparing against a locally stored cryptographic hash. This is no better than curl with some automatic caching. At the end of the day why not store your dependencies along side your scripts and then not worry about it changing unless you change it yourself?
- endgame 6y agoThe dhall language does https imports, but with two important differences: 1. The language is _pure_ and _total_, so it will _eventually_ terminate and cannot perform side-effects 2. You can specify the SHA256 of the referenced resources inline, next to the URL. This gets you caching and breakage warnings.
- treve 6y agoPiping a url into bash is not more or less risky than downloading a windows binary, installing a snap, installing a npm/composer/whatever package with post-install hooks, downloading a binary for your Mac. It's impossible to read every line of code we execute (many of which are closed source anyway). Downloading a large script and reading it before running it in bash is also hardly good security measure. It's much more practical to teach people how to recognize trustworthy sources, and how HTTPS works. Or you can switch to a platform like iOS where there's complete control, sandboxes and reviews.
- ary 6y agoThe risks involved with executing code are legion, and wanting to promote cultural change in the direction of better safety isn't invalidated by pointing out failings in other areas. > Downloading a large script and reading it before running it in bash is also hardly good security measure. > It's much more practical to teach people how to recognize trustworthy sources, and how HTTPS works. Or you can switch to a platform like iOS where there's complete control, sandboxes and reviews. Downloading bash scripts and inspecting them before running is definitely a good security measure, as is running untrusted software in a VM or container where possible, and generally promoting caution when running anything acquired from the Internet. Even if the source is considered trustworthy there is still the possibility that it has been compromised. npm, pypi, and even the Linux kernel source tree are all examples of this. In this particular situation the use case is extremely weak. The problem is solved by incorporating dependencies locally, as I pointed out above.
- earthboundkid 6y agoYour scientists were so preoccupied with whether or not they could &c
- edw 6y agoColor me old, but wtf, the JS script kiddie obsession of never writing a line of code that can be packaged in a module has now come to shell scripting?!
- choeger 6y agoHello old, nice to meet you. I am skeptical.
- aeyes 6y agoNice! Who says that you have to download from arbitrary URLs? If you have to use public modules you are free to copy the source to some internal server first. It's just a single file! It looks like modules can't import other modules so this shouldn't get too out of hand.
- coredog64 6y agoAt my last employer (large US bank you’ve definitely heard of), they had something like this internally. Tools/scripts were packaged up as modules and you could/would “module load foo”. It knew what OS you were running, so Windows users could do the same thing for cross-platform tools. It was very handy and I’m looking forward to trying this as a substitute.
- alganet 6y agoI did a similar thingy a while back: https://github.com/alganet/coral/blob/master/doc/spec/require.md https://github.com/alganet/coral/blob/master/doc/spec/requir... It also does references (`require `something from inside modules) and some stuff. It is missing the download component though. I've also worked on https://github.com/Mosai/workshop https://github.com/Mosai/workshop, a previous attempt at shell tooling (please do not use it!) Also, there was a popular https://github.com/pixelomer/bashpm https://github.com/pixelomer/bashpm at some point. Most stable thing would probably be: https://github.com/shellfire-dev https://github.com/shellfire-dev Here are my thoughts working in this "hobby" for the last 6 years: - External tools are bad for shell portability and performance. It costs more to invoke "cut" than to do a loop and a couple of parameter expansions. Tools like cut, sed and similar are not suited for small strings, they are for large files. - ksh93 and ksh2020 support is possible, see: https://gist.github.com/bk2204/9dd24df0e4499a02a300578ebdca4728 https://gist.github.com/bk2204/9dd24df0e4499a02a300578ebdca4... These kshs are fast, I want them as supported engines. - Pipes are bad for performance. Local variable buffer is faster in all shells. There is a proof here somewhere: https://github.com/alganet/shell-proto https://github.com/alganet/shell-proto - We need to create a shell "dialect". Leave a lot of stuff out to simplify its syntax, so we can write a shell parser written in pure shell. This is a crucial point towards a better shell ecosystem. I have some research related to these issues, I'd love to collaborate in something. Standards, maybe? I think the shell community needs to talk as much as the early JS community did in the past.
- varbhat 6y agoI think that this is bad idea and it increases non-working scripts(what if url stops working? ) . Also, it increases malwares.(what if url pointed to is changed to malware script?) Golang's package management is very good, but this isn't. Why can't they just paste the required script in main script itself? What's the necessity of import here?
- maddyboo 6y agoThis would be perfect if it also included a bundler/linker to build single-file executables. I also find it amusing each time the argument against using URLs as import specs comes up. Go already does this, we do it on the web with <script> tags, and now Deno is doing it. It's proven to work as long as you can reasonably trust the URL.
- idclip 6y agothis makes me uncomfortable... reasons: 1. i dont believe package systems help bash. we already have ansible for systems ... if this goes the way of npm, bash will get bloat and chaos. 2. shells are a system process thats usually very privileged .. i see alot of abuse potential since it interprets everything you source. 3. see 1, and think performance on medium sized scripts, or if an expensive "module" is loaded. 4. versioning chaos, killing old scripts ... in the unix and linux space, some decade olds scripts are still used. this would expose weaknesses ... but also destabilise a lot. edit: 3,4, also thank you regardless of my opinion, your code and execution look and feels great.
- lma21 6y agowhat's wrong with source <file_path>
- kasabali 6y agoNot webscale /s
- risson 6y agoI hate the way a lot of tools nowadays assume it's okay to just create a new directory in the user's home. For instance, this tools create ~/.import-cache. Why doesn't it use a subdirectory in ~/.cache? What's the next step? ~/.import-config? And then ~/import-tmp? This is just getting out of hand. There are places where your tool can store config and cache data, use them.
- augustk 6y agoOr even better: $XDG_CACHE_HOME https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html https://specifications.freedesktop.org/basedir-spec/basedir-...
- remix2000 6y agoImpressive… How people manage to poison even old-school shell scripts with useless bloat. If you want to source remote scripts that badly (despite this being against policies of almost every company), better keep it simple: telesource() { f=~/.cache/${1/\//} ; [ -f "$f" ] || curl -SsLo "$f" "$1" && source "$f" ; } So you can invoke it like that: # The URL is downloaded once, cached forever, and then sourced telesource https://git.io/fAWiz What does the name ``import`` even mean? Does it anyhow relate to sourcing? Or even shell scripting at all? Now I'm waiting for a full JIT ECMAScript VM in pure portable Bash. Impress me even more ;) Nonetheless, that's what it looks like to me. I'm just not fond of spilling idioms from one technology to everything else its users touch. That's much like our IT equivalent of Midas touch.
- emiliovesprini 6y ago> What does the name ``import`` even mean? import name-clashes with ImageMagick's import, which comes pre installed in lots of Linux distros. Issue (by me): https://github.com/importpw/import/issues/35 https://github.com/importpw/import/issues/35