15 ms·
Never run ‘python’ in your downloads folder
- codethief 6y agoPython's import system would be my biggest complaint about the language. I've used Python for more than ten years, but there are so many pitfalls with regards to imports that I still have to look things up on a regular basis. What makes things even worse is that packages sometimes even meddle with how imports work. (See e.g. Tensorflow – I recently had to debug the import magic they're doing. It was a nightmare.) Don't get me wrong, I LOVE Python! It's my go-to language for both small and big tasks. But sometimes I do wish Python had a function like PHP's require()… (Obviously for package-local imports, not for importing other packages.)
- kokooo 6y agoI've seen this complaint for almost every language, and the few languages i don't see this complaint for, people end up complaining about something else. It's too late to change how imports work and it's not especially complicated so no use complaining.
- Trasmatta 6y agoI pretty much never have problems with imports in Ruby. It's not like it's an unsolvable problem. Nothing would ever get better if we just took the "no use complaining" approach.
- BiteCode_dev 6y agoWhat's the differences between python and ruby import systems ?
- masklinn 6y agoThat they never had a problem with one. I’ve never had a problem with python’s import system either.
- pabs3 6y agoIt is never too late to change a language's default import paths. Perl removed . from @INC a couple of years ago.
- staticautomatic 6y agoI recently spent an entire day trying to import AllenNLP and its dependencies. I gave up after trying multiple interpreters, virtual environments, and operating systems.
- chishaku 6y agoBefore Docker, I remember yak shaving like this with lxml, numpy, scipy and other libraries. Now that I do everything in a container, life is much improved. Try Docker as documented in the AllenNLP README: $ mkdir -p $HOME/.allennlp/ $ docker run --rm -v $HOME/.allennlp:/root/.allennlp allennlp/allennlp:latest
- hinkley 6y agoThat was probably the first things that attracted me to Docker. If I started a new job or new project, it was the same story every time. “Do these five things and then you should be able to run the code” after half a day of back and forth I’d have a list of ten things, and the next new person would discover it is actually 12. Docker doesn’t just change your experience, it forces the author to actually capture most of their prerequisites. To the point where even if you don’t use docker, you benefit from a project having at least tried to use it.
- jhayward 6y agoInteresting, I just went to the repo's README and followed the directions for conda and had it installed and running in 3 commands.
- staticautomatic 6y agoFrankly I was just trying to get a pre-trained ELMO model deployed via Cortex. As best I could tell the problem boiled down to a failure related to installing tensorflow with gpu disabled. I am a NN noob but still, it just shouldn’t be this hard.
- FridgeSeal 6y ago
- nonbirithm 6y agoLua uses a "require()" function instead of an import statement for loading modules. I think that this is a crucial difference from a lot of programming languages that is vastly understated. What this means is that you can completely replace "require()" with your own function that has complete control over how the module importing system works. Because of this it is possible to build your own runtime code hotloader entirely in Lua without needing to modify the source of the compiler, and thus Lua is the only "mainstream" programming language listed in the list of notable live coding environments on Wikipedia[0]. (I'm surprised that Clojure isn't listed there, though.) The only way this can be accomplished is by designing the import system from the start to use a function instead of a statement. It's too late for Python to redo things and change its import system, unfortunately. It really is the one language feature you have to get right the first time or you'll miss out on those kinds of enhancements forever. [0] https://en.m.wikipedia.org/wiki/Live_coding#Notable_live_coding_environments https://en.m.wikipedia.org/wiki/Live_coding#Notable_live_cod...
- sakex 6y agoNodejs has require too
- yzmtf2008 6y agoUntrue, you could literally replace `__import__` in python and achieve the same thing: https://docs.python.org/3.8/library/functions.html#__import__ https://docs.python.org/3.8/library/functions.html#__import_...
- Izkata 6y agoUnless something has changed or there's a detail involving builtins I'm not aware of, I don't think this would work, for the same reason as the most common pitfall when mocking functions in python: You have to replace it where it's used, not where it's defined. To do that on a module, you have to first import that module - and then all the imports at the top of it would've already run.
- yzmtf2008 6y ago
- yongjik 6y agoWell, to be fair, Tensorflow is a Google library - pretty much any Google-made library will have five levels of indirection before it does anything, regardless of language. It's in their DNA.
- hinkley 6y agoMaybe I should convince a couple coworkers to apply to Google. They’d fit right in.
- Lammy 6y agoThe different styles of programmers need each other! Checks and balances :)
- kristopolous 6y agoOnce a company gets sufficiently large it's indistinguishable from a cult.
- kinkrtyavimoodh 6y agoIs there actually any veracity to this claim or is it general anti-google shit-flinging? I ask because from what I understand from Google's monorepo, it should be easy for engineers to take direct dependencies on whatever they want. Of course TF is an exported out library so things might be different, but surely it won't then be a good example of Google's DNA?
- galkk 6y ago> I ask because from what I understand from Google's monorepo, it should be easy for engineers to take direct dependencies on whatever they want. Not correct. The dependency system supports visibility restrictions - you can take direct dependencies to anything only if you're in your own experimental tree. Otherwise package authors can control access. You can see that even in the Bazel: https://docs.bazel.build/versions/master/visibility.html https://docs.bazel.build/versions/master/visibility.html (google employee)
- bobbyi_settv 6y agoDoes anyone know of a write-up that describes Tensorflow's import magic? I'd be curious to read about what they're doing and why.
- codethief 6y agoI wish I had a write-up but here's a starting point at least: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/api_template.__init__.py https://github.com/tensorflow/tensorflow/blob/master/tensorf... Inside the source code files, they then use a decorator (@tf_export) to define where in the package/module hierarchy a function/class should get placed, see e.g. https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/summary_ops_v2.py https://github.com/tensorflow/tensorflow/blob/master/tensorf... For instance, the write() function in that module actually gets exposed as `tf.summary.write()` even though it's contained in the file tensorflow/python/ops/summary_ops_v2.py. Needless to say, this even brings auto-completion of an IDE like IntelliJ to its knees at times. To make things even worse, things like Tensorboard also make use of global state everywhere, see e.g. the function _should_record_summaries_internal() in the above file. But now the question is: Which module's global state does the function actually access if it gets "moved" to a different module by TF? At the end of the day, I gave up hunting down that bug (_should_record_summaries_internal() always returned False, no matter what) and resorted to mocking the function out whenever I used Tensorboard. To do that (i.e. to find its parent module in the first place), I in turn used the `inspect` package. This is what I meant by "import magic". :)
- shp0ngle 6y agothis hurts.
- lmuench 6y agoPython’s import system is weird but it does make sense when you use absolute imports: project ├── main.py ├── package1 │ ├── module1.py │ └── module2.py └── package2 ├── __init__.py ├── module3.py ├── module4.py └── subpackage1 └── module5.py As long as the file containing the main() function resides in the top-level folder you can use absolute imports in every file, e.g. from package1 import module1 from package1.module2 import function1 from package2.subpackage1.module5 import function2
- Galanwe 6y agoAgree, python import system is annoyingly configurable, and way too much libraries use these features to do magic. I regularly have to spend some time on pandas source code just to untangle the mess of import aliases magic they do. But the standard library is no exception, importing os will dynamically import os.path.
- x86_64Ubuntu 6y agoI ran into that issue a few weeks ago. I had a small project that was supposed to migrate rows from a MySQL server into a SQL Server instance. For some reason that I was too exhausted to figure out, the program would only work if I imported the mysql connector before the pydobc stuff. That was very frustrating to say the least.
- taeric 6y agoPython seems to embrace more magic and other subtle visual cues than any other language. I am convinced it is only "beginner friendly" because of how much folks insist it is so.
- zitterbewegung 6y agoI have heard the opposite from my friends who interacted with people saying that the language is intimidating because its "a real programming language". But, to be fair deployment and package management of python is a mess when you try to scale up.
- dnautics 6y agoThat's also true of its "explicit is better than implicit" rule.
- kevin_thibedeau 6y agoPython has little magic in it. It's 3rd party libraries that do un-Pythonic things.
- kissgyorgy 6y ago> Python has little magic in it. LOL
- quietbritishjim 6y agoThe descriptor protocol is definitely magic! It normally doesn't matter that it's magic because most people don't need to interact with it. And it causes a very intuitive result in the part of the language that most people actually do interact with: that x = obj.foo; x() is the same as obj.foo() (not true in Javascript!). But that doesn't change the fact that method binding happens in a more complex way under the hood than you'd expect.
- coldtea 6y ago>I am convinced it is only "beginner friendly" because of how much folks insist it is so. Well, also because you can open a text editor and have a script with very straighforward code in 10 minutes...
- heavenlyblue 6y agoWell, to be clear if you accidentally downloaded a main.c file with a virus and then accidentally copied someone’s advice to compile an app using “cpp main.c” while being accidentally in your Downloads folder, you will also accidentally end up with a virus on your computer. This article is too long for what it is trying to say. Although I will go the author a kudos for being able to write so well about nothing - it took me to scroll about a third of the article to realise that what they were saying was blatantly obvious. PS: the only reason I clicked on it was because I thought python by default would import files with some “magic” names in the current directory.
- kgwgk 6y agoFor most people it’s not blatantly obvious that running python -m pip install ./totally-legit-package.whl in a folder will execute a file pip.py in that folder
- verroq 6y agoThe real question is who runs python -m pip when there is a perfectly good /usr/bin/pip?
- coldtea 6y agoThat's an easy question to answer. python -m pip is often advised over pip or /usr/bin/pip, because it uses the version of pip associated with that particular major+minor python install. https://snarky.ca/why-you-should-use-python-m-pip/ https://snarky.ca/why-you-should-use-python-m-pip/
- travisoneill1 6y agoWait, but isn't the point of the article that it will not do that, and rather just run the pip.py in you current working directory?
- Sebb767 6y ago
- rurban 6y agoWith Perl we fixed that security risk earlier (3 years ago) and better. The path of the script is never added to the global script path anymore. This fixed all these attacks automatically. If the script was in /tmp, ~/Downloads or just ~/. We don't need to warn our users to take care, they won't. Most of them don't even read such best practice warnings.
- marcosdumay 6y agoThe problem then is that more complex programs are harder break down, and people will tend to write their entire program in a single file. What does happen a lot with Perl.
- JJMcJ 6y agoSimilar to how Bash usually doesn't have current directory on the default PATH for executables, you must explicitly reference it ./local-executable for similar concerns.
- rahimnathwani 6y agocsh (tcsh?) used to be my default shell, and the current directory was first in the path. That was really convenient. At the time, I didn't realise the risks.
- wilsonrocks 6y agoI have always wondered why this was, compared to DOS!
- 0x0 6y agoI got burnt by an unfortunate combination of ActivePerl registering script execution as the default handler for .pl files, and some random file transfer application's "view file" dispatching to the Windows default file handler once. Wanted to view the source of a remote perl script; it ended up downloading and immediately executing the script locally with no warning. Boom!
- WesolyKubeczek 6y agoAh, that's also how you made a lot of CPAN simply not installable, prompting people who had actual jobs and deadlines just to drop in the variable USE_UNSAFE_INC=1 to get the old behavior back, and call it a day. Or keep their Perl at 5.24.3.
- teddyh 6y agoThe Python 3 executable has the “-I” (isolated mode) option to mitigate this kind of thing.
- jwilk 6y ago-I is hardly useful, because it also removes the user's site directory packages directory (typically ~/.local/lib/python3.X/site-packages/) from sys.path. So packages installed with, say, "python3 -m pip install --user" wouldn't be importable.
- tgflynn 6y agoAside from the Python related stuff one thing that confuses me about this article is the claim than a browser can put files in your Downloads directory without you requesting that. Exactly what browser/HTTP feature allows that ? I don't think I've never encountered this behavior and any time I've downloaded a file I've had the opportunity to see and change the name it is stored under.
- Sebb767 6y agoIt is possibly when you select 'always save to downloads' in your preferred browser.
- minitech 6y ago(And this is the default in all major browsers!)
- tgflynn 6y agoYou're right, if I just click on a link to a file that's not a PDF in chrome (in this case it was a .ps) it does immediately download the file, but it at least displays the filename in the downloads bar below the main window. So as far as I can tell it would be hard for a site to cause a download to occur without the user noticing.
- masklinn 6y agoThe user not noticing is not necessarily relevant if they don’t act on it immediately, browsing to their download folder to remove the offending file. The file could stay in there for years just waiting for the system’s user to slip up. This is obviously not a targeted attack under time constraints.
- AlphaSite 6y agoIn safari it will not let you download without explicit user interaction.
- dwheeler 6y agoHmm.. it seems that part of the problem is that Python interprets "empty entry in PYTHONPATH" as being the same as ".". That just seems wrong. If you want the current directory as one of your PYTHONPATH entries, then you should be required to insert ".". Perhaps the sharp edge would be reduced if Python skipped empty entries in PYTHONPATH. This might require some people to explicitly add "." in their configurations, but they should have done that anyway, and it wouldn't require code to change. PYTHONPATH is also getting less used anyway, so this might be a relatively unnoticed change by most. What do you think? Would that be worth proposing?
- mlvljr 6y agoYes, looks like a bug.
- AlphaSite 6y agoI think this would break the fundamental workflow of python where you can run a script and it can import from your CWD.
- masklinn 6y agoScripts being run (`python foo.py`) are not imported. What it would break is running a script and trying to import from a utility module next to it.
- jwilk 6y agoNo, that's unrelated to PYTHONPATH; it wouldn't break either.
- masklinn 6y agoIt's not. That sys.path includes "." when running a script lets you import files next to the script. This is trivial to confirm: > cat "import bar" > foo.py > touch bar.py > python foo.py > python -I foo.py Traceback (most recent call last): File "foo.py", line 1, in <module> import bar ModuleNotFoundError: No module named 'bar' and this is exactly the feature which makes it risky to run `python` from an untrusted location.
- richard_g 6y agoReminds me of a Wordpress site that ended up having a bunch of files uploaded to the uploads directory with the extension .php -- and the webserver allowed them to be executed. Unfortunately things like this are often overlooked!
- fomine3 6y agoSome Windows programs has similar issue (DLL preloading attack). The vulnerability is common since around 10 years ago but the mitigation is not enabled by default so some apps/installers still vulnerable.
- da39a3ee 6y agoThis isn't a solution, but I recommend always using /tmp as a Downloads folder (i.e. symlinking ~/Downloads to /tmp under macos so that it is at least purged every time you reboot.
- ezekiel68 6y agoGiven that python 0.9.0 was released in 1991, it's kind of astonishing (in a good way) that issues like these take nearly 30 years to creep into our collective consciousness. In my view, the focus on low friction and getting new users quickly productive with minimum fuss (shared by many scripting languages) sets the stage for these inevitable discoveries. And I'm not just hand-waving with the vague "these" characterization. Focus on ease-of-use creates roving best-practices factions that wax and wane over the years, with consequences perfectly captured in the famous xkcd "Python Environment" cartoon.[0] The fact that the empty string interpretation in PYTHONPATH hasn't been a bigger deal much earlier raises interesting questions. (Are bad guys not trying as hard as we imagine? Will newbie-friendly languages always end up with these kinds of exploitable surfaces? etc) [0] https://xkcd.com/1987/ https://xkcd.com/1987/
- jwilk 6y agoRelevant Python bug report: https://bugs.python.org/issue33053 https://bugs.python.org/issue33053
- cool-RR 6y agoGreat observation about Python scripts in the downloads folder. Here's a way to make this vulnerability even easier to exploit: Place a `sitecustomize.py` file in there. That way, you don't even need the `-m pip`, since that file will be imported automatically on any Python run.
- choeger 6y agoSELinux is made to prevent this kind of issues. Or rather: To encode the safety advice made by the author as a set of rules/constraints and ship it to everyone. Now the question is: Is SELinux the sawstop the author asks for?
- axilmar 6y agoAll these security issues stem from the fact that all the programs have a common view of the system. It's a major operating system flaw that no solution for it is in the horizon.
- Const-me 6y agoI wonder who and why decided that downloading files from the internets into a fixed location is a good idea? A downloaded file may be a PDF document, installer of a software, a song, a photo, CAD file, anything at all. Makes no sense to save everything into a same location. Hierarchical file systems were invented for a reason. For the same reason, I want browsers and chat apps to ask me where to save each file I download. Fortunately, most of these apps have an option to enable such behavior. As a nice side effects, this prevents security issues like the one described in the article.