6 ms·
Magicimport.py: Python code that fetches its dependencies without complaining
- tetris11 6y agoSince moving to Nix, I've been forced to setup virtual envs for my more interesting projects (oh sure, I install the common libraries at the system level), and though I resisted this workflow a lot initially (I like to keep a tight ship), I've come to embrace its practicality when working on projects that have libs with breaking changes due to rapid releases. it's saved me a world of pain a number of times to have different versions of the same libs installed in different envs
- rbanffy 6y agoI would discourage distros from even shipping easy_install and pip by default. One should never mess with the system Python.
- rowanG077 6y agoThis is what I love about Nix. It has actually completely solved dependency hell for me. It's always a little painful to move an existing project to Nix but I have never regretted it.
- _rs 6y agoI opened this expecting Python to allow you to define custom import handlers at runtime, but this is cool too.
- ekimekim 6y agoThat is in fact also possible, and you could use it to make this magicimport system get called automatically whenever you try to import a non-existent library. https://docs.python.org/3/reference/import.html#import-hooks https://docs.python.org/3/reference/import.html#import-hooks
- rbanffy 6y agoThis is one of those cases when something that can be done just shouldn't.
- makeworld 6y agoCool solution for small scripts! Hopefully this will lead to more virtualenvs and dep pinning in Python. I wonder if this works cross-platform... Would definitely recommend Poetry for anything larger, it's been a great experience.
- bemmu 6y agoAnother thing I'd like for small projects is for "obvious" modules to be imported without having to say so. For example if I do glob.glob("*.txt") it's rather obvious that I also wanted to "import glob".
- zb1plus 6y agoI think this would be super helpful within the context of a jupyter notebook
- refactor_master 6y agoAn IDE like PyCharm should be able to suggest the import for you.
- mhh__ 6y agoD has import std; which just imports the whole standard library in one go (for writing). You might be able to write your own (not sure if python has public imports)
- makapuf 6y agoYou can check https://pypi.org/project/smart_imports/ https://pypi.org/project/smart_imports/
- fulafel 6y ago> Get source code of the module, from which smart_imports.all() has called. > Parse it, find all not initialized variables. > Search imports, suitable for found variables. Import them. Genius. (Evil genius mabye...)
- black3r 6y agoPyCharm has functionality that allows you to find the correct import in libraries across your project. If it finds multiple functions / objects with the same name, it gives you choices. For me on larger projects it shows choices ~80% of the time. I wonder how can this work correctly on anything larger than the provided example.
- mherrmann 6y agoAs someone who has been coding for 20 years, "magic" is a negative adjective for me. I don't want magic code. I want code that is simple and works in obvious ways. > You can even specify a version number Sorry to be sarcastic here but wow, you mean I can even make sure that my code will not break at an arbitrary point in the future, when my dependency inevitably introduces a breaking change? Fantastic. I understand that installing dependencies is a bit of a hassle. But really, once you have done it twice it takes seconds. And I would not want a mechanism that fetches dependencies as a side effect of my app running. I want to know what's going on when my app runs. The fewer moving parts, the better. Often, it's hard work to achieve this simplicity. But, at least in my experience, it pays off.
- TTPrograms 6y agoThis is a toy/neat experiment that someone made and they explicitly say you should not use it for production. This sort of criticism is inappropriate (or at best, poorly framed). This is Hacker News, and this is a Hack. If I made a bot that drives cars around in GTA would you make a post ridiculing how dangerous my lane detection algorithm would be for a self-driving car in snow? Furthermore, the point OP makes about L5 cars being presumably achievable while "L5 software" is not is a 100% valid point - why should we not aspire to that level of consistency and effectiveness in our systems? Maybe achieving this is actually possible and projects like this inspire the approaches that actually deliver those capabilities in a production-mature form.
- mherrmann 6y agoYou're basically saying that I am not allowed to give my opinion on this project, because it happens to be critical. I don't know how we can have interesting discussions in this forum if criticism is not allowed. At the end of the day, I think you're saying it's a fun toy and I shouldn't worry, whereas my point was that I wouldn't even want to use it as a toy, and I am afraid it will lead especially less experienced programmers who do use it in production down a path that hurts them in the long run.
- 6y ago
- ampdepolymerase 6y agoRepl.it also offers a more production ready version called Universal Package Manager https://github.com/replit/upm https://github.com/replit/upm
- makeworld 6y agoWhat makes the repl.it version universal is that it works for multiple languages. It just uses Poetry[0] for Python under the hood. [0]: https://python-poetry.org/ https://python-poetry.org/
- brbsix 6y agoI can't say I really like the idea of a "magic" import, but I'm sure some people might find it handy. It just sounds like a recipe for having things break later in mysterious ways but if it's just a one-time use sort of thing I can see the appeal. Personally I've been really happy with with pyenv[0].You can install multiple different versions of Python side-by-side and it does a great job managing virtualenvs. pipx[1] is also pretty good for managing virtualenvs to install applications in but IMHO less so for doing actual development. You can use it to seamlessly install venv'd Python apps or spin up apps in ephemeral environments. [0]: https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv [1]: https://github.com/pipxproject/pipx https://github.com/pipxproject/pipx
- choeger 6y agoThe downside of such an approach is that it hides the inevitable complexity of dependency management. Things like version constraints, licensing and platform specifica are important. If developers don't see them, they won't learn about them. For instance I encountered a python wheel lately that "conveniently" shipped a precompiled .so file. Not only was that a security problem but also did that file not work properly under Ubuntu 18.04.
- shoo 6y agothis is a good example of a "simple versus easy" trade off [1] , i.e. this is something that is easier to get started at some expense of simplicity: it is over-complicated & hides complexity behind a simple interface, but not in a way that is reliable - so eventually things will break and then you'll be strictly more confused than if you did things in a less easy but simpler way - learning a bit about dependency management & setting up a reproducible environment for running your python script/app. that said, it is an amusing hack to run the virtualenv'd python in a subprocess and then extract the environment vars and inject them into the already running python process. some of the code could be cleaned up a bit -- e.g. it could be using https://docs.python.org/3/library/tempfile.html#tempfile.TemporaryDirectory https://docs.python.org/3/library/tempfile.html#tempfile.Tem... to just ask for a uniquely named temp dir, some of the subprocess invocations dont appear to have error handling, etc. for my python hobby project these days i only depend upon packages i can install using apt in debian stable. containers for isolation help too. [1] https://www.infoq.com/presentations/Simple-Made-Easy/ https://www.infoq.com/presentations/Simple-Made-Easy/
- jamesmishra 6y agoI'm disappointed to see so many negative HN comments about this project. I think it is really cool! You shouldn't use this in a production web service--and the author says as much in the README--I think there are some great uses for this code: 1. Demonstrating how the Python import system works. There is a lot of nuance at hand that other programming languages (like C) don't have an analogy for. 2. Creating truly self-contained Jupyter notebooks. Right now, if I want to send someone a working Jupyter notebook, I have to send them a requirements.txt (or, better yet, a Dockerfile) for them to install in a virtualenv. This is too bad, because specifying the dependencies in the notebook itself is better for teaching purposes.
- nemosaltat 6y agoRe: 2, IMO this is where makefiles shine. When I got my start on Linux with Ubuntu (Breezy Badger IIRC) I compiled many (most?) of the utilities I wanted from source. When `make` didn’t “JustWorkTM”, it give very clear reasons why, and even as a noob I could usually correct the issue. Python , Docker, and NodeJS etc still leave much to be desired in that regard.
- rbanffy 6y agoI'm concerned people will take it as a good idea instead of a demonstration of what can, but shouldn't, be done.
- Nullabillity 6y agoSadly, this is already a thing in a relatively common JS bundler..[0] :/ [0]: https://en.parceljs.org/hmr.html#automagically-installed-dependencies https://en.parceljs.org/hmr.html#automagically-installed-dep...
- Kaze404 6y ago> This only occurs in development (using serve or watch), however in production (using build) autoinstall is disabled to prevent unwanted side-effects on deployment. > You can disable this feature using --no-autoinstall. Seems fine to me.
- JosephRedfern 6y agoThis is cool! I wrote a script that would try and automatically determine missing packages and their version numbers a while a go: https://gist.github.com/JosephRedfern/ef52916e70497bc9631d6e018a97f84d https://gist.github.com/JosephRedfern/ef52916e70497bc9631d6e... It was a joke rather than a serious tool. Caught import errors and runtime errors and would install pip packages going back from most recent to oldest, until the script under test worked without error. At least, that was the idea —- I never really finished it.
- nemosaltat 6y agoObligatory xkcd: https://xkcd.com/1654/ https://xkcd.com/1654/
- xg15 6y ago> Don't use this in production. This is mostly just a convenience hack for small personal projects. Famous last words.
- boublepop 6y agoLooks cool, I have a small non-open source solution for this purpose that caries this out just using different syntax. The focus there was getting jupyter notebooks and dashboards that also detail their runtime environment requirements. Compared to R, pythons imports syntax and handling is a god-sent when sharing code, but we still saw a lot of people get stuck after cloning a notebook or dashboard trying to figure out all the dependencies they needed to set up to run it. Now they just copy the notebook/dashboard and first run configures everything For them referencing both internal and external dependencies. It’s a small but important difference especially for juniors. In the future I would love for python just to support something like “import pandas==1.1.3@pypi” and having that automatically do what every python developer who works with virtual envs would do if they saw the “==1.1.3@pupi” in a comment after the import. But then obviously also supporting installs from other sources than pypi.
- sakebomb 6y agoWhat happens if someones code accidentally (or purposely) uses a malicious module [0] in their code? Seems like a great way to spread some malware... It would be cool if there was some sort of dependency validation system integrated into this. [0](https://www.zdnet.com/article/two-malicious-python-libraries-removed-from-pypi/ https://www.zdnet.com/article/two-malicious-python-libraries...)
- BerislavLopac 6y ago> The ONLY package it will automatically install to your system (and in user-space, only) is virtualenv itself I don't understand why it even uses virtualenv, when venv is a) part of the standard library, and b) the recommended way to create virtual environments in Python. While the import from __future__ indicates that it's supposed to support Python 2, the shebang line is explicitly Python 3...
- nitroll 6y agoOne big reason is that ubuntu splits out venv into its own package basically "breaking" python. People then don't know about the venv module and continues using the virtualenv, both due to this preceding venv, and because that is usually the word used to describe the concept.
- d0mine 6y agoThere is the fades utility that installs marked dependencies automatically For example, one could replace a python shebang with fades' one and mark dependencies using # fades comments: #!/usr/bin/fades import dependency # fades https://github.com/PyAr/fades https://github.com/PyAr/fades