11 ms·

Show HN: Pip Imports in Deno

192 points by eliassjogreen 3y ago

deno_python 0.3.1 adds support for importing python pip packages directly in JavaScript! Fun and useful, slightly cursed.

yohskar 3y ago
Nice !
amelius 3y ago
Does this work for packages like PyTorch too, in GPU mode?
- eliassjogreen 3y ago
  Yes, it uses your native python installation so everything that works with python should work with this. I have had success running sbert, faster-whisper and various other GPU accelerated packages.
motform 3y ago
A truly incredible feat of FFI!
parabyl 3y ago
Super cool! Fun, useful and slightly cursed indeed
abusaidm 3y ago
This is really fun indeed, well done, I like it. It’s funny how you decided to demo it with a library that does lots of c++ bindings it’s self. Now someone can put a wrapper around this to hide the python imports, so we can claim JS is the only language we need.
rolisz 3y ago
So you can now shoot yourself in the foot with both Python and JavaScript package management?
- Tajnymag 3y ago
  The package targets deno. It can be imported directly without using any package manager.
  - bandergirl 3y ago
    So you can shoot just one foot with pip
- agumonkey 3y ago
  maybe they will cancel each other ?
  - rolisz 3y ago
    Hah, we wish. In practice what will happen is we'll get the black hole spacetime disruption due to NPM package weight combined with dependency hell from pip
- IshKebab 3y ago
  Deno fixed the mess of JavaScript package management... But even NPM is light years ahead of Python. I have no idea why you'd want to inflict that on yourself once you've escaped it. I guess the only reason is there's some Python package that you have no option but to use.
- saiojd 3y ago
  Javascript's package management is so much better than Python's IMO...
  - ShamelessC 3y ago
    Package management, sure. The packages themselves leave a lot to be desired in JS.
  - throwawaymaths 3y ago
    I believe that's the joke, the two package systems have orthogonal nightmares
jadbox 3y ago
It would be magical if we also got typescript type hints too.
- Tajnymag 3y ago
  That sounds like a completely different beast of a tool
  - vorticalbox 3y ago
    Not so much you can use inspect [0] to see a whole lot of information. I used this at work to auto generate json scheme of passed functions for openAI function feature [1] [0] https://docs.python.org/3/library/inspect.html https://docs.python.org/3/library/inspect.html [1] https://platform.openai.com/docs/guides/gpt/function-calling https://platform.openai.com/docs/guides/gpt/function-calling
punnerud 3y ago
Let Python from Deno write to SQLite, Deno serve a webpage that use JS to accessing «static» SQLite using HTTP range requests, and we have come full circle (Joke, but would be a cool prototype)
FireInsight 3y ago
You can also do this in Nim, which basically means you can write any program you could in Python with libraries in Nim. https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy
qazxcvbnm 3y ago
Is there an equivalent for Node? I'm glad I haven't needed it since, but I couldn't find anything nearly so seamless when I last looked.
- swyx 3y ago
  why can't you just exec a python shell command to do the same thing? i'm actually slightly puzzled that "FFI" is its own thing when every language has a way to exec shell commands
  - nathan_phoenix 3y ago
    You can if you are just executing a static python script never changes. But if you want to call some python functions with your TS input then you would need to serialize the TS input, pass it as an arg to the script, there deserialize it and then call the python function with the input. And if you wanted to call multiple python functions then your first arg to the python script would need to be the name of the function, only then the serialized data. Plus you would need to implement a switch case on the python side to execute the right function.
    - swyx 3y ago
      right. ok. thats not hard^? like at all? and in fact ive had multiple bugs that ultimately had to do with serialization being quietly abstracted over for me that if i had just handrolled it myself i probably wouldve been consciously aware of my bad choices. ^i know this is probably arrogance out of naivete, just teasing out what exactly i dont know that i dont know about what FFI does.
      - viraptor 3y ago
        There are limits to what the commandline arguments can take. And even if you fallback to files instead, you still need to serialise a potentially huge amount of data. (Wanna serialise a 1GB tree that is going to get only one lookup in each call to the script?) On top of that, you're missing any chance of in-memory caching, because the script call can't preserve that, while FFI could. Then there are things that are just not possible to serialise like pointers to mapped memory, file descriptors (outside of unix sockets), etc. Finally even if everything works, a command execution and parsing results is massively slower than a function call. So yeah, there's lots of restrictions on what you can practically do that way.
willquack 3y ago
This is awesome and the right level of cursed! Reminds me of PythonMonkey, which provides really high level bindings to SpiderMonkey in Python. It ships as both a pip package and a standalone js runtime, the latter of which is very similar to this project! https://github.com/Distributive-Network/PythonMonkey#pythonmonkey https://github.com/Distributive-Network/PythonMonkey#pythonm...
- arjvik 3y ago
  Connecting these two would be even more cursed
arthur2e5 3y ago
The organization name "denosaurs" very roughly works for a combination of deno and python too: snakes fall into the clade of Lepidosaurs. Dinos are Archosaurs. I wonder if there's another programming language featuring a reptile. What's this "Lizard" language Duckduckgo's telling me about?
- pjot 3y ago
  I always get a kick out of well named projects - good eye!
- mbStavola 3y ago
  Maybe not to the same degree, but Zig! Its users are even called Ziguanas.
- mellosouls 3y ago
  Cobra https://en.m.wikipedia.org/wiki/Cobra_(programming_language) https://en.m.wikipedia.org/wiki/Cobra_(programming_language) And pushing it a bit as "a language": Anaconda
- tetris11 3y ago
  Xahlee features a lizard for Common Lisp on his webpage: http://xahlee.info/UnixResource_dir/writ/logo_lisp.html http://xahlee.info/UnixResource_dir/writ/logo_lisp.html
baq 3y ago
Can’t decide if I’m more amazed or terrified at the prospect of fighting both JS and Python package managers in the same process.
- tomcam 3y ago
  Deno package management is saner but more limited
rowanG077 3y ago
Everyday we stray further from gods light.
skybrian 3y ago
Next step would be getting ObservableJS running and using it to write notebooks. (There is a vscode plugin for this that might be adapted.)
re-thc 3y ago
Is Deno really performing so badly that it has to get PIP'ed? I thought NodeJs would be first...
- vorticalbox 3y ago
  Not at all but if you have an api in js and you want to do any sort of machine learning, ai, forecasting well all the best tools are in python. The you have two choices you rewrite your api into python or you call python from your js.
throwaway888abc 3y ago
I'm using this package so far. https://github.com/hmenyus/node-calls-python https://github.com/hmenyus/node-calls-python Excited to compare to Deno. Thanks guys!
kwhinnery 3y ago
ngl this is a little cursed, but awesome
nextaccountic 3y ago
An alternative is metacall. The example in the readme is about calling Python from Javascript, but it also works with other languages, like Ruby, C#, Java, and other languages https://github.com/metacall/core https://github.com/metacall/core List of supported languages here https://github.com/metacall/core/blob/develop/docs/README.md#21-loaders-backends https://github.com/metacall/core/blob/develop/docs/README.md... In the future, maybe webidl (or extensions of it) will bring interoperability between languages too. At the moment there is https://mozilla.github.io/uniffi-rs/ https://mozilla.github.io/uniffi-rs/ for interoperability between Rust and a number of languages (basically the ones mozilla needs: Swift, Kotlin, Javascript)
ditz 3y ago
[flagged]
Alifatisk 3y ago
Even though this is a but cursed, it could allow more people to access Pythons large and exclusive ecosystem libraries!

192 points by eliassjogreen 3y ago

deno_python 0.3.1 adds support for importing python pip packages directly in JavaScript! Fun and useful, slightly cursed.

yohskar 3y ago

Nice !

amelius 3y ago

Does this work for packages like PyTorch too, in GPU mode?

eliassjogreen 3y ago
Yes, it uses your native python installation so everything that works with python should work with this. I have had success running sbert, faster-whisper and various other GPU accelerated packages.

motform 3y ago

A truly incredible feat of FFI!

parabyl 3y ago

Super cool! Fun, useful and slightly cursed indeed

abusaidm 3y ago

This is really fun indeed, well done, I like it. It’s funny how you decided to demo it with a library that does lots of c++ bindings it’s self. Now someone can put a wrapper around this to hide the python imports, so we can claim JS is the only language we need.

rolisz 3y ago

So you can now shoot yourself in the foot with both Python and JavaScript package management?

Tajnymag 3y ago
The package targets deno. It can be imported directly without using any package manager.
- bandergirl 3y ago
  So you can shoot just one foot with pip
agumonkey 3y ago
maybe they will cancel each other ?
- rolisz 3y ago
  Hah, we wish. In practice what will happen is we'll get the black hole spacetime disruption due to NPM package weight combined with dependency hell from pip
IshKebab 3y ago
Deno fixed the mess of JavaScript package management... But even NPM is light years ahead of Python. I have no idea why you'd want to inflict that on yourself once you've escaped it. I guess the only reason is there's some Python package that you have no option but to use.
saiojd 3y ago
Javascript's package management is so much better than Python's IMO...
- ShamelessC 3y ago
  Package management, sure. The packages themselves leave a lot to be desired in JS.
- throwawaymaths 3y ago
  I believe that's the joke, the two package systems have orthogonal nightmares

jadbox 3y ago

It would be magical if we also got typescript type hints too.

Tajnymag 3y ago
That sounds like a completely different beast of a tool
- vorticalbox 3y ago
  Not so much you can use inspect [0] to see a whole lot of information. I used this at work to auto generate json scheme of passed functions for openAI function feature [1] [0] https://docs.python.org/3/library/inspect.html https://docs.python.org/3/library/inspect.html [1] https://platform.openai.com/docs/guides/gpt/function-calling https://platform.openai.com/docs/guides/gpt/function-calling

punnerud 3y ago

Let Python from Deno write to SQLite, Deno serve a webpage that use JS to accessing «static» SQLite using HTTP range requests, and we have come full circle (Joke, but would be a cool prototype)

FireInsight 3y ago

You can also do this in Nim, which basically means you can write any program you could in Python with libraries in Nim. https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy

qazxcvbnm 3y ago

Is there an equivalent for Node? I'm glad I haven't needed it since, but I couldn't find anything nearly so seamless when I last looked.

swyx 3y ago
why can't you just exec a python shell command to do the same thing? i'm actually slightly puzzled that "FFI" is its own thing when every language has a way to exec shell commands
- nathan_phoenix 3y ago
  You can if you are just executing a static python script never changes. But if you want to call some python functions with your TS input then you would need to serialize the TS input, pass it as an arg to the script, there deserialize it and then call the python function with the input. And if you wanted to call multiple python functions then your first arg to the python script would need to be the name of the function, only then the serialized data. Plus you would need to implement a switch case on the python side to execute the right function.
  - swyx 3y ago
    right. ok. thats not hard^? like at all? and in fact ive had multiple bugs that ultimately had to do with serialization being quietly abstracted over for me that if i had just handrolled it myself i probably wouldve been consciously aware of my bad choices. ^i know this is probably arrogance out of naivete, just teasing out what exactly i dont know that i dont know about what FFI does.
    - viraptor 3y ago
      There are limits to what the commandline arguments can take. And even if you fallback to files instead, you still need to serialise a potentially huge amount of data. (Wanna serialise a 1GB tree that is going to get only one lookup in each call to the script?) On top of that, you're missing any chance of in-memory caching, because the script call can't preserve that, while FFI could. Then there are things that are just not possible to serialise like pointers to mapped memory, file descriptors (outside of unix sockets), etc. Finally even if everything works, a command execution and parsing results is massively slower than a function call. So yeah, there's lots of restrictions on what you can practically do that way.

willquack 3y ago

This is awesome and the right level of cursed! Reminds me of PythonMonkey, which provides really high level bindings to SpiderMonkey in Python. It ships as both a pip package and a standalone js runtime, the latter of which is very similar to this project! https://github.com/Distributive-Network/PythonMonkey#pythonmonkey https://github.com/Distributive-Network/PythonMonkey#pythonm...

arjvik 3y ago
Connecting these two would be even more cursed

arthur2e5 3y ago

The organization name "denosaurs" very roughly works for a combination of deno and python too: snakes fall into the clade of Lepidosaurs. Dinos are Archosaurs. I wonder if there's another programming language featuring a reptile. What's this "Lizard" language Duckduckgo's telling me about?

pjot 3y ago
I always get a kick out of well named projects - good eye!
mbStavola 3y ago
Maybe not to the same degree, but Zig! Its users are even called Ziguanas.
mellosouls 3y ago
Cobra https://en.m.wikipedia.org/wiki/Cobra_(programming_language) https://en.m.wikipedia.org/wiki/Cobra_(programming_language) And pushing it a bit as "a language": Anaconda
tetris11 3y ago
Xahlee features a lizard for Common Lisp on his webpage: http://xahlee.info/UnixResource_dir/writ/logo_lisp.html http://xahlee.info/UnixResource_dir/writ/logo_lisp.html

baq 3y ago

Can’t decide if I’m more amazed or terrified at the prospect of fighting both JS and Python package managers in the same process.

tomcam 3y ago
Deno package management is saner but more limited

rowanG077 3y ago

Everyday we stray further from gods light.

skybrian 3y ago

Next step would be getting ObservableJS running and using it to write notebooks. (There is a vscode plugin for this that might be adapted.)

re-thc 3y ago

Is Deno really performing so badly that it has to get PIP'ed? I thought NodeJs would be first...

vorticalbox 3y ago
Not at all but if you have an api in js and you want to do any sort of machine learning, ai, forecasting well all the best tools are in python. The you have two choices you rewrite your api into python or you call python from your js.

throwaway888abc 3y ago

I'm using this package so far. https://github.com/hmenyus/node-calls-python https://github.com/hmenyus/node-calls-python Excited to compare to Deno. Thanks guys!

kwhinnery 3y ago

ngl this is a little cursed, but awesome

nextaccountic 3y ago

An alternative is metacall. The example in the readme is about calling Python from Javascript, but it also works with other languages, like Ruby, C#, Java, and other languages https://github.com/metacall/core https://github.com/metacall/core List of supported languages here https://github.com/metacall/core/blob/develop/docs/README.md#21-loaders-backends https://github.com/metacall/core/blob/develop/docs/README.md... In the future, maybe webidl (or extensions of it) will bring interoperability between languages too. At the moment there is https://mozilla.github.io/uniffi-rs/ https://mozilla.github.io/uniffi-rs/ for interoperability between Rust and a number of languages (basically the ones mozilla needs: Swift, Kotlin, Javascript)

ditz 3y ago

[flagged]

Alifatisk 3y ago

Even though this is a but cursed, it could allow more people to access Pythons large and exclusive ecosystem libraries!