5 ms·

Fun fact: you can overload the Python import system to work with other languages that you create. I use this for my Python-based Lisp: https://github.com/shaww

Fun fact: you can overload the Python import system to work with other languages that you create. I use this for my Python-based Lisp: https://github.com/shawwn/pymen/blob/ml/importer.py https://github.com/shawwn/pymen/blob/ml/importer.py import foo checks for "foo.l", and if found, compiles it to foo.py on the fly and imports that instead. It's so cursed. I love it.

toomuchtodo 5y ago
A master of the dark arts I presume.
th0ma5 5y ago
JPype does this for importing Java classes.
Fordec 5y ago
Wait, what? > that you create As in existing language supplied eg. Perl, Java, etc., or literally anything? Like bootstrapping your own home made language from scratch?
- sillysaurusx 5y ago
  Literally anything. 'Tis a homemade homegrown lisp, grown by Scott Bell for several years till I took it all for myself. Nom nom. It starts with reader.l: https://github.com/shawwn/pymen/blob/ml/reader.l https://github.com/shawwn/pymen/blob/ml/reader.l where the raw character stream is turned into a bunch of nested arrays. E.g. (+ 1 2) becomes ["+", 1, 2] Then it's punted over to compiler.l https://github.com/shawwn/pymen/blob/ml/compiler.l https://github.com/shawwn/pymen/blob/ml/compiler.l where it's passed through `expand`, which does a `macroexpand` followed by a `lower`. E.g. (do (do (do (print 'hi)))) becomes ["print", "\"hi\""] Then the final thing is thrown to the compile function, which spits out print("hi") -- the final valid Python code that gets passed into the standard python `exec` function. Works with all the standard python things, like async functions and `with` contexts. Been screwing around with it for a few years.
  - Fordec 5y ago
    That's absolutely disgusting. But in a Web Assembly sort of way... I don't know whether to spit on it or give it a medal.
  - jazzyjackson 5y ago
    I have a homemade language that's not a lisp, but is lispy in some ways. I've only got to the point where I expand the code to an abstract syntax tree, and deciding how to go from there is the hard part for me right now. It's never crossed my mind to just compile it to valid python code. Thanks for the inspiration!
- masklinn 5y ago
  > As in existing language supplied eg. Perl, Java, etc., or literally anything? Like bootstrapping your own home made language from scratch? Should work with anything as long as you can ultimately generate Python bytecode (and provide a module object). The import system is not simple, but it's really open.
- aasasd 5y ago
  If the import system allows your code to run instead of the ‘import’ statement, and to produce the module however you want, then of course you can do whatever: load code from Google or StackOverflow results, if you wish.
bloopernova 5y ago
Can I just say that the introduction to Lisp you made in your README.md is really good! I keep trying to get into Lisp (and JavaScript, and TypeScript, etc etc) but I've been a sysadmin my whole professional life and also a chronic pain sufferer. That translates into mostly having energy only for work and that's it, not much motivation to learn after work or on the weekend. In my DevOps job, I write Terraform, plus read javascript and cloudformation yaml. I do wish I could convert my current stuff to AWS CDK, but I don't want to fragment the multiple projects that are using Terraform. (I haven't looked into tf-cdk much at all yet)
- sillysaurusx 5y ago
  Not mine! That was all Scott Bell. It's forked from Lumen: https://github.com/sctb/lumen https://github.com/sctb/lumen But, I did make an interactive tutorial here: https://docs.ycombinator.lol/ https://docs.ycombinator.lol/ If you have any questions about it, I'd be happy to answer. This stuff is pure fun mixed with a shot of professionalism. For what it's worth, as someone with narcolepsy, I relate quite a lot to your chronic pain. (https://twitter.com/theshawwn/status/1392213804684038150 https://twitter.com/theshawwn/status/1392213804684038150) For me, it mostly translated into wandering aimlessly from job to job, since I thought no one would have me. I hope that you find your way -- there's nothing wrong at all with taking it slow and spending years on something that takes others a few months. Everyone is different, and it's all about the fun.
  - mh- 5y ago
    Thanks for sharing this. I didn't expect to read that whole gist but I did and I'm glad I did. Happy for you.
  - markrages 5y ago
    Does "-5e4" really evaluate to "-5000" in this language?
    - sillysaurusx 5y ago
      Hah! Good catch! That readme typo has been in there since Lumen’s inception. It evaluates to -500000, as you’d expect. (Just kidding, it’s -50000. Amusingly, the https://docs.ycombinator.lol https://docs.ycombinator.lol version gets it right, since it has to; every expression is actually evaluated in the browser.)
dheera 5y ago
I don't get it. Where do you define which Lisp-to-Python translator to use? It certainly doesn't seem to know on its own. $ touch foo.l $ python3 >>> import foo ModuleNotFoundError: No module named 'foo'
- petters 5y ago
  I'm guessing something else needs to be imported first.
- sillysaurusx 5y ago
  Close! git clone https://github.com/shawwn/pymen -b ml cd pymen touch foo.l bin/pymen (import foo) It's a bit of a WIP (notice this is on the `ml` branch, not mainline), but it does work. >:) You need nodejs to be installed too, ha.
catlifeonmars 5y ago
NodeJS allows this as well. I think this is pretty much a must-have feature for any serious dynamic language. Edit: A must have for any prolific dynamic language. But now I’m not sure that’s true, because even though it apparently works in Python, it’s certainly not widely used. In NodeJS this feature is used quite heavily for typescript, coffeescript (etcetera) interop.
- dragonwriter 5y ago
  > But now I’m not sure that’s true, because even though it apparently works in Python, it’s certainly not widely used. I mean, it is used—even in the standard library—but often for alternative packaging (e.g., loading python modules in zip files) rather than alternative languages. It may be used less prominently than in Node, but it definitely is used for a variety of things.
- aasasd 5y ago
  It's pretty much the exact feature that's behind the saying that ‘to parse Perl you must have the Perl interpreter’. Because Perl allows some kind of language/imports handlers, as exhibited by tricks like Lingua::Romana::Perligata.
gilch 5y ago
Hylang also does this. Macropy too.
amelius 5y ago
You mean that you can override the import mechanism, which means that it allows you to do just about anything, including making it work with other languages. That doesn't sound cursed to me, just flexible.
- cobbal 5y ago
  What’s flexible for a dynamic scripting language is often cursed from a static perspective. Knowing what imports resolve to statically can be nice.
andrewaylett 5y ago
Lots of things you can do with Python but probably shouldn't and people typically don't. That's one reason I prefer it to Ruby, or even Node, where monkey-patching or otherwise exposing bad magical behaviours is common and even encouraged -- the power is all there, but the ecosystem encourages you to use it for good, not evil. This sounds very much like the good kind of magic, though.
- andrewaylett 5y ago
  To add to this: bad magic is magic that the user has to be aware of in order to use safely. Good magic is an implementation detail that the user doesn't need to know anything about.
- chucke 5y ago
  I don't even know what that means.
jnwatson 5y ago
See also https://github.com/drathier/stack-overflow-import https://github.com/drathier/stack-overflow-import for stupid Python import tricks. "from stackoverflow import quick_sort will go through the search results of [python] quick sort looking for the largest code block that doesn’t syntax error in the highest voted answer from the highest voted question and return it as a module. If that answer doesn’t have any valid python code, it checks the next highest voted answer for code blocks." I once implemented a custom importer as part of a system where the Python interpreter never touched the filesystem.
- neoncontrails 5y ago
  That stackoverflow library is so gloriously bonkers. Is it useful in practice?
  - jnwatson 5y ago
    No. Perhaps for a hackathon.
    - drathier 5y ago
      Lies!
pjmlp 5y ago
There are other purposes as well. In the first .com wave we used a similar mechanism, just with TCL instead of Python, to ship our scripts encrypted. Only our modified loader, written in C, could handle them.

toomuchtodo 5y ago

A master of the dark arts I presume.

th0ma5 5y ago

JPype does this for importing Java classes.

Fordec 5y ago

Wait, what? > that you create As in existing language supplied eg. Perl, Java, etc., or literally anything? Like bootstrapping your own home made language from scratch?

sillysaurusx 5y ago
Literally anything. 'Tis a homemade homegrown lisp, grown by Scott Bell for several years till I took it all for myself. Nom nom. It starts with reader.l: https://github.com/shawwn/pymen/blob/ml/reader.l https://github.com/shawwn/pymen/blob/ml/reader.l where the raw character stream is turned into a bunch of nested arrays. E.g. (+ 1 2) becomes ["+", 1, 2] Then it's punted over to compiler.l https://github.com/shawwn/pymen/blob/ml/compiler.l https://github.com/shawwn/pymen/blob/ml/compiler.l where it's passed through `expand`, which does a `macroexpand` followed by a `lower`. E.g. (do (do (do (print 'hi)))) becomes ["print", "\"hi\""] Then the final thing is thrown to the compile function, which spits out print("hi") -- the final valid Python code that gets passed into the standard python `exec` function. Works with all the standard python things, like async functions and `with` contexts. Been screwing around with it for a few years.
- Fordec 5y ago
  That's absolutely disgusting. But in a Web Assembly sort of way... I don't know whether to spit on it or give it a medal.
- jazzyjackson 5y ago
  I have a homemade language that's not a lisp, but is lispy in some ways. I've only got to the point where I expand the code to an abstract syntax tree, and deciding how to go from there is the hard part for me right now. It's never crossed my mind to just compile it to valid python code. Thanks for the inspiration!
masklinn 5y ago
> As in existing language supplied eg. Perl, Java, etc., or literally anything? Like bootstrapping your own home made language from scratch? Should work with anything as long as you can ultimately generate Python bytecode (and provide a module object). The import system is not simple, but it's really open.
aasasd 5y ago
If the import system allows your code to run instead of the ‘import’ statement, and to produce the module however you want, then of course you can do whatever: load code from Google or StackOverflow results, if you wish.

bloopernova 5y ago

Can I just say that the introduction to Lisp you made in your README.md is really good! I keep trying to get into Lisp (and JavaScript, and TypeScript, etc etc) but I've been a sysadmin my whole professional life and also a chronic pain sufferer. That translates into mostly having energy only for work and that's it, not much motivation to learn after work or on the weekend. In my DevOps job, I write Terraform, plus read javascript and cloudformation yaml. I do wish I could convert my current stuff to AWS CDK, but I don't want to fragment the multiple projects that are using Terraform. (I haven't looked into tf-cdk much at all yet)

sillysaurusx 5y ago
Not mine! That was all Scott Bell. It's forked from Lumen: https://github.com/sctb/lumen https://github.com/sctb/lumen But, I did make an interactive tutorial here: https://docs.ycombinator.lol/ https://docs.ycombinator.lol/ If you have any questions about it, I'd be happy to answer. This stuff is pure fun mixed with a shot of professionalism. For what it's worth, as someone with narcolepsy, I relate quite a lot to your chronic pain. (https://twitter.com/theshawwn/status/1392213804684038150 https://twitter.com/theshawwn/status/1392213804684038150) For me, it mostly translated into wandering aimlessly from job to job, since I thought no one would have me. I hope that you find your way -- there's nothing wrong at all with taking it slow and spending years on something that takes others a few months. Everyone is different, and it's all about the fun.
- mh- 5y ago
  Thanks for sharing this. I didn't expect to read that whole gist but I did and I'm glad I did. Happy for you.
- markrages 5y ago
  Does "-5e4" really evaluate to "-5000" in this language?
  - sillysaurusx 5y ago
    Hah! Good catch! That readme typo has been in there since Lumen’s inception. It evaluates to -500000, as you’d expect. (Just kidding, it’s -50000. Amusingly, the https://docs.ycombinator.lol https://docs.ycombinator.lol version gets it right, since it has to; every expression is actually evaluated in the browser.)

dheera 5y ago

I don't get it. Where do you define which Lisp-to-Python translator to use? It certainly doesn't seem to know on its own. $ touch foo.l $ python3 >>> import foo ModuleNotFoundError: No module named 'foo'

petters 5y ago
I'm guessing something else needs to be imported first.
sillysaurusx 5y ago
Close! git clone https://github.com/shawwn/pymen -b ml cd pymen touch foo.l bin/pymen (import foo) It's a bit of a WIP (notice this is on the `ml` branch, not mainline), but it does work. >:) You need nodejs to be installed too, ha.

catlifeonmars 5y ago

NodeJS allows this as well. I think this is pretty much a must-have feature for any serious dynamic language. Edit: A must have for any prolific dynamic language. But now I’m not sure that’s true, because even though it apparently works in Python, it’s certainly not widely used. In NodeJS this feature is used quite heavily for typescript, coffeescript (etcetera) interop.

dragonwriter 5y ago
> But now I’m not sure that’s true, because even though it apparently works in Python, it’s certainly not widely used. I mean, it is used—even in the standard library—but often for alternative packaging (e.g., loading python modules in zip files) rather than alternative languages. It may be used less prominently than in Node, but it definitely is used for a variety of things.
aasasd 5y ago
It's pretty much the exact feature that's behind the saying that ‘to parse Perl you must have the Perl interpreter’. Because Perl allows some kind of language/imports handlers, as exhibited by tricks like Lingua::Romana::Perligata.

gilch 5y ago

Hylang also does this. Macropy too.

amelius 5y ago

You mean that you can override the import mechanism, which means that it allows you to do just about anything, including making it work with other languages. That doesn't sound cursed to me, just flexible.

cobbal 5y ago
What’s flexible for a dynamic scripting language is often cursed from a static perspective. Knowing what imports resolve to statically can be nice.

andrewaylett 5y ago

Lots of things you can do with Python but probably shouldn't and people typically don't. That's one reason I prefer it to Ruby, or even Node, where monkey-patching or otherwise exposing bad magical behaviours is common and even encouraged -- the power is all there, but the ecosystem encourages you to use it for good, not evil. This sounds very much like the good kind of magic, though.

andrewaylett 5y ago
To add to this: bad magic is magic that the user has to be aware of in order to use safely. Good magic is an implementation detail that the user doesn't need to know anything about.
chucke 5y ago
I don't even know what that means.

jnwatson 5y ago

See also https://github.com/drathier/stack-overflow-import https://github.com/drathier/stack-overflow-import for stupid Python import tricks. "from stackoverflow import quick_sort will go through the search results of [python] quick sort looking for the largest code block that doesn’t syntax error in the highest voted answer from the highest voted question and return it as a module. If that answer doesn’t have any valid python code, it checks the next highest voted answer for code blocks." I once implemented a custom importer as part of a system where the Python interpreter never touched the filesystem.

neoncontrails 5y ago
That stackoverflow library is so gloriously bonkers. Is it useful in practice?
- jnwatson 5y ago
  No. Perhaps for a hackathon.
  - drathier 5y ago
    Lies!

pjmlp 5y ago

There are other purposes as well. In the first .com wave we used a similar mechanism, just with TCL instead of Python, to ship our scripts encrypted. Only our modified loader, written in C, could handle them.