7 ms·

Show HN: Python library to overload functions based on interpreter version

45 points by ttymck 5y ago

tyingq 5y ago
I wish Python had come with some equivalent to the "BEGIN {}" blocks that Perl and Awk have. Lacking that, there's no way to write a (single file) script where a Python 2 interpreter would just get a "You need Python 3" error instead of a syntax error. Because testing for the interpreter version happens too late...after the syntax error already happened. Perl's "BEGIN {}" block lets you easily test for versions or language features before the script itself is parsed.
- int_19h 5y ago
  There's an interesting extensibility point in Python in form of codecs and the encoding comment: # -*- coding: ... -*- The trick is that encoding names come from an extensible registry of codecs (https://docs.python.org/3/library/codecs.html#codecs.register https://docs.python.org/3/library/codecs.html#codecs.registe...), so you can register your own. And when you look at codec API, you get the raw byte stream as input, and spit out strings - so it's basically as powerful as reader macros. The problem is getting that registration code to run before your script starts. Easy if you have multiple scripts, but it requires .pth hacks if you go for single-file.
- ltbarcly3 5y ago
  You couldn't be more mistaken, most libraries supported both python 2 and 3 for years. Doing what you describe is extremely trivial. If you thought it was a good idea you could write a script that would exec trampoline itself into a different interpreter version entirely. You can also catch a SyntaxError like any other exception.
  - tyingq 5y ago
    Put an f-string into a script and get python2 not to choke on it, or try catching that SyntaxError as you describe...without using eval, futures, or similar. You can't catch something that already happened. Yes, people are able to write backwards compatible scripts... by avoiding some features, or by using more than a single file. What I'm describing was just a desire for a very simple "this script requires python version >= X" error, with the ability to write unfettered python version X code below it, in a single file script.
    - dec0dedab0de 5y ago
      What I'm describing was just a desire for a very simple "this script requires python version >= X" error, with the ability to write unfettered python version X code below it, in a single file script. import sys if sys.version_info.major < 3: raise Exception("this script requires python version >= 3") plus an extra line or two if you're worried about minor versions.
      - tyingq 5y ago
        Now put an f-string (or other new feature that pops a syntax error) below that: print(f"whee") And run it with python2, or a python3 < 3.6.
        dec0dedab0de 5y ago
        ahh, I see now. I would add an edit to my original comment, but it's too late.
      - jaster 5y ago
        > cat test.py import sys if sys.version_info.major < 3: raise Exception("this script requires python version >= 3") print(f"test") > python2 test.py File "test.py", line 5 print(f"test") ^ SyntaxError: invalid syntax Doesn't seem that simple
        int_19h 5y ago
        It needs to also do: from __future__ import absolute_import, division, print_function, unicode_literals This will give you syntax much closer to Python 3 in Python 2.7. The remaining differences can be papered over with function and class helpers. Well, except for metaclasses, but I doubt you'd have need for those in a single-file script.
        deleted 5y ago
        [deleted]
    - 5y ago
- villasv 5y ago
  It doesn’t seem common to write single file libraries in Python; and it doesn’t seem to be too unreasonable to add an entrypoint script for version checks.
  - tyingq 5y ago
    Yes, I didn't mean for libraries, but rather for single file scripts.
- jwilk 5y ago
  Not ideal, but in my code, I intentionally provoke syntax error near the beginning of the file, with a comment explaining what's going on, e.g.: 0_0 # Python >= 3.6 is required I have a collection of such incantations for many Python versions here: https://github.com/jwilk/python-syntax-errors https://github.com/jwilk/python-syntax-errors
  - tyingq 5y ago
    Ah, thanks. That's short and does almost what I want.
  - neoncontrails 5y ago
    That's hilarious. Which PEP made 0_0 syntactically okay? I typed it into an AST visualizer (I'm on mobile) and if I'm not mistaken this is equivalent to `0`. But why?
    - kuratkull 5y ago
      Underscores in Numeric Literals. (PEP 515) https://www.python.org/dev/peps/pep-0515/ https://www.python.org/dev/peps/pep-0515/ Your question about 0_0 is just 00 which is 0. (This is really common in other modern languages. Readable example: 1_000_000 vs 1000000)
- smitty1e 5y ago
  Sure to please ~17% of the people, PEP 638 -- Syntactic Macros => https://www.python.org/dev/peps/pep-0638/ https://www.python.org/dev/peps/pep-0638/ (It's only Standards Track at the moment)
  - tyingq 5y ago
    >Created: 24-Sep-2020 I imagine it would have been more popular than 17% if it were available when v2 -> v3 pain was at the high point.
    - smitty1e 5y ago
      Disagree. The breaking changes then were a tough pill. More would not have been better.
      - tyingq 5y ago
        I wasn't proposing adding yet another change at the time. I'm saying if the capability were already in place, dealing with breaking changes would have been easier. The whole reason this thought popped in my head is that I still have people asking today: "Tried to run your script, and I got a syntax error...is it broken?". When the issue is that the default python on their box is 2.x.
        smitty1e 5y ago
        Excellent point.
- BiteCode_dev 5y ago
  My way to work around this is to make a zipapp that contains several files, hence the entry point can do the checking and import another file that contains the code. But since its a zipapp, it behaves like a single file script: https://docs.python.org/3/library/zipapp.html https://docs.python.org/3/library/zipapp.html Nowaday, I use shiv to produce the zipapp: https://shiv.readthedocs.io/en/latest/ https://shiv.readthedocs.io/en/latest/ It's handy because you can also now embed 3rd party dependencies, even for small quick scripts if you wish.
- kazinator 5y ago
  Awk's BEGIN blocks are not parsed separately from the rest of the program. Awk's Yacc-based implementations (like the original One True Awk and GNU Awk) parse the entire input in a single call to yyparse() before executing anything. Therefore, I suspect it is not possible to write a test in the BEGIN block which avoids a version-dependent syntax error elsewhere in favor of terminating with a graceful error.
  - tyingq 5y ago
    Ah, good catch. They do work in Perl, perhaps in Ruby also.
remram 5y ago
I'm not sure I get the point of this. If you already have to write the code using the equivalent old syntax, and keep maintaining it, why write the code a second time with the new syntax?
- theptip 5y ago
  Old versions get deprecated in libraries, you don’t just cut over to a new way and break your callers on upgrade. If you own all the callers of your api in the same repo, you probably don’t want this, you just replace all the old callers along with the implementation.
  - remram 5y ago
    What I mean is (looking at the example in the README): when `asyncio.run()` was added, `asyncio.get_event_loop().run_until_complete()` didn't stop working. What is the point of making two versions of the functions, when one of them already supports every Python you target?
    - kuratkull 5y ago
      The ecosystem is a bit broken, but the owners want to use the latest stuff. The outcome is obvious.
      - remram 5y ago
        What is the obvious outcome? How are you "using the latest stuff", which here is a syntax shortcut, if you have to write out and maintain the old syntax too?
ltbarcly3 5y ago
If Python had macros this library would have a very different implementation indeed. I think 200LOC and a complex registration/namespace/cache system is a bit much to avoid an if statement around a def for a handful of functions. This is especially true since this just turns the if into a decorator, and imports will still need to be done the old way for compatibility. The cognitive overhead of this is not a good tradeoff.
- ttymck 5y ago
  You're absolutely right! I built this mainly to see if it could be done. I noticed uvicorn's main function uses an if-statement like you described, and I wondered what it would look like as a decorator. My next goal is to see if I can make the resolution "static", i.e. Once all the functions are loaded, avoid the dictionary lookup at runtime, and just assign the appropriate function version as the bare module attribute. It's fun to do things like this in python, simply because you can. But it's also important to point out that there's a lot of things you _shouldn't_ (this may be one of them!). I'm glad to see this sparked some discussion!
- kazinator 5y ago
  If Python simply parsed and evaluated top-level expressions one by one, you could test the value of some version variable and act accordingly. Like for this range of versions, load this file, otherwise load that one. If I think of this problem in a Lisp frame, it does not scream "use macros" at me.
sebst 5y ago
That looks great, maybe some libraries can leverage this to offer a wider range of compatibility -- even if "recent" syntax changes, such like the walrus operator or the match/case statements cannot be covered by this. Btw, I've taken the liberty to post your project to my Python News aggregator here: https://news.python.sc/item/ff8f92de-666e-4913-8aa3-27af90df7eaa https://news.python.sc/item/ff8f92de-666e-4913-8aa3-27af90df...
goodside 5y ago
I don’t want to discourage the author, but this seems like a misguided use of decorators. Maybe there’s some scenario where it’s practical to do version checks only at function boundaries, but why limit yourself that? It’s already easy (and clear, and non-magical) to check sys.version yourself in an if-statement. You can do that inside a def block to modify a function slightly, or outside a def block if you truly need the function definition to change by version. Given that, I don’t see how this library adds enough value to justify it as a dependency. The project should also maybe note that it doesn’t help at all with Python 2 compatibility. A newer Python user might assume it enables freely mixing Python 2 and Python 3 in the same file, but even without reading the code I’m pretty sure the library doesn’t (and couldn’t) support this.

45 points by ttymck 5y ago

I wish Python had come with some equivalent to the "BEGIN {}" blocks that Perl and Awk have. Lacking that, there's no way to write a (single file) script where a Python 2 interpreter would just get a "You need Python 3" error instead of a syntax error. Because testing for the interpreter version happens too late...after the syntax error already happened. Perl's "BEGIN {}" block lets you easily test for versions or language features before the script itself is parsed.

int_19h 5y ago
There's an interesting extensibility point in Python in form of codecs and the encoding comment: # -*- coding: ... -*- The trick is that encoding names come from an extensible registry of codecs (https://docs.python.org/3/library/codecs.html#codecs.register https://docs.python.org/3/library/codecs.html#codecs.registe...), so you can register your own. And when you look at codec API, you get the raw byte stream as input, and spit out strings - so it's basically as powerful as reader macros. The problem is getting that registration code to run before your script starts. Easy if you have multiple scripts, but it requires .pth hacks if you go for single-file.
ltbarcly3 5y ago
You couldn't be more mistaken, most libraries supported both python 2 and 3 for years. Doing what you describe is extremely trivial. If you thought it was a good idea you could write a script that would exec trampoline itself into a different interpreter version entirely. You can also catch a SyntaxError like any other exception.
- tyingq 5y ago
  Put an f-string into a script and get python2 not to choke on it, or try catching that SyntaxError as you describe...without using eval, futures, or similar. You can't catch something that already happened. Yes, people are able to write backwards compatible scripts... by avoiding some features, or by using more than a single file. What I'm describing was just a desire for a very simple "this script requires python version >= X" error, with the ability to write unfettered python version X code below it, in a single file script.
  - dec0dedab0de 5y ago
    What I'm describing was just a desire for a very simple "this script requires python version >= X" error, with the ability to write unfettered python version X code below it, in a single file script. import sys if sys.version_info.major < 3: raise Exception("this script requires python version >= 3") plus an extra line or two if you're worried about minor versions.
    - tyingq 5y ago
      Now put an f-string (or other new feature that pops a syntax error) below that: print(f"whee") And run it with python2, or a python3 < 3.6.
      - dec0dedab0de 5y ago
        ahh, I see now. I would add an edit to my original comment, but it's too late.
    - jaster 5y ago
      > cat test.py import sys if sys.version_info.major < 3: raise Exception("this script requires python version >= 3") print(f"test") > python2 test.py File "test.py", line 5 print(f"test") ^ SyntaxError: invalid syntax Doesn't seem that simple
      - int_19h 5y ago
        It needs to also do: from __future__ import absolute_import, division, print_function, unicode_literals This will give you syntax much closer to Python 3 in Python 2.7. The remaining differences can be papered over with function and class helpers. Well, except for metaclasses, but I doubt you'd have need for those in a single-file script.
      - deleted 5y ago
        [deleted]
  - 5y ago
villasv 5y ago
It doesn’t seem common to write single file libraries in Python; and it doesn’t seem to be too unreasonable to add an entrypoint script for version checks.
- tyingq 5y ago
  Yes, I didn't mean for libraries, but rather for single file scripts.
jwilk 5y ago
Not ideal, but in my code, I intentionally provoke syntax error near the beginning of the file, with a comment explaining what's going on, e.g.: 0_0 # Python >= 3.6 is required I have a collection of such incantations for many Python versions here: https://github.com/jwilk/python-syntax-errors https://github.com/jwilk/python-syntax-errors
- tyingq 5y ago
  Ah, thanks. That's short and does almost what I want.
- neoncontrails 5y ago
  That's hilarious. Which PEP made 0_0 syntactically okay? I typed it into an AST visualizer (I'm on mobile) and if I'm not mistaken this is equivalent to `0`. But why?
  - kuratkull 5y ago
    Underscores in Numeric Literals. (PEP 515) https://www.python.org/dev/peps/pep-0515/ https://www.python.org/dev/peps/pep-0515/ Your question about 0_0 is just 00 which is 0. (This is really common in other modern languages. Readable example: 1_000_000 vs 1000000)
smitty1e 5y ago
Sure to please ~17% of the people, PEP 638 -- Syntactic Macros => https://www.python.org/dev/peps/pep-0638/ https://www.python.org/dev/peps/pep-0638/ (It's only Standards Track at the moment)
- tyingq 5y ago
  >Created: 24-Sep-2020 I imagine it would have been more popular than 17% if it were available when v2 -> v3 pain was at the high point.
  - smitty1e 5y ago
    Disagree. The breaking changes then were a tough pill. More would not have been better.
    - tyingq 5y ago
      I wasn't proposing adding yet another change at the time. I'm saying if the capability were already in place, dealing with breaking changes would have been easier. The whole reason this thought popped in my head is that I still have people asking today: "Tried to run your script, and I got a syntax error...is it broken?". When the issue is that the default python on their box is 2.x.
      - smitty1e 5y ago
        Excellent point.
BiteCode_dev 5y ago
My way to work around this is to make a zipapp that contains several files, hence the entry point can do the checking and import another file that contains the code. But since its a zipapp, it behaves like a single file script: https://docs.python.org/3/library/zipapp.html https://docs.python.org/3/library/zipapp.html Nowaday, I use shiv to produce the zipapp: https://shiv.readthedocs.io/en/latest/ https://shiv.readthedocs.io/en/latest/ It's handy because you can also now embed 3rd party dependencies, even for small quick scripts if you wish.
kazinator 5y ago
Awk's BEGIN blocks are not parsed separately from the rest of the program. Awk's Yacc-based implementations (like the original One True Awk and GNU Awk) parse the entire input in a single call to yyparse() before executing anything. Therefore, I suspect it is not possible to write a test in the BEGIN block which avoids a version-dependent syntax error elsewhere in favor of terminating with a graceful error.
- tyingq 5y ago
  Ah, good catch. They do work in Perl, perhaps in Ruby also.

remram 5y ago

I'm not sure I get the point of this. If you already have to write the code using the equivalent old syntax, and keep maintaining it, why write the code a second time with the new syntax?

theptip 5y ago
Old versions get deprecated in libraries, you don’t just cut over to a new way and break your callers on upgrade. If you own all the callers of your api in the same repo, you probably don’t want this, you just replace all the old callers along with the implementation.
- remram 5y ago
  What I mean is (looking at the example in the README): when `asyncio.run()` was added, `asyncio.get_event_loop().run_until_complete()` didn't stop working. What is the point of making two versions of the functions, when one of them already supports every Python you target?
  - kuratkull 5y ago
    The ecosystem is a bit broken, but the owners want to use the latest stuff. The outcome is obvious.
    - remram 5y ago
      What is the obvious outcome? How are you "using the latest stuff", which here is a syntax shortcut, if you have to write out and maintain the old syntax too?

ltbarcly3 5y ago

If Python had macros this library would have a very different implementation indeed. I think 200LOC and a complex registration/namespace/cache system is a bit much to avoid an if statement around a def for a handful of functions. This is especially true since this just turns the if into a decorator, and imports will still need to be done the old way for compatibility. The cognitive overhead of this is not a good tradeoff.

ttymck 5y ago
You're absolutely right! I built this mainly to see if it could be done. I noticed uvicorn's main function uses an if-statement like you described, and I wondered what it would look like as a decorator. My next goal is to see if I can make the resolution "static", i.e. Once all the functions are loaded, avoid the dictionary lookup at runtime, and just assign the appropriate function version as the bare module attribute. It's fun to do things like this in python, simply because you can. But it's also important to point out that there's a lot of things you _shouldn't_ (this may be one of them!). I'm glad to see this sparked some discussion!
kazinator 5y ago
If Python simply parsed and evaluated top-level expressions one by one, you could test the value of some version variable and act accordingly. Like for this range of versions, load this file, otherwise load that one. If I think of this problem in a Lisp frame, it does not scream "use macros" at me.

sebst 5y ago

That looks great, maybe some libraries can leverage this to offer a wider range of compatibility -- even if "recent" syntax changes, such like the walrus operator or the match/case statements cannot be covered by this. Btw, I've taken the liberty to post your project to my Python News aggregator here: https://news.python.sc/item/ff8f92de-666e-4913-8aa3-27af90df7eaa https://news.python.sc/item/ff8f92de-666e-4913-8aa3-27af90df...

goodside 5y ago

I don’t want to discourage the author, but this seems like a misguided use of decorators. Maybe there’s some scenario where it’s practical to do version checks only at function boundaries, but why limit yourself that? It’s already easy (and clear, and non-magical) to check sys.version yourself in an if-statement. You can do that inside a def block to modify a function slightly, or outside a def block if you truly need the function definition to change by version. Given that, I don’t see how this library adds enough value to justify it as a dependency. The project should also maybe note that it doesn’t help at all with Python 2 compatibility. A newer Python user might assume it enables freely mixing Python 2 and Python 3 in the same file, but even without reading the code I’m pretty sure the library doesn’t (and couldn’t) support this.