16 ms·

What is the core of the Python programming language?

232 points by BerislavLopac 6y ago

shakna 6y ago
When thinking about a minimal subset of Python to be truly useful, my mind jumps to MicroPython [0]. Which can be compiled to run on a standard platform, instead of the more common baremetal situation. MicroPython does feature a REPL - though you can disable it so you can control UART in a more expected way. A surprising amount of plain Python programs can run with the stripped down set. Even though you won't find any of the referenced functions in sys. However, locals does work, as does builtins. You also get access to pip and a few other unexpected things like that. [0] https://micropython.org https://micropython.org
- dukoid 6y ago
  MicroPython is addressing a different problem though: Minimalism wrt. libraries -- opposed to being more "static" in some aspects... In contrast, the author seems to be looking for an "acceptable" core that would still run most libraries -- but omitting some of the more dynamic aspects that are hard to emulate in WebASM....
  - pansa2 6y ago
    Nonetheless, MicroPython is more static than CPython. Of the three examples of dynamism in the article (`locals`, `sys.settrace` and `sys._getframe`), MicroPython only supports `locals`.
    - stinos 6y ago
      MicroPython does support sys.settrace
      - pansa2 6y ago
        Ah, so it does. I tested this on https://micropython.org/unicorn/ https://micropython.org/unicorn/ and didn't notice that it's running quite an old version.
      - shakna 6y ago
        However, it isn't documented [0]. And the sys.settrace tests are allowed to fail: [1] PY_SYS_SETTRACE is a compile time option. It won't necessarily be a part of the build. I don't think it is by default on any of the ports at the moment, though I could be wrong. Based on that, I'd call it an "unstable" part of the implementation at the moment. [0] https://docs.micropython.org/en/latest/library/sys.html https://docs.micropython.org/en/latest/library/sys.html [1] https://github.com/micropython/micropython/blob/master/tests/misc/sys_settrace_generator.py https://github.com/micropython/micropython/blob/master/tests...
        stinos 6y ago
        Fair point about the documentation, but I doubt it's allowed to fail: the 'except AttributeError' is only meant to skip the complete test for builds without support for it, builds which do support it will run the test and will result in a failed build status if the test fails. And yes it's a compile-time option, just like a 100 other features, that's just how it's done in MicroPython and doesn't necessarily equate to 'unstable'.
        shakna 6y ago
        Being a compile time option isn't what contributed to the unstable part there - MicroPython is modular. That's part of the benefit of the system. That I couldn't find it enabled on any of the default builds is what made me conclude it might be unstable, especially as it is undocumented.
        stinos 6y ago
        The reason for it not being enabled by default is probably twofold: code size increase/performance decrease, and it's not a feature which is used often especially not for the majority of targets.
- ntoll 6y ago
  MicroPython is an amazing achievement, and Damien has done fantastic work growing the project.
josh_fyi 6y ago
RPython is relevant. He does mention PyPy, which uses RPython, but I am surprised that he does not mention RPython itself -- a restricted subset of the Python language used for similar purposes to the ones that he describes.
- albertzeyer 6y ago
  I was interested a while ago (2013...) in this question, whether RPython might be an interesting general language by itself. I asked about that in the pypy-dev mailing list. Unfortunately I cannot find a link to that thread, but here are some answers (from Armin Rigo and Carl Friedrich Bolz): > we (= the general PyPy developers) are not really interested in this direction [...]. There is notably the fact that RPython is not meant as a general language used to produce stand-alone C libraries. It's certainly possible to make one in theory, but cumbersome. > One of the reasons why the RPython project does not care much about this use case anymore is that we had a way to make CPython extensions. It was called the "extension compiler". It turned out to be a) extremely annoying to implement b) not very useful in practice. Thus a totally frustrating experience overall. > The reasons for slowness was mainly: when compiling RPython code to be a C extension module, reference counting is used to reclaim garbage. That is extremely slow. Therefore the modules never got the performance anybody expected. > When people look at RPython, an obvious feature is that it is syntactically identical to Python. "RPython must be an easy language, given that it has got the syntax of Python, which is easy". This is a common misconception. In fact, pleasing the automatic type inference process can be difficult. It requires the programmer keeping in his head the global types of his whole program, and carefully writing code according to these implicit types. The process is much harder for newcomers, which don't have any written-down example to learn how to manipulate the types --- precisely because they are implicit. > So this is the reason we are recommending against RPython now (and for many years now). Anybody who thinks RPython is as easy as Python is someone who will happily try out RPython and be burned alive by it. Edit: I found one link to the thread: https://mail.python.org/pipermail/pypy-dev/2013-June/011503.html https://mail.python.org/pipermail/pypy-dev/2013-June/011503.... Edit: I also found a (closed, opinion based, obviously...) related StackOverflow question: https://stackoverflow.com/questions/17134479/why-do-people-say-that-rpython-is-an-unpleasant-language-to-program-in/ https://stackoverflow.com/questions/17134479/why-do-people-s... From the FAQ (https://rpython.readthedocs.io/en/latest/faq.html https://rpython.readthedocs.io/en/latest/faq.html): > First and foremost, RPython is a language designed for writing interpreters. It is a restricted subset of Python. If your program is not an interpreter but tries to do “real things”, like use any part of the standard Python library or any 3rd-party library, then it is not RPython to start with. You should only look at RPython if you try to write your own interpreter. A related project (probably abandoned since a long time): https://code.google.com/archive/p/rpythonic/ https://code.google.com/archive/p/rpythonic/
pcr910303 6y ago
Well, I just wanted to say that it is definitely possible for native-compiled languages to do things like that, mostly by just bundling up the compiler itself. (Not sure about wasm, but looks like the article is a bit more generic than just about wasm.) One great example is SBCL, a Common Lisp implementation which only has an compiler, but provides REPL, `eval`, and `compile` functions as per the CL spec. For the question about the core of Python, I would consider that everything not tied to the external environment as Python. So `locals`, `eval`, `compile` must be provided to be called as a full-fledged Python. On the other hand, I won't include all of the stdlib functions in the definition of Python, like handling the file system or environment variables. Think about node and the web browser - both are JS runtimes and they share the core language (like `this` or `with`, which provides similar capabilities of `locals`, and `eval`), but provide different stdlibs based on the environment. (Node provides web servers, file server interactions, etc... and the web browser provides APIs to manipulate the DOM, attach event handlers, etc...)
- phoe-krk 6y ago
  > which only has an compiler Nitpick: SBCL ships with an interpreter, but yes, it is turned off by default in favor of the native code compiler. See http://www.sbcl.org/manual/#Interpreter http://www.sbcl.org/manual/#Interpreter for details.
blickentwapft 6y ago
MicroPython seems to have done a fairly good job of deciding what’s important in a python subset.
david_draco 6y ago
It's Python if it can run popular libraries, such as scipy. That's already an achievement to get working outside CPython.
kilon 6y ago
A possible approach is to use Cython to convert Python code to C code and then compile that code to WebAssembly. I would not be surprised if there is a tool for that already
- uryga 6y ago
  from my experience digging through Cython output: unless you type-annotate EVERYTHING C-style, Cython just uses the CPython C-API (PyObject_New, PY_INCREF, etc). so you'd have to ship all of CPython anyway... as mentioned elsewhere in the thread, that's too big for most websites
dragonwriter 6y ago
> But when thinking about the daunting task of creating a new implementation of Python, my brain also began asking the question of what exactly is Python? The Python documentation is pretty clear about (1) What is the Python language, (2) What is the Python stdlib (“Python” is either the first or the combination of the two), and (3) what are CPython implementation quirks. If you have all of #1, you have a Python implementation, #1 + #2 is, I guess, a Python distribution. And, yes, most of the things the article questions are part of #1, and definitively part of the essentials of the language.
moonchild 6y ago
While juggling ideas for a web replacement a while back, I came across python2wasm[1], part of the pure python compiler infrastructure. Not complete, but interesting and well worth taking a look at. 1. https://bitbucket.org/windel/ppci/src/default/ppci/lang/python/python2wasm.py https://bitbucket.org/windel/ppci/src/default/ppci/lang/pyth...
raymondh 6y ago
I suspect that Python's use in education would be an early casualty if the interactive REPL were removed.
meredydd 6y ago
As a maintainer of the Skulpt Python-to-JS compiler (https://skulpt.org https://skulpt.org), I can reasonably attempt to answer this question. Skulpt compiles Python to JS in one shot, so it's not a million miles away from the architecture Brett envisages. (I gave a lightning talk about it here: https://anvil.works/blog/pycon-talk https://anvil.works/blog/pycon-talk). Things I've observed: - "Ordinary code" requires remarkably little fidelity. Python has a very strong "simplicity culture", so most code avoids being clever. If you're creating an environment for users to program against (as we do in Anvil), it's astonishing how little of Python you need to support. - "Be able to use popular Python libraries" is really, really hard. The ordinary user doesn't use all the esoteric stuff Brett mentions, but libraries do, left and right. Even porting PyPy's `datetime` implementation to Skulpt required filling out a lot of odd little corners of the object implementation. We still have a pretty crummy subset of `unittest` because it's eye-achingly dynamic - for example, it really puts metaclasses through their paces. (Metaclasses are a perfect example of a powerful feature that really depends on details that descend directly from CPython: The API exposes the fact that variable scopes are literally `dict` objects, that classes are created dynamically at runtime, etc.) - Finally (and most unfortunately): A lot of the strength of Python is in its ecosystem, and a lot of that ecosystem relies on native code. (Again, much more than you'd think. When porting stdlib modules from PyPy to Skulpt, we have often been disappointed by how many of them use native code - even in PyPy!) The Python-to-C interface is really, really CPython-shaped. Anyone who wants to be compatible with numpy, scipy, etc, needs to literally emulate CPython's data structures, and then translate them to whatever faster/simpler thing they use internally. This emulation overhead is the main reason PyPy is slower than CPython on some workloads. (In Skulpt, we haven't even attempted this. It doesn't bother us much in Anvil, of course, as the server-side CPython is just a function call away, but for Brett's use case I'm guessing it would really hurt. One day we might try, with a JS-to-wasm bridge and doing the emulation in wasm, but it would be a hell of ajob.)
- w0utert 6y ago
  >> - Finally (and most unfortunately): A lot of the strength of Python is in its ecosystem, and a lot of that ecosystem relies on native code. I was thinking the same thing... One can wonder how much value there is left if you build an alternative Python interpreter that can e.g. run in the browser, but cannot support C extensions. I mean I like Python for what it is, but once you stray from the common/intended use cases, e.g. trying to embed the interpreter in a C++ program, skeletons fall out of closets around every corner. For all it's merits, CPython feels like a house of cards, burdened by a legacy of bad engineering decisions it cannot get rid of anymore. It made me realize it's better to stop fighting it, use CPython as-is (which is great), and use other scripting languages for embedding, live-coding, web, etc. I very much prefer Python over JavaScript for example, but even then it wouldn't cross my mind to use some kind of Python-to-JS implementation for web frontend code, honestly.
qwerty456127 6y ago
Give up (at least temporarily) everything listed - eval(), locals(), REPL etc and it still is going to be extremely useful for, arguably, most of the real use cases. E.g. I personally haven't ever needed any of these. REPL happens to be handy occasionally to quick-check something but I can hardly imagine it being used in production, it only seems useful in development time and for education. eval() seems a big red flag everybody should avoid at all cost. Please do your best to support all the latest syntax though. Lack of support for some standard functions particularly hard to implement is Ok, lack of support for syntactic elements (e.g. type hints) is worse and, usually, easier to fix.
- ericvsmith 6y ago
  namedtuple and dataclasses both use eval(), so if you want to give up eval(), you'd need to give them up, too.
  - VectorLock 6y ago
    Interesting they use eval(), I wonder if their implementations could be changed to not use eval(). I guess in the end you could just use actual classes for these cases, but I sure do like namedtuple and dataclasses.
    - ericvsmith 6y ago
      My bad: it's exec(), not eval(). And I wrote dataclasses, so you'd think I'd know better! The issue is that it's dynamically generating methods. There has to be a way to get dynamically created code into python for this use case.
- VectorLock 6y ago
  REPL-centric development is something that has caught on a lot lately and is something that I find myself gravitating towards more and more. For 'production' the eval() might be able to be cast off, but for development the REPL is indispensable - if not a first class requirement.
fifticon 6y ago
It saddens me, that one of Pythons strengths - its C extensions - appear as a weakness in this context. The culprit is, that we perceive python as 'what you can DO through python' - which is a very reasonable perspective. But precisely python's useful boosting through C extensions, also anchors it to its friends. So, it's the old conundrum of what goes into a language core, versus what api library environment it lives in.
- throwaway894345 6y ago
  As I point out elsewhere it’s possible to have both performance and C-extensions, but the extension interface needs to be minimized so Python implementations have some breathing room to make optimizations—today almost any optimization would break the expansive extension interface. There are projects (like this: https://github.com/pyhandle/hpy https://github.com/pyhandle/hpy) which aim to make a smaller interface, but the community must adopt this overwhelmingly before the old interface could be deprecated and work could begin to optimize any Python implementation.
- physicsyogi 6y ago
  The C extensions are partly why Python reigns in the machine learning and data science world.
- gnulinux 6y ago
  This is why I still use Python in performance critical code (but not so performance critical that you need to squeeze each CPU cycle). It's simply too easy to write performance critical functions in C and ffi call it in python. Of course, compared to many other language it's still slow "in general" but for your give particular task it's pretty easy to make it very fast. Just write the bottlenecking tight loop in C and everything else can be in python.
  - carlmr 6y ago
    Rust with it's Cargo package manager, unicode string handling and strong type system is really great for Python speed-ups: https://developers.redhat.com/blog/2017/11/16/speed-python-using-rust/ https://developers.redhat.com/blog/2017/11/16/speed-python-u...
mastrsushi 6y ago
PyObject graphs that represent dynamic variables through pseudo OOP legacy C code. Check it out, it's a total nightmare of downcasting pointers. It amazes me in the 29+ years it hasn't been replaced by a C++ alternative that could retain compatability. I think AST interpreted languages are becoming something of the past. Yes I'm aware there's bytecode in CPython. I like the idea of IL's as target platforms rather than 100 leaky interpreters that you have to manually bind your C code with if you want support ti support any one of them.
- jashmatthews 6y ago
  Truffle is built around AST interpretation so that’s not going to happen any time soon.
- jnwatson 6y ago
  This is extraordinarily normal C code. I’ve worked with several large code bases that look like CPython code. Heck, a lot of Linux looks like that. It really speaks more to the failure of C++ than anything about CPython. C++ is so much more than C with objects, and all that extra is what folks don’t want.
  - mastrsushi 6y ago
    I never understood the argument of C++ being too big of a language. Code bases can enforce restrictions on what can be used.
    - stinos 6y ago
      Can you give a practical example of how a codebase can prevent restrict typical 'here's a bazooka to shoot yourself in the foot' C++ code?
  - gjvc 6y ago
    > This is extraordinarily normal C code. I’ve worked with several large code bases that look like CPython code. Heck, a lot of Linux looks like that. Manually-implemented dispatch / vtbls you mean?
gjvc 6y ago
https://docs.python.org/3/reference/datamodel.html https://docs.python.org/3/reference/datamodel.html https://docs.python.org/3/reference/executionmodel.html https://docs.python.org/3/reference/executionmodel.html
tannhaeuser 6y ago
As decidely a non-insider, I'll tell you that, for me, Python's core strength is to be a beginner's all-purpose programming language, and not very much else. I had a really good experience in teaching Python to 8-10 year olds; actually, the kids were able to work out most things for themselves as I couldn't help all that much really, since I'm not much into Python. And in fact, Python was forked or at least inspired from/by ABC which was specifically designed to be a less idiosyncratic PL. Fun fact: ABC was developed by Steven Pemberton (among others) who'd later led many W3C efforts, including the failed XHTML2 effort. I didn't get around to asking how he went from ABC to XForms (which is as idiosyncratic as it gets) when I saw him as a speaker a couple years ago ;) but anyway he's always been an inspiration.
- quietbritishjim 6y ago
  That's a good use of Python, and if that's all it's useful for for you, that's fine. I'm not sure whether or not you're trying to argue that that is the only reaosnable use of Python? To me, it sound like you are. If so, that is contradicted by the many people who have successfully used Python for many other purposes. That includes run-once scripts (where you might have used Perl in the past) where run time speed is not as important as speed of writing the code, but also certain types of code running in production. Like any other tool, you have to be aware of its strengths and weaknesses: maybe it's not the best choice for a code base running into 10s of KLOC or more, or for doing CPU-intensive work that can't be vectorised (e.g. multiplying a few large matrices in numpy is fine, whereas multiplying an extremely large number of very small matrices is maybe not fine). The fact it has weaknesses doesn't mean it should be written off for all tasks, even those where the weaknesses aren't so important, especially when it has strengths too (e.g. if Python would do for a particular task then using C++ blindly because it's "better" would be wasting developer time for no reason). It's often used for glue code where all the work is actually done in C libraries like deep learning, and it works very well for that: if 0.1% of your runtime is spent in the top-level glue instead of 0.01%, that's probably not a problem. Also, this discussion is fairly off topic to begin with. The article really isn't about the strengths and weaknesses of the language. It's about what consitutes the language versus an implementation of it (CPython).
1337shadow 6y ago
Or just use transcrypt to transcode your Python into native JavaScript. http://www.transcrypt.org/ http://www.transcrypt.org/
- jennasys 6y ago
  I came here to say this. Transcrypt takes the approach that there are already many good JavaScript libraries that exist for web development, and rather than try and replace those, it embraces them. What it allows you to do though is to write code that uses those libraries in Python, and then transpile that into JS for deployment. It has allowed me to write React + MaterialUI apps using 99% Python, without taking a performance hit in the browser.
deleted 6y ago
[deleted]
lazyloop 6y ago
Python for WebAssembly already exists and runs fine on iOS. https://holzschu.github.io/a-Shell_iOS/ https://holzschu.github.io/a-Shell_iOS/
- ascotan 6y ago
  Pretty neat. I didn't know that the app store now allows stuff like this. I thought they had banned programming interfaces.
  - rcarmo 6y ago
    Pythonista is still there.
pansa2 6y ago
> It's no secret that I want a Python implementation for WebAssembly. [...] with the fact that both iOS and Android support running JavaScript as part of an app it would also get Python on to mobile. WebAssembly is not JavaScript. So I assume iOS and Android also support running WebAssembly as part of an app?
- giancarlostoro 6y ago
  WebAssembly is compiled and executed by JavaScript though. You can create a runtime with no JS on the other hand probably for mobile. The beauty of WASM is eventually you could have it on all platforms as a way to make apps and expose API libraries to said runtime and now anyone can make apps in any language to run on WASM. So long as said language can be compiled to WASM. You also need JS to interop back from WASM.
bpyne 6y ago
Good question about the value of a REPL to Python. My first experience with a REPL was the SQL*Plus tool from Oracle. I don't remember it being called a REPL. But, as I read about Common Lisp's REPL, I realized that I had been using one. Early in my experience with Oracle's DBMS, I found the REPL handy to work out ideas. But a REPL is essentially a single line editor from which you can evaluate expressions/statements. As my ideas grew, the REPL became more of a hindrance. Having a full screen editor became more necessary. Nowadays I work out all ideas within an IDE, firing up the REPL only for quick data checks. So my answer is that a REPL is helpful but not necessary.
- giancarlostoro 6y ago
  > But a REPL is essentially a single line editor from which you can evaluate expressions/statements. It is not always single line, look at IDLE or even Firefox supports multiline REPL capabilities although it still needs some improvements (like not rerunning var declaration statements).
  - bpyne 6y ago
    Thanks. Multiline certainly raises the value of a REPL.
    - giancarlostoro 6y ago
      I think so too, though I guess single line makes more sense to an extent, there's a lot more that could be done with REPLs, like I'm surprised you can't just click on a previously submitted segment of code to be auto copied in the input text field (in the case of GUI based REPLs).
ascotan 6y ago
I'm no expert here but it seems like porting a duck-typed languages like python to webassembly is hard feat. Arbitrary evaluation of source code at runtime, seems to me, exactly like the type of thing webassembly wants prevent. This limits a lot of the flexibility of a language like python (repl, eval, etc) I feel like CPython isn't so much the problem as third party libraries that people have come to rely on that, under the hood, rely on architecture dependent code (numpy, scipy, psycopg, etc, etc). I would imaging that porting these is impossible. I think a repl is fundamental to python more than any other language. I wouldn't say that about something else, but python, yes.
pansa2 6y ago
> It might make sense to develop a compiler that translates Python code directly to WebAssembly and sacrifice some compatibility for performance. This seems to be the author's preferred approach, but I'm not sure it would provide significant performance improvements over simply compiling the existing Python interpreter to WebAssembly. Compiling dynamically-typed Python to statically-typed WebAssembly would be similar to how Nuitka [0] compiles Python to C. This compiled code may be a little faster than interpreted Python, but it's still going to be quite slow - much slower in a browser than JIT-compiled JavaScript. AFAIK the ideal way to maximize the performance of Python in the browser would be to JIT-compile Python code to WebAssembly. Are WebAssembly implementations able to support that? [0] http://nuitka.net/ http://nuitka.net/
acka 6y ago
> It's no secret that I want a Python implementation for WebAssembly. It would not only get Python into the browser, but with the fact that both iOS and Android support running JavaScript as part of an app it would also get Python on to mobile. That all excites me. If the main goal is to be able to run Python code in a browser, another option is Brython[0]. According to the FAQ[1] it should work in all modern browsers, including mobile. [0] https://brython.info/ https://brython.info/ [1] https://brython.info/static_doc/en/faq.html https://brython.info/static_doc/en/faq.html
- chx 6y ago
  Here's a great summary of the existing options https://stackoverflow.com/a/58684358/308851 https://stackoverflow.com/a/58684358/308851
phsilva 6y ago
CPython C API is one the main culprit in making Python hard to implement and optimize. As some commenters said, I don't think Python without access to C/C++ projects would be worth right now giving how sloe CPython is. About 100% of machine learning, data science and general science Python code relies on C/C++ extensions. Current CPython API and the need to keep compatibility reduces the capabilities of improving CPython. Look at PEP-620 (https://www.python.org/dev/peps/pep-0620/ https://www.python.org/dev/peps/pep-0620/) for more info on how python developers are addressing this.
- coliveira 6y ago
  > science Python code relies on C/C++ extensions Don't forget Fortran. A good portion of numpy is written in Fortran.
  - TheRealKing 6y ago
    scipy, as far as I am aware. but that not change your stated fact on the role of Fortran in Python.
robterrell 6y ago
Boo is a Python-like language for .NET that took a swing at answering this question: https://boo-language.github.io https://boo-language.github.io Boo's answer is "just syntax" -- libraries are .NET assemblies.
- CopyZero 6y ago
  Oh Man, I totally forgot about Boo. I remember looking at it years ago. This is just the thing I didn't know I was looking for. Thanks
dehrmann 6y ago
Some of the confusion is that people often blur languages and runtimes together. I started work on and then abandoned work on a Node implementation in Java backed by Nashorn. These distinctions become really clear when there's a mismatch like that. That, and some of the cpython issues would have been handled by pragmas in other languages (__slots__, for one), but got baked into the runtime. The unwritten zen of python is "bolt it on."
digitmg 6y ago
Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work! <a href="https://360digitmg.com/india/data-science-using-python-and-r-programming-in-guntur">data https://360digitmg.com/india/data-science-using-python-and-r... science course in guntur</a>

232 points by BerislavLopac 6y ago

When thinking about a minimal subset of Python to be truly useful, my mind jumps to MicroPython [0]. Which can be compiled to run on a standard platform, instead of the more common baremetal situation. MicroPython does feature a REPL - though you can disable it so you can control UART in a more expected way. A surprising amount of plain Python programs can run with the stripped down set. Even though you won't find any of the referenced functions in sys. However, locals does work, as does builtins. You also get access to pip and a few other unexpected things like that. [0] https://micropython.org https://micropython.org

dukoid 6y ago
MicroPython is addressing a different problem though: Minimalism wrt. libraries -- opposed to being more "static" in some aspects... In contrast, the author seems to be looking for an "acceptable" core that would still run most libraries -- but omitting some of the more dynamic aspects that are hard to emulate in WebASM....
- pansa2 6y ago
  Nonetheless, MicroPython is more static than CPython. Of the three examples of dynamism in the article (`locals`, `sys.settrace` and `sys._getframe`), MicroPython only supports `locals`.
  - stinos 6y ago
    MicroPython does support sys.settrace
    - pansa2 6y ago
      Ah, so it does. I tested this on https://micropython.org/unicorn/ https://micropython.org/unicorn/ and didn't notice that it's running quite an old version.
    - shakna 6y ago
      However, it isn't documented [0]. And the sys.settrace tests are allowed to fail: [1] PY_SYS_SETTRACE is a compile time option. It won't necessarily be a part of the build. I don't think it is by default on any of the ports at the moment, though I could be wrong. Based on that, I'd call it an "unstable" part of the implementation at the moment. [0] https://docs.micropython.org/en/latest/library/sys.html https://docs.micropython.org/en/latest/library/sys.html [1] https://github.com/micropython/micropython/blob/master/tests/misc/sys_settrace_generator.py https://github.com/micropython/micropython/blob/master/tests...
      - stinos 6y ago
        Fair point about the documentation, but I doubt it's allowed to fail: the 'except AttributeError' is only meant to skip the complete test for builds without support for it, builds which do support it will run the test and will result in a failed build status if the test fails. And yes it's a compile-time option, just like a 100 other features, that's just how it's done in MicroPython and doesn't necessarily equate to 'unstable'.
        shakna 6y ago
        Being a compile time option isn't what contributed to the unstable part there - MicroPython is modular. That's part of the benefit of the system. That I couldn't find it enabled on any of the default builds is what made me conclude it might be unstable, especially as it is undocumented.
        stinos 6y ago
        The reason for it not being enabled by default is probably twofold: code size increase/performance decrease, and it's not a feature which is used often especially not for the majority of targets.
ntoll 6y ago
MicroPython is an amazing achievement, and Damien has done fantastic work growing the project.

josh_fyi 6y ago

RPython is relevant. He does mention PyPy, which uses RPython, but I am surprised that he does not mention RPython itself -- a restricted subset of the Python language used for similar purposes to the ones that he describes.

albertzeyer 6y ago
I was interested a while ago (2013...) in this question, whether RPython might be an interesting general language by itself. I asked about that in the pypy-dev mailing list. Unfortunately I cannot find a link to that thread, but here are some answers (from Armin Rigo and Carl Friedrich Bolz): > we (= the general PyPy developers) are not really interested in this direction [...]. There is notably the fact that RPython is not meant as a general language used to produce stand-alone C libraries. It's certainly possible to make one in theory, but cumbersome. > One of the reasons why the RPython project does not care much about this use case anymore is that we had a way to make CPython extensions. It was called the "extension compiler". It turned out to be a) extremely annoying to implement b) not very useful in practice. Thus a totally frustrating experience overall. > The reasons for slowness was mainly: when compiling RPython code to be a C extension module, reference counting is used to reclaim garbage. That is extremely slow. Therefore the modules never got the performance anybody expected. > When people look at RPython, an obvious feature is that it is syntactically identical to Python. "RPython must be an easy language, given that it has got the syntax of Python, which is easy". This is a common misconception. In fact, pleasing the automatic type inference process can be difficult. It requires the programmer keeping in his head the global types of his whole program, and carefully writing code according to these implicit types. The process is much harder for newcomers, which don't have any written-down example to learn how to manipulate the types --- precisely because they are implicit. > So this is the reason we are recommending against RPython now (and for many years now). Anybody who thinks RPython is as easy as Python is someone who will happily try out RPython and be burned alive by it. Edit: I found one link to the thread: https://mail.python.org/pipermail/pypy-dev/2013-June/011503.html https://mail.python.org/pipermail/pypy-dev/2013-June/011503.... Edit: I also found a (closed, opinion based, obviously...) related StackOverflow question: https://stackoverflow.com/questions/17134479/why-do-people-say-that-rpython-is-an-unpleasant-language-to-program-in/ https://stackoverflow.com/questions/17134479/why-do-people-s... From the FAQ (https://rpython.readthedocs.io/en/latest/faq.html https://rpython.readthedocs.io/en/latest/faq.html): > First and foremost, RPython is a language designed for writing interpreters. It is a restricted subset of Python. If your program is not an interpreter but tries to do “real things”, like use any part of the standard Python library or any 3rd-party library, then it is not RPython to start with. You should only look at RPython if you try to write your own interpreter. A related project (probably abandoned since a long time): https://code.google.com/archive/p/rpythonic/ https://code.google.com/archive/p/rpythonic/

pcr910303 6y ago

Well, I just wanted to say that it is definitely possible for native-compiled languages to do things like that, mostly by just bundling up the compiler itself. (Not sure about wasm, but looks like the article is a bit more generic than just about wasm.) One great example is SBCL, a Common Lisp implementation which only has an compiler, but provides REPL, `eval`, and `compile` functions as per the CL spec. For the question about the core of Python, I would consider that everything not tied to the external environment as Python. So `locals`, `eval`, `compile` must be provided to be called as a full-fledged Python. On the other hand, I won't include all of the stdlib functions in the definition of Python, like handling the file system or environment variables. Think about node and the web browser - both are JS runtimes and they share the core language (like `this` or `with`, which provides similar capabilities of `locals`, and `eval`), but provide different stdlibs based on the environment. (Node provides web servers, file server interactions, etc... and the web browser provides APIs to manipulate the DOM, attach event handlers, etc...)

phoe-krk 6y ago
> which only has an compiler Nitpick: SBCL ships with an interpreter, but yes, it is turned off by default in favor of the native code compiler. See http://www.sbcl.org/manual/#Interpreter http://www.sbcl.org/manual/#Interpreter for details.

blickentwapft 6y ago

MicroPython seems to have done a fairly good job of deciding what’s important in a python subset.

david_draco 6y ago

It's Python if it can run popular libraries, such as scipy. That's already an achievement to get working outside CPython.

kilon 6y ago

A possible approach is to use Cython to convert Python code to C code and then compile that code to WebAssembly. I would not be surprised if there is a tool for that already

uryga 6y ago
from my experience digging through Cython output: unless you type-annotate EVERYTHING C-style, Cython just uses the CPython C-API (PyObject_New, PY_INCREF, etc). so you'd have to ship all of CPython anyway... as mentioned elsewhere in the thread, that's too big for most websites

dragonwriter 6y ago

> But when thinking about the daunting task of creating a new implementation of Python, my brain also began asking the question of what exactly is Python? The Python documentation is pretty clear about (1) What is the Python language, (2) What is the Python stdlib (“Python” is either the first or the combination of the two), and (3) what are CPython implementation quirks. If you have all of #1, you have a Python implementation, #1 + #2 is, I guess, a Python distribution. And, yes, most of the things the article questions are part of #1, and definitively part of the essentials of the language.

moonchild 6y ago

While juggling ideas for a web replacement a while back, I came across python2wasm[1], part of the pure python compiler infrastructure. Not complete, but interesting and well worth taking a look at. 1. https://bitbucket.org/windel/ppci/src/default/ppci/lang/python/python2wasm.py https://bitbucket.org/windel/ppci/src/default/ppci/lang/pyth...

raymondh 6y ago

I suspect that Python's use in education would be an early casualty if the interactive REPL were removed.

meredydd 6y ago

As a maintainer of the Skulpt Python-to-JS compiler (https://skulpt.org https://skulpt.org), I can reasonably attempt to answer this question. Skulpt compiles Python to JS in one shot, so it's not a million miles away from the architecture Brett envisages. (I gave a lightning talk about it here: https://anvil.works/blog/pycon-talk https://anvil.works/blog/pycon-talk). Things I've observed: - "Ordinary code" requires remarkably little fidelity. Python has a very strong "simplicity culture", so most code avoids being clever. If you're creating an environment for users to program against (as we do in Anvil), it's astonishing how little of Python you need to support. - "Be able to use popular Python libraries" is really, really hard. The ordinary user doesn't use all the esoteric stuff Brett mentions, but libraries do, left and right. Even porting PyPy's `datetime` implementation to Skulpt required filling out a lot of odd little corners of the object implementation. We still have a pretty crummy subset of `unittest` because it's eye-achingly dynamic - for example, it really puts metaclasses through their paces. (Metaclasses are a perfect example of a powerful feature that really depends on details that descend directly from CPython: The API exposes the fact that variable scopes are literally `dict` objects, that classes are created dynamically at runtime, etc.) - Finally (and most unfortunately): A lot of the strength of Python is in its ecosystem, and a lot of that ecosystem relies on native code. (Again, much more than you'd think. When porting stdlib modules from PyPy to Skulpt, we have often been disappointed by how many of them use native code - even in PyPy!) The Python-to-C interface is really, really CPython-shaped. Anyone who wants to be compatible with numpy, scipy, etc, needs to literally emulate CPython's data structures, and then translate them to whatever faster/simpler thing they use internally. This emulation overhead is the main reason PyPy is slower than CPython on some workloads. (In Skulpt, we haven't even attempted this. It doesn't bother us much in Anvil, of course, as the server-side CPython is just a function call away, but for Brett's use case I'm guessing it would really hurt. One day we might try, with a JS-to-wasm bridge and doing the emulation in wasm, but it would be a hell of ajob.)

w0utert 6y ago
>> - Finally (and most unfortunately): A lot of the strength of Python is in its ecosystem, and a lot of that ecosystem relies on native code. I was thinking the same thing... One can wonder how much value there is left if you build an alternative Python interpreter that can e.g. run in the browser, but cannot support C extensions. I mean I like Python for what it is, but once you stray from the common/intended use cases, e.g. trying to embed the interpreter in a C++ program, skeletons fall out of closets around every corner. For all it's merits, CPython feels like a house of cards, burdened by a legacy of bad engineering decisions it cannot get rid of anymore. It made me realize it's better to stop fighting it, use CPython as-is (which is great), and use other scripting languages for embedding, live-coding, web, etc. I very much prefer Python over JavaScript for example, but even then it wouldn't cross my mind to use some kind of Python-to-JS implementation for web frontend code, honestly.

qwerty456127 6y ago

Give up (at least temporarily) everything listed - eval(), locals(), REPL etc and it still is going to be extremely useful for, arguably, most of the real use cases. E.g. I personally haven't ever needed any of these. REPL happens to be handy occasionally to quick-check something but I can hardly imagine it being used in production, it only seems useful in development time and for education. eval() seems a big red flag everybody should avoid at all cost. Please do your best to support all the latest syntax though. Lack of support for some standard functions particularly hard to implement is Ok, lack of support for syntactic elements (e.g. type hints) is worse and, usually, easier to fix.

ericvsmith 6y ago
namedtuple and dataclasses both use eval(), so if you want to give up eval(), you'd need to give them up, too.
- VectorLock 6y ago
  Interesting they use eval(), I wonder if their implementations could be changed to not use eval(). I guess in the end you could just use actual classes for these cases, but I sure do like namedtuple and dataclasses.
  - ericvsmith 6y ago
    My bad: it's exec(), not eval(). And I wrote dataclasses, so you'd think I'd know better! The issue is that it's dynamically generating methods. There has to be a way to get dynamically created code into python for this use case.
VectorLock 6y ago
REPL-centric development is something that has caught on a lot lately and is something that I find myself gravitating towards more and more. For 'production' the eval() might be able to be cast off, but for development the REPL is indispensable - if not a first class requirement.

fifticon 6y ago

It saddens me, that one of Pythons strengths - its C extensions - appear as a weakness in this context. The culprit is, that we perceive python as 'what you can DO through python' - which is a very reasonable perspective. But precisely python's useful boosting through C extensions, also anchors it to its friends. So, it's the old conundrum of what goes into a language core, versus what api library environment it lives in.

throwaway894345 6y ago
As I point out elsewhere it’s possible to have both performance and C-extensions, but the extension interface needs to be minimized so Python implementations have some breathing room to make optimizations—today almost any optimization would break the expansive extension interface. There are projects (like this: https://github.com/pyhandle/hpy https://github.com/pyhandle/hpy) which aim to make a smaller interface, but the community must adopt this overwhelmingly before the old interface could be deprecated and work could begin to optimize any Python implementation.
physicsyogi 6y ago
The C extensions are partly why Python reigns in the machine learning and data science world.
gnulinux 6y ago
This is why I still use Python in performance critical code (but not so performance critical that you need to squeeze each CPU cycle). It's simply too easy to write performance critical functions in C and ffi call it in python. Of course, compared to many other language it's still slow "in general" but for your give particular task it's pretty easy to make it very fast. Just write the bottlenecking tight loop in C and everything else can be in python.
- carlmr 6y ago
  Rust with it's Cargo package manager, unicode string handling and strong type system is really great for Python speed-ups: https://developers.redhat.com/blog/2017/11/16/speed-python-using-rust/ https://developers.redhat.com/blog/2017/11/16/speed-python-u...

mastrsushi 6y ago

PyObject graphs that represent dynamic variables through pseudo OOP legacy C code. Check it out, it's a total nightmare of downcasting pointers. It amazes me in the 29+ years it hasn't been replaced by a C++ alternative that could retain compatability. I think AST interpreted languages are becoming something of the past. Yes I'm aware there's bytecode in CPython. I like the idea of IL's as target platforms rather than 100 leaky interpreters that you have to manually bind your C code with if you want support ti support any one of them.

jashmatthews 6y ago
Truffle is built around AST interpretation so that’s not going to happen any time soon.
jnwatson 6y ago
This is extraordinarily normal C code. I’ve worked with several large code bases that look like CPython code. Heck, a lot of Linux looks like that. It really speaks more to the failure of C++ than anything about CPython. C++ is so much more than C with objects, and all that extra is what folks don’t want.
- mastrsushi 6y ago
  I never understood the argument of C++ being too big of a language. Code bases can enforce restrictions on what can be used.
  - stinos 6y ago
    Can you give a practical example of how a codebase can prevent restrict typical 'here's a bazooka to shoot yourself in the foot' C++ code?
- gjvc 6y ago
  > This is extraordinarily normal C code. I’ve worked with several large code bases that look like CPython code. Heck, a lot of Linux looks like that. Manually-implemented dispatch / vtbls you mean?

gjvc 6y ago

https://docs.python.org/3/reference/datamodel.html https://docs.python.org/3/reference/datamodel.html https://docs.python.org/3/reference/executionmodel.html https://docs.python.org/3/reference/executionmodel.html

tannhaeuser 6y ago

As decidely a non-insider, I'll tell you that, for me, Python's core strength is to be a beginner's all-purpose programming language, and not very much else. I had a really good experience in teaching Python to 8-10 year olds; actually, the kids were able to work out most things for themselves as I couldn't help all that much really, since I'm not much into Python. And in fact, Python was forked or at least inspired from/by ABC which was specifically designed to be a less idiosyncratic PL. Fun fact: ABC was developed by Steven Pemberton (among others) who'd later led many W3C efforts, including the failed XHTML2 effort. I didn't get around to asking how he went from ABC to XForms (which is as idiosyncratic as it gets) when I saw him as a speaker a couple years ago ;) but anyway he's always been an inspiration.

quietbritishjim 6y ago
That's a good use of Python, and if that's all it's useful for for you, that's fine. I'm not sure whether or not you're trying to argue that that is the only reaosnable use of Python? To me, it sound like you are. If so, that is contradicted by the many people who have successfully used Python for many other purposes. That includes run-once scripts (where you might have used Perl in the past) where run time speed is not as important as speed of writing the code, but also certain types of code running in production. Like any other tool, you have to be aware of its strengths and weaknesses: maybe it's not the best choice for a code base running into 10s of KLOC or more, or for doing CPU-intensive work that can't be vectorised (e.g. multiplying a few large matrices in numpy is fine, whereas multiplying an extremely large number of very small matrices is maybe not fine). The fact it has weaknesses doesn't mean it should be written off for all tasks, even those where the weaknesses aren't so important, especially when it has strengths too (e.g. if Python would do for a particular task then using C++ blindly because it's "better" would be wasting developer time for no reason). It's often used for glue code where all the work is actually done in C libraries like deep learning, and it works very well for that: if 0.1% of your runtime is spent in the top-level glue instead of 0.01%, that's probably not a problem. Also, this discussion is fairly off topic to begin with. The article really isn't about the strengths and weaknesses of the language. It's about what consitutes the language versus an implementation of it (CPython).

1337shadow 6y ago

Or just use transcrypt to transcode your Python into native JavaScript. http://www.transcrypt.org/ http://www.transcrypt.org/

jennasys 6y ago
I came here to say this. Transcrypt takes the approach that there are already many good JavaScript libraries that exist for web development, and rather than try and replace those, it embraces them. What it allows you to do though is to write code that uses those libraries in Python, and then transpile that into JS for deployment. It has allowed me to write React + MaterialUI apps using 99% Python, without taking a performance hit in the browser.

deleted 6y ago

[deleted]

lazyloop 6y ago

Python for WebAssembly already exists and runs fine on iOS. https://holzschu.github.io/a-Shell_iOS/ https://holzschu.github.io/a-Shell_iOS/

ascotan 6y ago
Pretty neat. I didn't know that the app store now allows stuff like this. I thought they had banned programming interfaces.
- rcarmo 6y ago
  Pythonista is still there.

pansa2 6y ago

> It's no secret that I want a Python implementation for WebAssembly. [...] with the fact that both iOS and Android support running JavaScript as part of an app it would also get Python on to mobile. WebAssembly is not JavaScript. So I assume iOS and Android also support running WebAssembly as part of an app?

giancarlostoro 6y ago
WebAssembly is compiled and executed by JavaScript though. You can create a runtime with no JS on the other hand probably for mobile. The beauty of WASM is eventually you could have it on all platforms as a way to make apps and expose API libraries to said runtime and now anyone can make apps in any language to run on WASM. So long as said language can be compiled to WASM. You also need JS to interop back from WASM.

bpyne 6y ago

Good question about the value of a REPL to Python. My first experience with a REPL was the SQL*Plus tool from Oracle. I don't remember it being called a REPL. But, as I read about Common Lisp's REPL, I realized that I had been using one. Early in my experience with Oracle's DBMS, I found the REPL handy to work out ideas. But a REPL is essentially a single line editor from which you can evaluate expressions/statements. As my ideas grew, the REPL became more of a hindrance. Having a full screen editor became more necessary. Nowadays I work out all ideas within an IDE, firing up the REPL only for quick data checks. So my answer is that a REPL is helpful but not necessary.

giancarlostoro 6y ago
> But a REPL is essentially a single line editor from which you can evaluate expressions/statements. It is not always single line, look at IDLE or even Firefox supports multiline REPL capabilities although it still needs some improvements (like not rerunning var declaration statements).
- bpyne 6y ago
  Thanks. Multiline certainly raises the value of a REPL.
  - giancarlostoro 6y ago
    I think so too, though I guess single line makes more sense to an extent, there's a lot more that could be done with REPLs, like I'm surprised you can't just click on a previously submitted segment of code to be auto copied in the input text field (in the case of GUI based REPLs).

ascotan 6y ago

I'm no expert here but it seems like porting a duck-typed languages like python to webassembly is hard feat. Arbitrary evaluation of source code at runtime, seems to me, exactly like the type of thing webassembly wants prevent. This limits a lot of the flexibility of a language like python (repl, eval, etc) I feel like CPython isn't so much the problem as third party libraries that people have come to rely on that, under the hood, rely on architecture dependent code (numpy, scipy, psycopg, etc, etc). I would imaging that porting these is impossible. I think a repl is fundamental to python more than any other language. I wouldn't say that about something else, but python, yes.

pansa2 6y ago

> It might make sense to develop a compiler that translates Python code directly to WebAssembly and sacrifice some compatibility for performance. This seems to be the author's preferred approach, but I'm not sure it would provide significant performance improvements over simply compiling the existing Python interpreter to WebAssembly. Compiling dynamically-typed Python to statically-typed WebAssembly would be similar to how Nuitka [0] compiles Python to C. This compiled code may be a little faster than interpreted Python, but it's still going to be quite slow - much slower in a browser than JIT-compiled JavaScript. AFAIK the ideal way to maximize the performance of Python in the browser would be to JIT-compile Python code to WebAssembly. Are WebAssembly implementations able to support that? [0] http://nuitka.net/ http://nuitka.net/

acka 6y ago

> It's no secret that I want a Python implementation for WebAssembly. It would not only get Python into the browser, but with the fact that both iOS and Android support running JavaScript as part of an app it would also get Python on to mobile. That all excites me. If the main goal is to be able to run Python code in a browser, another option is Brython[0]. According to the FAQ[1] it should work in all modern browsers, including mobile. [0] https://brython.info/ https://brython.info/ [1] https://brython.info/static_doc/en/faq.html https://brython.info/static_doc/en/faq.html

chx 6y ago
Here's a great summary of the existing options https://stackoverflow.com/a/58684358/308851 https://stackoverflow.com/a/58684358/308851

phsilva 6y ago

CPython C API is one the main culprit in making Python hard to implement and optimize. As some commenters said, I don't think Python without access to C/C++ projects would be worth right now giving how sloe CPython is. About 100% of machine learning, data science and general science Python code relies on C/C++ extensions. Current CPython API and the need to keep compatibility reduces the capabilities of improving CPython. Look at PEP-620 (https://www.python.org/dev/peps/pep-0620/ https://www.python.org/dev/peps/pep-0620/) for more info on how python developers are addressing this.

coliveira 6y ago
> science Python code relies on C/C++ extensions Don't forget Fortran. A good portion of numpy is written in Fortran.
- TheRealKing 6y ago
  scipy, as far as I am aware. but that not change your stated fact on the role of Fortran in Python.

robterrell 6y ago

Boo is a Python-like language for .NET that took a swing at answering this question: https://boo-language.github.io https://boo-language.github.io Boo's answer is "just syntax" -- libraries are .NET assemblies.

CopyZero 6y ago
Oh Man, I totally forgot about Boo. I remember looking at it years ago. This is just the thing I didn't know I was looking for. Thanks

dehrmann 6y ago

Some of the confusion is that people often blur languages and runtimes together. I started work on and then abandoned work on a Node implementation in Java backed by Nashorn. These distinctions become really clear when there's a mismatch like that. That, and some of the cpython issues would have been handled by pragmas in other languages (__slots__, for one), but got baked into the runtime. The unwritten zen of python is "bolt it on."

digitmg 6y ago

Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work! <a href="https://360digitmg.com/india/data-science-using-python-and-r-programming-in-guntur">data https://360digitmg.com/india/data-science-using-python-and-r... science course in guntur</a>