10 ms·

Python and Lua (2019)

99 points by ttymck 5y ago

andy-x 5y ago
Lua version of the code is incomprehensible. Could it be that if you implement the same thing in Python it will also run faster?
- byteface 5y ago
  you mean with buffers i.e. array.array . i thought that.
- munch117 5y ago
  Much of the difference is due to rewriting complex numbers into pairs of floating-point variables. Probably because Lua doesn't have complex numbers? The same rewrite in Python would not speed things up, it would slow things down. It's more than that, though. It looks like the Lua code does only 49 repeats where the Python code does 200. It's clear that this isn't just a Python->Lua rewrite, this is a rewrite that was followed by a lot of optimisation work on the Lua code. Despite that, I kinda buy it. I can believe that you won't get speeds in pure-Python code for this particular problem that are competitive with a JIT. numpy doesn't help, because the number of iterations varies for each point, so you can't easily parallelise. And speaking of parallelisation, he has a point about threading: Doing the computations in pure-Python would not perform well over multiple threads, you need Cython or something. And that something can apparently be Lua.
  - kzrdude 5y ago
    It turns out using numpy is enough for Python to parallelize this adequately using threads, see this code: https://news.ycombinator.com/item?id=28817357 https://news.ycombinator.com/item?id=28817357
    - munch117 5y ago
      It's not the same algorithm: That code completes all 200 iterations unconditionally. For some parts of the input domain, the numpy speed-up is enough to compensate and make it faster anyway, but for other parts it will be slower.
      - kzrdude 5y ago
        Agree, even if it compensates by supplying a gradually shrinking compute mask, but not to all operations. But an effect is there, less populated chunks compute faster.
- kzrdude 5y ago
  No. Lua here uses luajit, a jit compiler and interpreter for Lua which Python can not beat. Python has the numba package, which is in the same direction, less easy to use, more restricted, and I don't know if numba can approach luajit's speed here, we'd have to try it out. (After working on the numpy code posted in this thread, I think Python is pretty competitive, but it requires quite a few contortions - rewriting it using numpy.)
sarvasana 5y ago
Just pick a different language all together.
blondin 5y ago
how is lua doing these days?
- kzrdude 5y ago
  One ray of light is that Mike Pall is working on luajit again https://github.com/LuaJIT/LuaJIT https://github.com/LuaJIT/LuaJIT which at this point is a fork of mainline Lua in terms of version history
  - wruza 5y ago
    I don’t want to sound too negative, but that also means that LuaJIT failed to increase its bus factor with these guys who took over the maintenance recently.
    - kzrdude 5y ago
      Not too surprising, after all Mike wrote LuaJIT as cleverly as possible (*). And I really admire him. (*) https://www.linusakesson.net/programming/kernighans-lever/index.php https://www.linusakesson.net/programming/kernighans-lever/in...
- nerdponx 5y ago
  Neovim adopted it as its modern scripting language alongside Vimscript. And it now has at least two interesting scripting languages that compile to it, Fennel and Teal. So it's got quite a community of Neovim plugin developers writing Lua and these languages that compile to it.
  - hkt 5y ago
    Make that three, as Moonscript is also great: https://moonscript.org/ https://moonscript.org/
    - nerdponx 5y ago
      Of course! And Moonscript long predates the other two.
swalls 5y ago
Why not just call into a C/other compiled language library in this case? It'll be faster and it's not like you're getting the advantages of Lua in this case because you're not actually integrating it. In the ideal case Lua likes to call into native functions to do the heavy lifting anyway.
- phyrex 5y ago
  Writing C is a lot harder than writing Lua, especially coming from Python
  - Rochus 5y ago
    With LuaJIT you can just write Lua and call C functions as if they were Lua. The LuaJIT FFI is very powerful and a nice way to profit from the C speedup without the hassles.
  - int_19h 5y ago
    That's what Cython is for, no?
pansa2 5y ago
> You can parallelize with threads! This is a great example of something that's more than the sum of its parts. A GIL-constrained language and a single-threaded language working together to run code in parallel - something that neither language can achieve on its own. Although I believe there is a plan to add support for this pattern (multiple threads with one interpreter state each) to pure Python in the form of "subinterpreters".
- jjice 5y ago
  Interesting, how does that compare to the multiprocessing module python has now?
  - O5vYtytb 5y ago
    Multiprocessing requires multiple python processes which are expensive with regards to startup time and resource use. Multiple interpreters give you parallelism without so much overhead.
    - wruza 5y ago
      Wait, isn’t python startup slow because of the interpreter startup and not process creation itself? A thread also has to be created (saves dll mapping time but that’s it) and python state also has to be initialized in a complete isolation on that thread (means setting up the entire scene again and loading stdlib from sources). Am I wrong?
      - O5vYtytb 5y ago
        It's a number of factors, including the loading of the modules necessary to start python and the standard library. Check out https://github.com/faster-cpython/ideas/issues?q=is%3Aissue+is%3Aopen+faster+startup https://github.com/faster-cpython/ideas/issues?q=is%3Aissue+... for some ideas regarding improvement there.
      - maxnoe 5y ago
        The point is each process has to start its own interpreter.
kzrdude 5y ago
Lua is so nice, but I don't really use it for anything practical. I'd like to think that I'd rather have Lua as a language, with Python's class definitions and Python's ecosystem. What I mean by Lua as a language is the interpreter - and luajit - but also its `local` variables and scoping rules (Python's rules are rather strange, which the nonlocal keyword signals clearly).
- nerdponx 5y ago
  The main downsides of Lua are the overloaded handling of nil, and the lack of a real standard library. If it weren't for that, it'd be a pretty nice general-purpose scripting language, arguably better than Python. I probably disagree with 90% of the criticisms leveled against Python, but it definitely is slow in "loopy" tasks, and the indentation-based syntax can make for relatively verbose code in quick ad-hoc scripts. I'd be plenty happy using Lua instead, if only it had a more thorough standard library. The nil-handling thing is annoying, but fine enough for quick scripts.
  - pansa2 5y ago
    > the indentation-based syntax can make for relatively verbose code in quick ad-hoc scripts. Isn't significant indentation less verbose than Lua's use of `do ... end`?
    - scythe 5y ago
      Not if you write one-liners like if x < 3 then x = 3 end I actually really like Lua's syntax, though I think a form for short Python-style lambdas would really help (short lambdas are the easiest ones to understand, so I don't see a comprehensibility objection, and Lua's grammar already makes the necessary expression-statement distinction). The Lua ecosystem was seriously damaged by the 5.1>5.2 split for a while since LuaJIT (hence Torch/nginx) and LÖVE both basically stayed with 5.1 and those were major projects that drew people to the language. The other cool unique thing in the Lua ecosystem, the native-widget (Windows/Gtk) IUP toolkit, remains barely known about and never ported to OS X, possibly due to a lack of Mac users in Brazil. Hence the "language without an ecosystem": it didn't happen for no reason, though the willingness to make breaking changes is a reason the language became so popular in the first place. NB. IUP is still around, but it still doesn't have a Cocoa port.
bch 5y ago
> Does it make sense to have another scripting language inside Python scripts? Point of interest, Python already has another embedded language if you have Tkinter available: Tcl. Larry Wall of Perl fame once said: “Tcl tends to be ported to weird places like routers.”[0] Routers are just the beginning… see here[1] for calling Tcl scripts directly into a Python-embedded Tcl interpreter. I believe at least one of the Perl/Tk attempts did something similar (embedding Tcl wholesale) too. [0] https://github.com/thibaultduponchelle/perl-fortunes/blob/master/data/larry-wall https://github.com/thibaultduponchelle/perl-fortunes/blob/ma... [1] https://stackoverflow.com/questions/7362846/running-tcl-code-from-python https://stackoverflow.com/questions/7362846/running-tcl-code...
- watersb 5y ago
  Tcl was briefly used for the monitor and control system for the original 12-meter radio telescope on Kitt Peak in Arizona. The Tcl system was ported from the Very Large Telescope VLT, in Chile. So that's a couple of weird places. https://en.m.wikipedia.org/wiki/ARO_12m_Radio_Telescope https://en.m.wikipedia.org/wiki/ARO_12m_Radio_Telescope
- BiteCode_dev 5y ago
  The supported python version: >>> import tkinter as tk >>> tk.Tcl().eval('puts {hello, world}') hello, world '' I wonder if this releases the GIL.
pjbk 5y ago
Through the years starting from around 1998 I deployed Lua into several embedded systems: hardware debugging tools, scientific instruments, medical devices, home automation and finally robotics. Back then you couldn't beat the small footprint of the environment and the decent availability of libraries so you didn't need to start everything from scratch. In contrast, for example the Pawn language was also competent for similar applications but its ecosystem was quite lacking. Some years ago when Squirrel started to get traction, particularly due to its adoption in a couple of IoT frameworks, I fiddled with it for some months and it almost got its way into a commercial electronics project. However by then uPython was getting more popular, had better tooling support and ARM microcontroller chips had become very fast. Being able to reuse the same code on the host with minimal or no changes, or having an interface or emulation layer that you can also program in Python is a game changer. It took me less than a week to rewrite the Squirrel code to Python, with even better features thanks to the language facilities. I thought I was never going to touch Lua again but a couple of months ago I found myself hacking some REAPER audio processing scripts, and cursing Ierusalimschy once more for his decision on neglecting Dijkstra's advice and not starting indexes from zero...
- rstuart4133 5y ago
  Lua is optimised for interpreter size, Python for "language niceness / flexibility", and it shows. Lua has lots of nits, such as the index starts at 1 thing, or the "all numbers are floats" thing. I also find its class mechanism ugly, but the most annoying one for me is it suffers from the totally unnecessary "a variable is global by default" bug. However, life is full of unexpected surprises. While Lua small engine size was a deliberate strategy to make it cheap to embed, it also meant it's much easier to optimise than something the size of Python. And ... that's what happened, so LuaJit is the fastest interpreted language - even faster than the best javascript engines when I last looked. That despite javascript engines having had man decades of optimisation effort put into them.
  - portaltonowhere 5y ago
    FWIW, Lua 5.3 (released in 2015) introduced integers. But agree with the rest.
aasasd 5y ago
This is missing one large avenue of speedup: the startup time, which is bad with Python. Some of my Lua scripts are done faster than Python starts up, even when doing file i/o. This is especially crucial for interactive ‘productivity’ helper scripts, which need to respond instantly. As an anecdote, when I was making my first Lua script for Alfred on a horribly underpowered machine, I briefly couldn't figure out if I actually configured the script properly, because the dummy example output indeed appeared right as I typed the last input key. Turns out, Lua really runs that fast. Alas, the lack of proper ‘null’ bugs me—it may interfere with making middleware for external APIs, receiving and transmitting schema-less data. A bunch of libraries use cjson.null or interoperable lua-null or somesuch, but I'm sure I'll bump into a lib that won't know about those. Ah, and by the way, from what I vaguely heard, actual native multithreading might be doable in Lua with libraries. At least I've seen a couple of such libs, but haven't tried them.
- BiteCode_dev 5y ago
  Yes, I'm hoping the latest MS effort on speeding up Python will also tackles that. The JIT effort is great, but projects like mercurial had to hack lazy import because they are so slow in python. I wish this can be baked in the language some day: loading half the stdlib to run hello world is not very efficient.
dottrap 5y ago
If you want even more speed for Lua, Lua's new experimental sister language, Pallene is really exciting. Pallene is Lua with strong types, designed for the purpose of being able to (AOT) generate faster and native code. It's compiler design also has deep knowledge of the Lua implementation, so it short circuits the formal Lua/C API so crossing the Lua/C language boundary in Pallene is even faster (and Lua's is already good...much less overhead than crossing in Python). https://news.ycombinator.com/item?id=18038619 https://news.ycombinator.com/item?id=18038619
- wruza 5y ago
  titan-lang.org Why is it down? I found other links for those interested: https://github.com/titan-lang/titan https://github.com/titan-lang/titan https://github.com/titan-lang/titan/blob/master/doc/manual.md https://github.com/titan-lang/titan/blob/master/doc/manual.m...
  - dottrap 5y ago
    This is the Pallene repo: https://github.com/pallene-lang/pallene https://github.com/pallene-lang/pallene
  - hkt 5y ago
    I find it fascinating to know the same university department that is affiliated with lua is related to other well designed languages. See also, Terra: https://terralang.org/ https://terralang.org/ Easily interoperable languages present an interesting proposition: a vertically integrated approach to language design where tools are more specialised, but designed from the get-go to work together. Fascinating, and awesome.
- Rochus 5y ago
  Unfortunately Pallene is not Lua, not even a true subset; it's just another language with some similarities to Lua.
  - dottrap 5y ago
    I think they define Pallene as a true subset to Lua in that the Pallene compiler can emit a pure Lua version (which strips all the types), and will produce the same results when run.
emrah 5y ago
This is a contrived example. Sure yes it can be done but seems hard to work with. It would be hard to debug for example. For speeding up pure python, the first thing I would do is reach for numpy. It would be interesting to compare the performance of the lua-embedded code with a version using numpy.
pbourke 5y ago
The Python-only version can be sped up using vectorized numpy operations: import time from PIL import Image import numpy as np def PyMandelbrot(size): t1 = time.time() xPts = np.arange(-1.5, 0.5, 2.0 / size).reshape((1, size)) yPts = (np.arange(-1, 1, 2.0 / size) * 1j).reshape((size, 1)) cs = xPts + yPts z = np.zeros_like(cs) red = np.zeros_like(cs, dtype=np.uint8) mask = np.full_like(cs, True, dtype=np.bool_) max_idx = 0 while max_idx < 199: np.putmask(z, mask, np.multiply(z, z) + cs) np.putmask(red, mask, red + 1) mask &= np.abs(z) < 2 max_idx += 1 dt = time.time() - t1 print(f"dt={dt:.2f}") zeros = np.zeros_like(red) pixels = np.dstack((red, zeros, zeros)) Image.fromarray(pixels).show() if __name__ == '__main__': PyMandelbrot(1_280) On my machine (Core i9, Win 10, CPython 3.7): > python .\mandelbrot_orig.py dt=32.01 > python .\mandelbrot.py dt=5.86 I'm sure my version can be improved upon.
- kzrdude 5y ago
  Thanks for sharing, that's a big improvement! The details are easier to see, for me, in grayscale, that is using: Image.fromarray(255 - red).show() I'm learning as I go here, using numpy functions I've never used before, but we can use more in-place modifying operations like this (np.multiply, np.add with where= and out=) and it decreases runtime of the benchmark by 25%. Then, after that, this problem is embarassingly parallel and thanks to numpy we can parallelize your code very well, using all cores (improves runtime for me from 3.5 to 0.92 seconds) Using Python3.8.: import concurrent.futures import time from PIL import Image import numpy as np def compute_mandelbrot(cs): """ cs: complex grid subset """ z = np.zeros_like(cs) red = np.zeros_like(cs, dtype=np.uint8) mask = np.ones_like(cs, dtype=bool) for _ in range(199): # z² + c -> z np.multiply(z, z, where=mask, out=z) np.add(z, cs, where=mask, out=z) np.add(red, 1, where=mask, out=red) mask &= np.abs(z) < 2 return red def py_mandelbrot(size): t1 = time.time() yPts, xPts = np.ogrid[-1j:1j:size * 1j, -1.5:0.5:size * 1j] cs = xPts + yPts # over-split because chunks are not equal in time taken nsplit = 32 nthreads = 8 css = np.array_split(cs, nsplit) with concurrent.futures.ThreadPoolExecutor(max_workers=nthreads) as executor: reds_out = list(executor.map(compute_mandelbrot, css)) red = np.concatenate(reds_out) dt = time.time() - t1 print(f"dt={dt:.2f}") Image.fromarray(255 - red).show() if __name__ == '__main__': py_mandelbrot(1280) Python makes it easy to get computed chunks in the order they are submitted, using executor.map. I still tried out what it would look like if I concatenated the chunks in the order they were received. That's when I noticed that emptier tiles were computed faster. So then, we can split the problem into more fine-grained chunks and submit more of them while keeping the number of threads the same, for a small speedup since some chunks are faster to compute.
BiteCode_dev 5y ago
It's fun and probably useful, but Python is not 30 times slower than lua (from 33s to less than a sec). The 2 codes don't do the same thing, if you perform the same computation on the python code (monochrome mandelbrot, inline loop, 49 iterations, etc), you get 0.8 vs 4.8 sec, which seems about right. I've removed numpy since it's cheating if we compare raw language perfs: https://0bin.net/paste/nOtLmeMA#0DKGHSvmgoDLPzHlQOhwvVhxxayflBBHSeY0ktv-jpR https://0bin.net/paste/nOtLmeMA#0DKGHSvmgoDLPzHlQOhwvVhxxayf... So yes, Python is about 6 times slower. Significative, but can be worth the compromise if you consider the Python version is much, much easier to read. And of course, a vectorized algo would be even faster (see pbourke's comment). What's more, with SharedMemory from 3.8, you could speed that up even in pure Python using multiprocessing by sharing efficiently the array.
cycomanic 5y ago
If you have a c++ compiler available you can get significant speedups without any changes to the code (except for using complex not np.complex, because it's deprecated). Just add a comment #pythran export PyMandelbrot(uint8[:,:,:], int) Note I passed the array into the function to remove the image creation from the module. This yields a 18.6 ms vs 1.1s for the original for size 320. As a side note I find it worrying that a blog which is called speed matters uses time instead of timeit.
edrx 5y ago
There's a list of languages that compile to Lua here: https://github.com/hengestone/lua-languages https://github.com/hengestone/lua-languages

99 points by ttymck 5y ago

Lua version of the code is incomprehensible. Could it be that if you implement the same thing in Python it will also run faster?

byteface 5y ago
you mean with buffers i.e. array.array . i thought that.
munch117 5y ago
Much of the difference is due to rewriting complex numbers into pairs of floating-point variables. Probably because Lua doesn't have complex numbers? The same rewrite in Python would not speed things up, it would slow things down. It's more than that, though. It looks like the Lua code does only 49 repeats where the Python code does 200. It's clear that this isn't just a Python->Lua rewrite, this is a rewrite that was followed by a lot of optimisation work on the Lua code. Despite that, I kinda buy it. I can believe that you won't get speeds in pure-Python code for this particular problem that are competitive with a JIT. numpy doesn't help, because the number of iterations varies for each point, so you can't easily parallelise. And speaking of parallelisation, he has a point about threading: Doing the computations in pure-Python would not perform well over multiple threads, you need Cython or something. And that something can apparently be Lua.
- kzrdude 5y ago
  It turns out using numpy is enough for Python to parallelize this adequately using threads, see this code: https://news.ycombinator.com/item?id=28817357 https://news.ycombinator.com/item?id=28817357
  - munch117 5y ago
    It's not the same algorithm: That code completes all 200 iterations unconditionally. For some parts of the input domain, the numpy speed-up is enough to compensate and make it faster anyway, but for other parts it will be slower.
    - kzrdude 5y ago
      Agree, even if it compensates by supplying a gradually shrinking compute mask, but not to all operations. But an effect is there, less populated chunks compute faster.
kzrdude 5y ago
No. Lua here uses luajit, a jit compiler and interpreter for Lua which Python can not beat. Python has the numba package, which is in the same direction, less easy to use, more restricted, and I don't know if numba can approach luajit's speed here, we'd have to try it out. (After working on the numpy code posted in this thread, I think Python is pretty competitive, but it requires quite a few contortions - rewriting it using numpy.)

sarvasana 5y ago

Just pick a different language all together.

blondin 5y ago

how is lua doing these days?

kzrdude 5y ago
One ray of light is that Mike Pall is working on luajit again https://github.com/LuaJIT/LuaJIT https://github.com/LuaJIT/LuaJIT which at this point is a fork of mainline Lua in terms of version history
- wruza 5y ago
  I don’t want to sound too negative, but that also means that LuaJIT failed to increase its bus factor with these guys who took over the maintenance recently.
  - kzrdude 5y ago
    Not too surprising, after all Mike wrote LuaJIT as cleverly as possible (*). And I really admire him. (*) https://www.linusakesson.net/programming/kernighans-lever/index.php https://www.linusakesson.net/programming/kernighans-lever/in...
nerdponx 5y ago
Neovim adopted it as its modern scripting language alongside Vimscript. And it now has at least two interesting scripting languages that compile to it, Fennel and Teal. So it's got quite a community of Neovim plugin developers writing Lua and these languages that compile to it.
- hkt 5y ago
  Make that three, as Moonscript is also great: https://moonscript.org/ https://moonscript.org/
  - nerdponx 5y ago
    Of course! And Moonscript long predates the other two.

swalls 5y ago

Why not just call into a C/other compiled language library in this case? It'll be faster and it's not like you're getting the advantages of Lua in this case because you're not actually integrating it. In the ideal case Lua likes to call into native functions to do the heavy lifting anyway.

phyrex 5y ago
Writing C is a lot harder than writing Lua, especially coming from Python
- Rochus 5y ago
  With LuaJIT you can just write Lua and call C functions as if they were Lua. The LuaJIT FFI is very powerful and a nice way to profit from the C speedup without the hassles.
- int_19h 5y ago
  That's what Cython is for, no?

pansa2 5y ago

> You can parallelize with threads! This is a great example of something that's more than the sum of its parts. A GIL-constrained language and a single-threaded language working together to run code in parallel - something that neither language can achieve on its own. Although I believe there is a plan to add support for this pattern (multiple threads with one interpreter state each) to pure Python in the form of "subinterpreters".

jjice 5y ago
Interesting, how does that compare to the multiprocessing module python has now?
- O5vYtytb 5y ago
  Multiprocessing requires multiple python processes which are expensive with regards to startup time and resource use. Multiple interpreters give you parallelism without so much overhead.
  - wruza 5y ago
    Wait, isn’t python startup slow because of the interpreter startup and not process creation itself? A thread also has to be created (saves dll mapping time but that’s it) and python state also has to be initialized in a complete isolation on that thread (means setting up the entire scene again and loading stdlib from sources). Am I wrong?
    - O5vYtytb 5y ago
      It's a number of factors, including the loading of the modules necessary to start python and the standard library. Check out https://github.com/faster-cpython/ideas/issues?q=is%3Aissue+is%3Aopen+faster+startup https://github.com/faster-cpython/ideas/issues?q=is%3Aissue+... for some ideas regarding improvement there.
    - maxnoe 5y ago
      The point is each process has to start its own interpreter.

kzrdude 5y ago

Lua is so nice, but I don't really use it for anything practical. I'd like to think that I'd rather have Lua as a language, with Python's class definitions and Python's ecosystem. What I mean by Lua as a language is the interpreter - and luajit - but also its `local` variables and scoping rules (Python's rules are rather strange, which the nonlocal keyword signals clearly).

nerdponx 5y ago
The main downsides of Lua are the overloaded handling of nil, and the lack of a real standard library. If it weren't for that, it'd be a pretty nice general-purpose scripting language, arguably better than Python. I probably disagree with 90% of the criticisms leveled against Python, but it definitely is slow in "loopy" tasks, and the indentation-based syntax can make for relatively verbose code in quick ad-hoc scripts. I'd be plenty happy using Lua instead, if only it had a more thorough standard library. The nil-handling thing is annoying, but fine enough for quick scripts.
- pansa2 5y ago
  > the indentation-based syntax can make for relatively verbose code in quick ad-hoc scripts. Isn't significant indentation less verbose than Lua's use of `do ... end`?
  - scythe 5y ago
    Not if you write one-liners like if x < 3 then x = 3 end I actually really like Lua's syntax, though I think a form for short Python-style lambdas would really help (short lambdas are the easiest ones to understand, so I don't see a comprehensibility objection, and Lua's grammar already makes the necessary expression-statement distinction). The Lua ecosystem was seriously damaged by the 5.1>5.2 split for a while since LuaJIT (hence Torch/nginx) and LÖVE both basically stayed with 5.1 and those were major projects that drew people to the language. The other cool unique thing in the Lua ecosystem, the native-widget (Windows/Gtk) IUP toolkit, remains barely known about and never ported to OS X, possibly due to a lack of Mac users in Brazil. Hence the "language without an ecosystem": it didn't happen for no reason, though the willingness to make breaking changes is a reason the language became so popular in the first place. NB. IUP is still around, but it still doesn't have a Cocoa port.

bch 5y ago

> Does it make sense to have another scripting language inside Python scripts? Point of interest, Python already has another embedded language if you have Tkinter available: Tcl. Larry Wall of Perl fame once said: “Tcl tends to be ported to weird places like routers.”[0] Routers are just the beginning… see here[1] for calling Tcl scripts directly into a Python-embedded Tcl interpreter. I believe at least one of the Perl/Tk attempts did something similar (embedding Tcl wholesale) too. [0] https://github.com/thibaultduponchelle/perl-fortunes/blob/master/data/larry-wall https://github.com/thibaultduponchelle/perl-fortunes/blob/ma... [1] https://stackoverflow.com/questions/7362846/running-tcl-code-from-python https://stackoverflow.com/questions/7362846/running-tcl-code...

watersb 5y ago
Tcl was briefly used for the monitor and control system for the original 12-meter radio telescope on Kitt Peak in Arizona. The Tcl system was ported from the Very Large Telescope VLT, in Chile. So that's a couple of weird places. https://en.m.wikipedia.org/wiki/ARO_12m_Radio_Telescope https://en.m.wikipedia.org/wiki/ARO_12m_Radio_Telescope
BiteCode_dev 5y ago
The supported python version: >>> import tkinter as tk >>> tk.Tcl().eval('puts {hello, world}') hello, world '' I wonder if this releases the GIL.

pjbk 5y ago

Through the years starting from around 1998 I deployed Lua into several embedded systems: hardware debugging tools, scientific instruments, medical devices, home automation and finally robotics. Back then you couldn't beat the small footprint of the environment and the decent availability of libraries so you didn't need to start everything from scratch. In contrast, for example the Pawn language was also competent for similar applications but its ecosystem was quite lacking. Some years ago when Squirrel started to get traction, particularly due to its adoption in a couple of IoT frameworks, I fiddled with it for some months and it almost got its way into a commercial electronics project. However by then uPython was getting more popular, had better tooling support and ARM microcontroller chips had become very fast. Being able to reuse the same code on the host with minimal or no changes, or having an interface or emulation layer that you can also program in Python is a game changer. It took me less than a week to rewrite the Squirrel code to Python, with even better features thanks to the language facilities. I thought I was never going to touch Lua again but a couple of months ago I found myself hacking some REAPER audio processing scripts, and cursing Ierusalimschy once more for his decision on neglecting Dijkstra's advice and not starting indexes from zero...

rstuart4133 5y ago
Lua is optimised for interpreter size, Python for "language niceness / flexibility", and it shows. Lua has lots of nits, such as the index starts at 1 thing, or the "all numbers are floats" thing. I also find its class mechanism ugly, but the most annoying one for me is it suffers from the totally unnecessary "a variable is global by default" bug. However, life is full of unexpected surprises. While Lua small engine size was a deliberate strategy to make it cheap to embed, it also meant it's much easier to optimise than something the size of Python. And ... that's what happened, so LuaJit is the fastest interpreted language - even faster than the best javascript engines when I last looked. That despite javascript engines having had man decades of optimisation effort put into them.
- portaltonowhere 5y ago
  FWIW, Lua 5.3 (released in 2015) introduced integers. But agree with the rest.

aasasd 5y ago

This is missing one large avenue of speedup: the startup time, which is bad with Python. Some of my Lua scripts are done faster than Python starts up, even when doing file i/o. This is especially crucial for interactive ‘productivity’ helper scripts, which need to respond instantly. As an anecdote, when I was making my first Lua script for Alfred on a horribly underpowered machine, I briefly couldn't figure out if I actually configured the script properly, because the dummy example output indeed appeared right as I typed the last input key. Turns out, Lua really runs that fast. Alas, the lack of proper ‘null’ bugs me—it may interfere with making middleware for external APIs, receiving and transmitting schema-less data. A bunch of libraries use cjson.null or interoperable lua-null or somesuch, but I'm sure I'll bump into a lib that won't know about those. Ah, and by the way, from what I vaguely heard, actual native multithreading might be doable in Lua with libraries. At least I've seen a couple of such libs, but haven't tried them.

BiteCode_dev 5y ago
Yes, I'm hoping the latest MS effort on speeding up Python will also tackles that. The JIT effort is great, but projects like mercurial had to hack lazy import because they are so slow in python. I wish this can be baked in the language some day: loading half the stdlib to run hello world is not very efficient.

dottrap 5y ago

If you want even more speed for Lua, Lua's new experimental sister language, Pallene is really exciting. Pallene is Lua with strong types, designed for the purpose of being able to (AOT) generate faster and native code. It's compiler design also has deep knowledge of the Lua implementation, so it short circuits the formal Lua/C API so crossing the Lua/C language boundary in Pallene is even faster (and Lua's is already good...much less overhead than crossing in Python). https://news.ycombinator.com/item?id=18038619 https://news.ycombinator.com/item?id=18038619

wruza 5y ago
titan-lang.org Why is it down? I found other links for those interested: https://github.com/titan-lang/titan https://github.com/titan-lang/titan https://github.com/titan-lang/titan/blob/master/doc/manual.md https://github.com/titan-lang/titan/blob/master/doc/manual.m...
- dottrap 5y ago
  This is the Pallene repo: https://github.com/pallene-lang/pallene https://github.com/pallene-lang/pallene
- hkt 5y ago
  I find it fascinating to know the same university department that is affiliated with lua is related to other well designed languages. See also, Terra: https://terralang.org/ https://terralang.org/ Easily interoperable languages present an interesting proposition: a vertically integrated approach to language design where tools are more specialised, but designed from the get-go to work together. Fascinating, and awesome.
Rochus 5y ago
Unfortunately Pallene is not Lua, not even a true subset; it's just another language with some similarities to Lua.
- dottrap 5y ago
  I think they define Pallene as a true subset to Lua in that the Pallene compiler can emit a pure Lua version (which strips all the types), and will produce the same results when run.

emrah 5y ago

This is a contrived example. Sure yes it can be done but seems hard to work with. It would be hard to debug for example. For speeding up pure python, the first thing I would do is reach for numpy. It would be interesting to compare the performance of the lua-embedded code with a version using numpy.

pbourke 5y ago

The Python-only version can be sped up using vectorized numpy operations: import time from PIL import Image import numpy as np def PyMandelbrot(size): t1 = time.time() xPts = np.arange(-1.5, 0.5, 2.0 / size).reshape((1, size)) yPts = (np.arange(-1, 1, 2.0 / size) * 1j).reshape((size, 1)) cs = xPts + yPts z = np.zeros_like(cs) red = np.zeros_like(cs, dtype=np.uint8) mask = np.full_like(cs, True, dtype=np.bool_) max_idx = 0 while max_idx < 199: np.putmask(z, mask, np.multiply(z, z) + cs) np.putmask(red, mask, red + 1) mask &= np.abs(z) < 2 max_idx += 1 dt = time.time() - t1 print(f"dt={dt:.2f}") zeros = np.zeros_like(red) pixels = np.dstack((red, zeros, zeros)) Image.fromarray(pixels).show() if __name__ == '__main__': PyMandelbrot(1_280) On my machine (Core i9, Win 10, CPython 3.7): > python .\mandelbrot_orig.py dt=32.01 > python .\mandelbrot.py dt=5.86 I'm sure my version can be improved upon.

kzrdude 5y ago
Thanks for sharing, that's a big improvement! The details are easier to see, for me, in grayscale, that is using: Image.fromarray(255 - red).show() I'm learning as I go here, using numpy functions I've never used before, but we can use more in-place modifying operations like this (np.multiply, np.add with where= and out=) and it decreases runtime of the benchmark by 25%. Then, after that, this problem is embarassingly parallel and thanks to numpy we can parallelize your code very well, using all cores (improves runtime for me from 3.5 to 0.92 seconds) Using Python3.8.: import concurrent.futures import time from PIL import Image import numpy as np def compute_mandelbrot(cs): """ cs: complex grid subset """ z = np.zeros_like(cs) red = np.zeros_like(cs, dtype=np.uint8) mask = np.ones_like(cs, dtype=bool) for _ in range(199): # z² + c -> z np.multiply(z, z, where=mask, out=z) np.add(z, cs, where=mask, out=z) np.add(red, 1, where=mask, out=red) mask &= np.abs(z) < 2 return red def py_mandelbrot(size): t1 = time.time() yPts, xPts = np.ogrid[-1j:1j:size * 1j, -1.5:0.5:size * 1j] cs = xPts + yPts # over-split because chunks are not equal in time taken nsplit = 32 nthreads = 8 css = np.array_split(cs, nsplit) with concurrent.futures.ThreadPoolExecutor(max_workers=nthreads) as executor: reds_out = list(executor.map(compute_mandelbrot, css)) red = np.concatenate(reds_out) dt = time.time() - t1 print(f"dt={dt:.2f}") Image.fromarray(255 - red).show() if __name__ == '__main__': py_mandelbrot(1280) Python makes it easy to get computed chunks in the order they are submitted, using executor.map. I still tried out what it would look like if I concatenated the chunks in the order they were received. That's when I noticed that emptier tiles were computed faster. So then, we can split the problem into more fine-grained chunks and submit more of them while keeping the number of threads the same, for a small speedup since some chunks are faster to compute.

BiteCode_dev 5y ago

It's fun and probably useful, but Python is not 30 times slower than lua (from 33s to less than a sec). The 2 codes don't do the same thing, if you perform the same computation on the python code (monochrome mandelbrot, inline loop, 49 iterations, etc), you get 0.8 vs 4.8 sec, which seems about right. I've removed numpy since it's cheating if we compare raw language perfs: https://0bin.net/paste/nOtLmeMA#0DKGHSvmgoDLPzHlQOhwvVhxxayflBBHSeY0ktv-jpR https://0bin.net/paste/nOtLmeMA#0DKGHSvmgoDLPzHlQOhwvVhxxayf... So yes, Python is about 6 times slower. Significative, but can be worth the compromise if you consider the Python version is much, much easier to read. And of course, a vectorized algo would be even faster (see pbourke's comment). What's more, with SharedMemory from 3.8, you could speed that up even in pure Python using multiprocessing by sharing efficiently the array.

cycomanic 5y ago

If you have a c++ compiler available you can get significant speedups without any changes to the code (except for using complex not np.complex, because it's deprecated). Just add a comment #pythran export PyMandelbrot(uint8[:,:,:], int) Note I passed the array into the function to remove the image creation from the module. This yields a 18.6 ms vs 1.1s for the original for size 320. As a side note I find it worrying that a blog which is called speed matters uses time instead of timeit.

edrx 5y ago

There's a list of languages that compile to Lua here: https://github.com/hengestone/lua-languages https://github.com/hengestone/lua-languages