8 ms·
Multicore Programming in PyPy and CPython
- batgaijin 14y agoHaha wtf. After learning Haskell and seeing stuff like lparallel for CL, I just fundamentally can't pretend this is even worth learning. The GIL and the lack of foresight in designing Python are problems that are too fundamental to the language. I love python, I've used it for a long time, but using it as anything other than for readable shell scripting is a waste of time. It's using the wrong tool for the job. If you are worrying about speed aside from algorithms, learn a different language.
- wahnfrieden 14y agoSometimes performance is unrelated to the ability to thread, so the GIL is not always a relevant issue.
- JulianWasTaken 14y agoDo you have examples of lack of foresight?
- jerf 14y agoThere are many examples of things that people now call "lack of foresight", but I think people quickly gloss over just how long Python has been around. It started in December 1989, when the first Pentium was still three years into the future. If Guido had tried to "foresight" his way to a language with an acceptable multicore solution at that point (on any hardware he could afford), the only possible result is total failure. I don't think there are many attributes the language currently possesses that were truly that avoidable. Most people's snap suggestions result in incurring costs to the language that it may not be able to recover from when it is young. Personally, I think the truth is that Python is what it is; a very, very, very good OO language that has reached the mature stage of its life. It probably won't be able to make the leap to "true" multicore, and besides, even if it does it won't be likely to be very successful anyhow because it'll still be a very slow-but-powerful language. (There's very little point in taking a language very near the bottom of the Shootout and adding 4 or even 8 way parallelism to it, when you could just rewrite the target hotspot in C and get the same performance on one core. The math just doesn't favor trying to add lots of "multicore" to such a slow language. Python's one of my favorite languages, but that doesn't mean I can't see where it is weak.)
- batgaijin 14y agoHaskell 1.0 was released in 1990.
- hythloday 14y agoThe "History of Haskell" paper, a fascinating read in and of itself, is replete with admissions of lack of foresight. The reason Haskell has fared better than Python is probably only that it has a much bigger emphasis on trying new ideas in the reference implementation. The biggest problems with "Python" (GIL, FFI, standard library) are mostly issues with the reference implementation. http://research.microsoft.com/en-us/um/people/simonpj/papers/history-of-haskell/history.pdf http://research.microsoft.com/en-us/um/people/simonpj/papers...
- jerf 14y agoAnd where Python entered production usage in the late 90s and was building up a huge community and a base of code and continued building that base all the way to the present time, Haskell didn't really begin that until around the mid-200xs. It may be an older language in calendar terms, but in terms of libraries and tooling support it's basically 5 years old or so.
- fijal 14y agoPyPy (together with LuaJIT and tracemonkey) has been banned from shootout, partly because we complained about unfair rules. Looking at shootout to asses the relative performance of languages where you cannot compare high-performance implementations of a language is a very very bad idea. Looking at shootout to compare would very likely be a bad idea anyhow, but right now it's just completely useless.
- lmm 14y agoCan you give details and/or a citation for the complaints and the banning? I use the shootout a lot to give me a rough idea of a specific language's properties, so if there are valid criticisms then I'd like to hear them, but your post doesn't really stand on its own to someone who doesn't know the backstory.
- jspiral 14y agoDid you read the article? There are some interesting ideas being explored, really just theory for now. I don't see it as asking you to learn anything unless you want to participate in theorycrafting. To me it seems like a waste of time to build crud websites in CL, but if people want to write articles about ways to make that experience better, i'm not going to write 'haha wtf' in response. (Yes, all analogies suck)
- andybak 14y agoYou rather lack imagination if you can't think of an appropriate domain for using Python other than "readable shell scripting".
- freyrs3 14y ago> The GIL and the lack of foresight in designing Python are problems that are too fundamental to the language. The GIL isn't part of the Python language, it's part of the implementation of CPython. Jython for example doesn't have a GIL.
- mmariani 14y agoHere's his EuroPython talk http://m.youtube.com/watch?v=pDkrkP0yf70 http://m.youtube.com/watch?v=pDkrkP0yf70 And here's another talk about an alternative approach By Robert Hancock http://lanyrd.com/2011/pygotham/shppw http://lanyrd.com/2011/pygotham/shppw
- nubela 14y agoI find this incredibly interesting. I have a production server running a Flask app, performance is not stellar, and I've been meaning to try running it with Pypy w/ a web server, but I have yet to find a good resource for this. Anyone has any tips? Seems kinda counterintuitive with all these performance numbers with Pypy > Cpython when I can't even run it in production.
- pinneycolton 14y agoCan you elaborate on what is not performing well?
- nigma 14y agoAre you sure your app is not db or I/O bound? I would suggest giving https://github.com/mgood/flask-debugtoolbar https://github.com/mgood/flask-debugtoolbar a try and getting some numbers before making a switch.
- dalke 14y agoIt's strange that you have a problem with Flask in production. What I've heard, there shouldn't be much of a problem with it. It should easily be able to handle 1,000 basic requests per second. Have you done any performance analysis to figure out where the bottlenecks might be? Have you tried the flask-debugtoolbar (http://pypi.python.org/pypi/Flask-DebugToolbar http://pypi.python.org/pypi/Flask-DebugToolbar)? How many requests do you get? What's the load average of the process? Is there an un-indexed SQL query you don't know about? In the most extreme case, have you put a debugging sleep(1) call in your code which you've forgotten to remove? Without knowing more about where your Flask app is being bogged down, it's not that worthwhile to switch to pypy.
- andrewcooke 14y agothis gives a route for AME on STM in CPython via the LLVM. it assumes (afaict - it took me a while to grok the article and i'm still not sure i have it right) that the CPython GIL will be removed. but we already have a failed attempt to remove the GIL via LLVM (unladen swallow). so the obvious question is - how meaningful is that failure? does it mean that removing the GIL via LLVM is hard? how hard? because without removing the GIL, AME and STM are pointless, right? [edit: ok, thanks for the reply. it turns out it was dropped from the roadmap for unladen swallow because the garbage collection library they thought would help turned out not to - see https://code.google.com/p/unladen-swallow/wiki/ProjectPlan#Global_Interpreter_Lock https://code.google.com/p/unladen-swallow/wiki/ProjectPlan#G... - so i guess this question is much less interesting. sorry.] unrelated note / question. this is the first time i've heard that HTM is restricted to the cache(s) (although it makes a lot of sense in retrospect). how serious a limitation is that for other languages? is haskell planning to use it? what about clojure (JVM - guess not)? and what happens to HTM if the cache overflows? AME: automatic mutual exclusion, STM: software transactional memory, HTM: hardware transactional memory, LLVM: itself [fixed, thanks], GIL: global interpreter lock, JVM: java virtual machine, uff.
- kingkilr 14y agoUnladen Swallow never failed to remove the GIL, that goal was never even attempted before the project fizzled out.
- _delirium 14y agoAcronym nitpick: LLVM originally stood for low-level virtual machine, not little-language. It doesn't really have a connection to the little-language or domain-specific-language communities, but instead originates in a desire to provide a compiler target that was lower-level than generating C code, but less platform-specific than directly generating machine code (C-- is another project in that space). But it's since been renamed to just LLVM, a brand that doesn't officially stand for anything.
- mercuryrising 14y agoAlthough it adds another piece of software on top, I've been using Python with Redis a lot lately, and it's wonderful. Here's a 'base' for building the larger program. It doesn't make a single python instance multithreaded, but it easily allows me to connect other computers on my network to join in the processing fun. filePath = "/home/user/file" p = Process(target=readFile, args=(filePath,)) p.start() numProcesses = 6 for x in range(numProcesses): p = Process(target=mapper, args=()) p.start() So I have a file reader to prevent a seek storm on a large file, I read something (like separations between tags that I want to process more), I put the binary data in a "ready:NAME" key, with "NAME" going into a "readyList", and things go wonderfully. It makes my data persistent without writing to files or pickling, and I can easily share data between process instances. My question is - what advantage does a multicore programming offer that my take wouldn't? I can import process and chain things out over my 8 cores, but why would I do that if I can have persistent data with the ability to more easily add processing power when I want to?
- andrewcooke 14y agowell it allows you to share in-memory, mutable state between processes. if you have problems that don't need that then yes, parallelization is easy (the python-only version of the above would be to use multiprocessing). but maybe i've missed the point (seem to be making error after error at the moment...) [edit: to clarify a bit more, the example given in the article linked is making each execution of a loop run in parallel, even if the loop affects some variable. so you would hope to get speedups in code that isn't written explicitly to be executed by multiple processes - without messaging etc. in particular, it guarantees safety while giving the chance (sometimes it doesn't work out) for speedups.]
- TillE 14y agoMessage passing is usually a "better" method of IPC anyway. The situations that genuinely require shared memory are rare.
- deleted 14y ago[deleted]
- exDM69 14y agoIt would be good to first make it possible to write Python code that can execute concurrently and use plain old locks and conditions -style synchronization. When that works efficiently, things like transactional memory can add value, but not before. What comes to automatic mutual exclusion "AME", that sounds pretty far fetched to me. I don't think there's a production-worthy implementation of anything like this out there, all parallel programming environments need some kind of annotations from the programmer to keep stuff in sync. Is anyone aware of any research into this kind of systems? If anyone is interested in parallel programming features in languages, Haskell's parallel programming facilities should be worth looking at. Not because the language or functional programming would be some kind of magic medicine, but simply because there's been years of research and hard work put into making Haskell work in a concurrent environment.
- zurn 14y agoSo are you saying the current STM prototype should be put on hold until a "plain old locks" Python comes along?
- lucian1900 14y agoI think the main problem is that the vast majority of Python programs assume that things like dict lookups are atomic. With small locks (which would be really hard to do with CPython anyway), that guarantee is hard to keep. The nicest thing about STM is that transaction start/stop map quite well to GIL aquire/release, so it should fit the common cases quite well.
- k_bx 14y agoWell you can do that in Jython, but as we can see, not so many people actually "need" this.