6 ms·

Sure, except when that doesn't make any sense. I embed Python in my application. My highly graphical application requires XY,000 KB of memory. It's probably no

by thorax 17y ago

Sure, except when that doesn't make any sense. I embed Python in my application. My highly graphical application requires XY,000 KB of memory. It's probably not wise to re-invoke my entire app Z times simply so that Python can properly take advantage of the user's multi-core OS while scripting inside my application. Any time you embed Python (e.g. a game, a browser, etc), the multiprocessing support isn't going to work very well as it is today. Ideally we get to the point where threading works great in the core interpreter. I asked Guido about this in front of his keynote audience a few Pycons ago, and his answer was a sad shrug and saying to use Jython or IronPython instead. I'd love to see the GIL improved/removed because CPython-embedding apps don't have the same options as everyday pure scripting.

kilowatt 17y ago
Just because your frontend uses XY,000 KB of memory doesn't mean that worker processes have to too. The working set size for a fresh Python 2.6 run on my Windows 7 box is 6,620KB. See Chrome for a good example of this approach :)
- tentonova2 17y ago
  Chrome's approach was not chosen because of its general purpose suitability as a multi-core processing model, but because of the security advantages of relying on OS process separation in a web browser. Chrome is a very specific use case with unusual requirements. If you take anything away from the Chrome example, it would be that securely replicating the functionality easily available from shared-memory threads in a multi-processing model is both very complicated and not very portable.
- thorax 17y ago
  We're talking about the multiprocessing package/modules in Python: http://docs.python.org/library/multiprocessing.html http://docs.python.org/library/multiprocessing.html It uses fork under the covers, so small worker processes aren't an option for that module (unless we forego interpreter embedding entirely which isn't the point). There are other techniques for spawning processes to do some external process work, but those, of course, need entirely separate plumbing.
scott_s 17y ago
I agree with you, but in practice, there won't be as much memory replication as you assume. Linux (and I assume most modern OSes) implement copy-on-write: no data is copied until a write occurs. So as long as the separate process only read that large data set, that large data set should exist in memory only once.
- FooBarWidget 17y ago
  Exception multiprocessing does not use fork() (without exec()) because it has to be compatible with Windows.
  - rbanffy 17y ago
    Like I use to say: that's Microsoft holding everybody back. ;-) Now, seriously, can we implement a better multiprocessing module for those with modern OSs and leave a less functional one (one that will raise a LesserOSError when certain functionality is invoked)?
  - jnoller 17y ago
    Incorrect; On linux and OS/X it uses fork. On windows it uses a best-effort clone.
malkia 17y ago
In case with Lua - I can simply create new context, and voila a whole new lua vm is there. Not sure whether that could be done with embeddable python.
- est 17y ago
  I guess because python vm is 2MB while Lua is just hundereds of kilobytes
- codexon 17y ago
  Does Lua also have a GIL?
  - silentbicycle 17y ago
    No, it doesn't. The whole interpreter was designed from the beginning to be embedded in another (typically C or C++) project, and they didn't want to have any concurrency infrastructure in the core language that would clash with the surrounding project's design. (It has excellent support for coroutines, though.) Rather than having static variables and a GIL, the interpreter keeps all its state behind an opaque pointer to a VM context. The VM is quite lightweight (a few hundred K), and running in each thread or process is not a big deal. With forking, you even share the standard libraries. In practice, this just means that every call to the Lua C API starts with ("lua_State *L"). Not a big deal. Lua also has real tail-calls, closures, and lambdas. It's a great language. Rather like a smaller, cleaner, less opinionated Python.
    - malkia 17y ago
      Also lua right now has the one of the best JIT's out there (luajit). The guy behind the project is simply genius. Really good stuff. And also kept small. It's x2-x3 faster than V8 on specific benchmarks that I've did (mainly dealing with 3d model meshes and modifying them). It's only for x86 right now, but he's working on x64 port.
      - silentbicycle 17y ago

by thorax 17y ago

kilowatt 17y ago

Just because your frontend uses XY,000 KB of memory doesn't mean that worker processes have to too. The working set size for a fresh Python 2.6 run on my Windows 7 box is 6,620KB. See Chrome for a good example of this approach :)

tentonova2 17y ago
Chrome's approach was not chosen because of its general purpose suitability as a multi-core processing model, but because of the security advantages of relying on OS process separation in a web browser. Chrome is a very specific use case with unusual requirements. If you take anything away from the Chrome example, it would be that securely replicating the functionality easily available from shared-memory threads in a multi-processing model is both very complicated and not very portable.
thorax 17y ago
We're talking about the multiprocessing package/modules in Python: http://docs.python.org/library/multiprocessing.html http://docs.python.org/library/multiprocessing.html It uses fork under the covers, so small worker processes aren't an option for that module (unless we forego interpreter embedding entirely which isn't the point). There are other techniques for spawning processes to do some external process work, but those, of course, need entirely separate plumbing.

scott_s 17y ago

I agree with you, but in practice, there won't be as much memory replication as you assume. Linux (and I assume most modern OSes) implement copy-on-write: no data is copied until a write occurs. So as long as the separate process only read that large data set, that large data set should exist in memory only once.

FooBarWidget 17y ago
Exception multiprocessing does not use fork() (without exec()) because it has to be compatible with Windows.
- rbanffy 17y ago
  Like I use to say: that's Microsoft holding everybody back. ;-) Now, seriously, can we implement a better multiprocessing module for those with modern OSs and leave a less functional one (one that will raise a LesserOSError when certain functionality is invoked)?
- jnoller 17y ago
  Incorrect; On linux and OS/X it uses fork. On windows it uses a best-effort clone.

malkia 17y ago

In case with Lua - I can simply create new context, and voila a whole new lua vm is there. Not sure whether that could be done with embeddable python.

est 17y ago
I guess because python vm is 2MB while Lua is just hundereds of kilobytes
codexon 17y ago
Does Lua also have a GIL?
- silentbicycle 17y ago
  No, it doesn't. The whole interpreter was designed from the beginning to be embedded in another (typically C or C++) project, and they didn't want to have any concurrency infrastructure in the core language that would clash with the surrounding project's design. (It has excellent support for coroutines, though.) Rather than having static variables and a GIL, the interpreter keeps all its state behind an opaque pointer to a VM context. The VM is quite lightweight (a few hundred K), and running in each thread or process is not a big deal. With forking, you even share the standard libraries. In practice, this just means that every call to the Lua C API starts with ("lua_State *L"). Not a big deal. Lua also has real tail-calls, closures, and lambdas. It's a great language. Rather like a smaller, cleaner, less opinionated Python.
  - malkia 17y ago
    Also lua right now has the one of the best JIT's out there (luajit). The guy behind the project is simply genius. Really good stuff. And also kept small. It's x2-x3 faster than V8 on specific benchmarks that I've did (mainly dealing with 3d model meshes and modifying them). It's only for x86 right now, but he's working on x64 port.
    - silentbicycle 17y ago