5 ms·

The drawback with multiple processes is that they each have all the compiled bytecode on their heap. When I "import nltk" my heap goes from ~12 to ~36 MB. Add a

The drawback with multiple processes is that they each have all the compiled bytecode on their heap. When I "import nltk" my heap goes from ~12 to ~36 MB. Add a few more dependencies and you end up wasting a non-trivial amount of RAM on python heaps.

lobster_johnson 15y ago
You could do the import and then fork the child processes. That should share all the memory used by the nitk module between all the children (for as long as the memory is not modified by anyone, which triggers a copy-on-write allocation of the affected pages).
- baq 15y ago
  most forks do copy-on-write and every access to a python object meddles with reference count, which is - you guessed it - a write.
  - froydnj 15y ago
    The context of the copy-on-write win was the bytecode for modules. I don't know that you'd have anybody meddling with the reference counts for that...but I haven't really looked.
    - rayiner 15y ago
      Bytecode is stored in function objects which IIRC are reference-counted.
      - lobster_johnson 15y ago
        Aside from ad-hoc classes (defined within the scope of a function, say), does it ever make sense to garbage-collect a module or a class? Say you do "import smtplib" in your main file. Now that module is imported -- forever. I don't know the internals of Python well enough, but I bet that the module reference has strong references to its contents, so that even if nobody is actually calling anything in smtplib, it will be there in case someone does. The same should be true about modules importing other modules; they stay visible at the scope-level, so they are permanently loaded. So for those cases it would make sense to keep them separate from global garbage collection. I'm pretty sure that the method tables of all the classes in the system take up considerable space, probably in the order of megabytes for many apps.
witten 15y ago
That's a valid point, but you can do things to mitigate this, like splitting up your program into processes that have different dependencies. For instance, if you have a UI process and a core logic process, you won't need to import your giant UI library twice.
- adambyrtek 15y ago
  If you split the code horizontally instead of vertically you have to deal with IPC, which could be problematic (and slow). I would do that only if there was already a clearly defined interface between the modules, not just for the sake of concurrency. Moreover, taking into account growing number of cores in modern machines such strategy doesn't seem very future-proof.
- rayiner 15y ago
  Or you could just use a language whose implementation doesn't blow.
  - FooBarWidget 15y ago
    And suffer from 5x slower development time. Yeah your deadline now has to be extended from 2012 to 2015 but at least the language is FAST and is technically excellent, right?
    - Peaker 15y ago
      False dichotomy...

lobster_johnson 15y ago

You could do the import and then fork the child processes. That should share all the memory used by the nitk module between all the children (for as long as the memory is not modified by anyone, which triggers a copy-on-write allocation of the affected pages).

baq 15y ago
most forks do copy-on-write and every access to a python object meddles with reference count, which is - you guessed it - a write.
- froydnj 15y ago
  The context of the copy-on-write win was the bytecode for modules. I don't know that you'd have anybody meddling with the reference counts for that...but I haven't really looked.
  - rayiner 15y ago
    Bytecode is stored in function objects which IIRC are reference-counted.
    - lobster_johnson 15y ago
      Aside from ad-hoc classes (defined within the scope of a function, say), does it ever make sense to garbage-collect a module or a class? Say you do "import smtplib" in your main file. Now that module is imported -- forever. I don't know the internals of Python well enough, but I bet that the module reference has strong references to its contents, so that even if nobody is actually calling anything in smtplib, it will be there in case someone does. The same should be true about modules importing other modules; they stay visible at the scope-level, so they are permanently loaded. So for those cases it would make sense to keep them separate from global garbage collection. I'm pretty sure that the method tables of all the classes in the system take up considerable space, probably in the order of megabytes for many apps.

witten 15y ago

That's a valid point, but you can do things to mitigate this, like splitting up your program into processes that have different dependencies. For instance, if you have a UI process and a core logic process, you won't need to import your giant UI library twice.

adambyrtek 15y ago
If you split the code horizontally instead of vertically you have to deal with IPC, which could be problematic (and slow). I would do that only if there was already a clearly defined interface between the modules, not just for the sake of concurrency. Moreover, taking into account growing number of cores in modern machines such strategy doesn't seem very future-proof.
rayiner 15y ago
Or you could just use a language whose implementation doesn't blow.
- FooBarWidget 15y ago
  And suffer from 5x slower development time. Yeah your deadline now has to be extended from 2012 to 2015 but at least the language is FAST and is technically excellent, right?
  - Peaker 15y ago
    False dichotomy...