3 ms·

can you explain how generators work with multiprocess (Thread based pool) ? is ps internal variable unique for each Thread or same? is it safe to execute your

by slt2021 3y ago

can you explain how generators work with multiprocess (Thread based pool) ? is ps internal variable unique for each Thread or same? is it safe to execute your primes() from different threads?

TrianguloY 3y ago
A yield will simply return a generator object, which contains information about the next value to use, and how to continue the function execution. That's why you need to use functions that yield things inside loops or list(...). If you run it from different threads I guess it will be the same as calling the function multiple times, it will return a new started-from-the-top generator. def sum(): yield 1 yield 2 print(repr(sum())) print(next(sum())) print(next(sum())) Prints <generator object sum at 0x7fc6f14823c0> 1 1
- slt2021 3y ago
  so Thread based based pool will have same instance of generator, while Process based pool with have unique instance of generator?
  - TrianguloY 3y ago
    I don't know if a generator can be shared across threads, but in that case ... I have no idea :/ You'll need to search, or try!
  - kzrdude 3y ago
    In this example, calling sum() creates a generator and returns it. Say g = sum(). If you share g between threads, they will all use the same generator object! If you call sum() separately per thread, they will be different generators. If you try to send g to a different process, you will get an error, because it doesn't serialize.
qsort 3y ago
> can you explain how generators work with multiprocess The best way to think of a generator is as an object implementing the iteration protocol. They don't really interact with concurrency, as far as multiprocess is concerned, they're just regular objects. So the answer is that it depends on how you plan to share memory between the processes. > is ps internal variable unique for each Thread or same? ps is local to the generator instance. def f(): x = 0 while True: yield (x := x + 1) >>> f() <generator object f at 0x10412e500> >>> x = f() >>> y = f() >>> next(x) 1 >>> next(x) 2 >>> next(y) 1 > is it safe to execute your primes() from different threads? For this specific generator, you would run into the GIL. More generally, if you're talking about non CPU-bound operations, you need to synchronize the threads. It's worth looking into asyncio for those use cases.

can you explain how generators work with multiprocess (Thread based pool) ? is ps internal variable unique for each Thread or same? is it safe to execute your primes() from different threads?

TrianguloY 3y ago

A yield will simply return a generator object, which contains information about the next value to use, and how to continue the function execution. That's why you need to use functions that yield things inside loops or list(...). If you run it from different threads I guess it will be the same as calling the function multiple times, it will return a new started-from-the-top generator. def sum(): yield 1 yield 2 print(repr(sum())) print(next(sum())) print(next(sum())) Prints <generator object sum at 0x7fc6f14823c0> 1 1

slt2021 3y ago
so Thread based based pool will have same instance of generator, while Process based pool with have unique instance of generator?
- TrianguloY 3y ago
  I don't know if a generator can be shared across threads, but in that case ... I have no idea :/ You'll need to search, or try!
- kzrdude 3y ago
  In this example, calling sum() creates a generator and returns it. Say g = sum(). If you share g between threads, they will all use the same generator object! If you call sum() separately per thread, they will be different generators. If you try to send g to a different process, you will get an error, because it doesn't serialize.

qsort 3y ago

> can you explain how generators work with multiprocess The best way to think of a generator is as an object implementing the iteration protocol. They don't really interact with concurrency, as far as multiprocess is concerned, they're just regular objects. So the answer is that it depends on how you plan to share memory between the processes. > is ps internal variable unique for each Thread or same? ps is local to the generator instance. def f(): x = 0 while True: yield (x := x + 1) >>> f() <generator object f at 0x10412e500> >>> x = f() >>> y = f() >>> next(x) 1 >>> next(x) 2 >>> next(y) 1 > is it safe to execute your primes() from different threads? For this specific generator, you would run into the GIL. More generally, if you're talking about non CPU-bound operations, you need to synchronize the threads. It's worth looking into asyncio for those use cases.