5 ms·

Can someone who's better at Python than me chime in, isn't the obvious transformation of def square_vals(x : list): return [val * val for val in x]

by CBLT 3y ago

Can someone who's better at Python than me chime in, isn't the obvious transformation of def square_vals(x : list): return [val * val for val in x] into a generator to instead write def square_vals(x : list): return (val * val for val in x) Does the yield syntax actually impact the performance metrics here?

jogjayr 3y ago
The only difference between your two solutions is the first one returns a list and the second, a tuple. Neither one is a generator. Edit: That was wrong. TIL if you write what looks like a tuple comprehension, it's actually a generator. To answer the original question though, if you write it as a generator with the yield keyword def square(nums: list): for num in nums: yield num * num will consume less memory because it won't have the return value in memory (the list). You can verify this by putting a print statement after the yield - it won't be executed until you call next on the generator.
- jkoudys 3y ago
  No. >>> x = (x for x in [1,2,3]) >>> x <generator object <genexpr> at 0x7bebccac10>
  - jogjayr 3y ago
    TIL. Updated, thanks.
- asplake 3y ago
  To your edit: Easy mistake to make. It's not the parentheses but rather the comma that make a tuple expression.
  - Izkata 3y ago
    Hence the "wtf is going on" result of typoing: return foo, Or this in the middle of some other lines: foo = some(bar, baz),
jkoudys 3y ago
It's basically the same. A lot of people feel it's more pythonic to spread this out across multiple lines with understandable indentation, while comprehensions can smush it all together. But the list comprehension was already there, of course.
throwaway81523 3y ago
It should be the same. I expect yield was used to make the behaviour more explicit for the reader.
el_oni 3y ago
It depends what you are doing with the value. If you are going to iterate through some of the resulting thing but not all of it then the generator means you aren't throwing away a bunch of the work that you've done. It can also be more cache friendly. It doesn't need to allocate a whole new lists worth of memory. One of the downsides of it being lazy is that if list x is mutated between when you create the generator and when you consume it then those changes are reflected in the generator. I've done some micro benchmarks and it really depends on what you are doing. Profiling it with cProfile, Pyspy or using %timeit in an Ipython shell will tell you if it makes a difference.
- el_oni 3y ago
  P.s. to make that function an actual generator you could use yield from Instead of return
  - keeganpoppen 3y ago
    the function isn't a generator, but that value is a generator expression: https://peps.python.org/pep-0289/ https://peps.python.org/pep-0289/
Jtsummers 3y ago
Running a benchmark on my system comparing your generator expression version versus their generator function shows that they perform within a small margin of each other. Yours was marginally (< 1%) faster. (not a real proper benchmark, there's quite a bit running on this computer right now but it still has a lot of free cores so should be close to the same result from a proper benchmark)
akubera 3y ago
I'd bet they have very similar performance-metrics, but the yield syntax is more extensible (i.e. you're not limited to one expression) and debug-able (you can put breakpoints within the function). Also the name and the generator is nicer (for some definition of nice): >>> def square_vals(x : list): ... return (v * v for v in x) ... >>> square_vals([1,2,3]) <generator object square_vals.<locals>.<genexpr> at 0x786b8511f5e0> >>> def square_vals_yields(x: list): ... for v in x: ... yield v * v ... >>> square_vals_yields([1,2,3]) <generator object square_vals_yields at 0x786b851f5ff0> I think it's more idiomatic to pass generator-comprehensions into functions rather than return them from functions >>> sum((v*v for v in x))
- iamcreasy 3y ago
  I think the second parenthesis is not needed to create the generator expression. Source [1] 6.2.8. Generator expressions ... The parentheses can be omitted on calls with only one argument. [1] https://docs.python.org/3/reference/expressions.html#generator-expressions https://docs.python.org/3/reference/expressions.html#generat...
igsomething 3y ago
They are different. def square_vals(x : list): return (val * val for val in x) is a function that takes a list and returns a generator. While def square_vals(x : list): for val in x: yield val * val is a generator itself. The first case creates an anonymous generator. When in doubt you can always use the dis module: https://docs.python.org/3/library/dis.html https://docs.python.org/3/library/dis.html
wfitz 3y ago
Don't forget you can chain multiple generators together too. This can have huge memory and compute time benefits with lazy evaluation depending how you structure your code. A very crude example below... # Really big generator that would probably crash as a list divmods = (divmod(x,5) for x in range(100000000000000000000000000000000)) # Create a generator from a generator remains = (remainder for n, remainder in divmods) # Runs instantly next(remains)

by CBLT 3y ago

jogjayr 3y ago

The only difference between your two solutions is the first one returns a list and the second, a tuple. Neither one is a generator. Edit: That was wrong. TIL if you write what looks like a tuple comprehension, it's actually a generator. To answer the original question though, if you write it as a generator with the yield keyword def square(nums: list): for num in nums: yield num * num will consume less memory because it won't have the return value in memory (the list). You can verify this by putting a print statement after the yield - it won't be executed until you call next on the generator.

jkoudys 3y ago
No. >>> x = (x for x in [1,2,3]) >>> x <generator object <genexpr> at 0x7bebccac10>
- jogjayr 3y ago
  TIL. Updated, thanks.
asplake 3y ago
To your edit: Easy mistake to make. It's not the parentheses but rather the comma that make a tuple expression.
- Izkata 3y ago
  Hence the "wtf is going on" result of typoing: return foo, Or this in the middle of some other lines: foo = some(bar, baz),

jkoudys 3y ago

It's basically the same. A lot of people feel it's more pythonic to spread this out across multiple lines with understandable indentation, while comprehensions can smush it all together. But the list comprehension was already there, of course.

throwaway81523 3y ago

It should be the same. I expect yield was used to make the behaviour more explicit for the reader.

el_oni 3y ago

It depends what you are doing with the value. If you are going to iterate through some of the resulting thing but not all of it then the generator means you aren't throwing away a bunch of the work that you've done. It can also be more cache friendly. It doesn't need to allocate a whole new lists worth of memory. One of the downsides of it being lazy is that if list x is mutated between when you create the generator and when you consume it then those changes are reflected in the generator. I've done some micro benchmarks and it really depends on what you are doing. Profiling it with cProfile, Pyspy or using %timeit in an Ipython shell will tell you if it makes a difference.

el_oni 3y ago
P.s. to make that function an actual generator you could use yield from Instead of return
- keeganpoppen 3y ago
  the function isn't a generator, but that value is a generator expression: https://peps.python.org/pep-0289/ https://peps.python.org/pep-0289/

Jtsummers 3y ago

Running a benchmark on my system comparing your generator expression version versus their generator function shows that they perform within a small margin of each other. Yours was marginally (< 1%) faster. (not a real proper benchmark, there's quite a bit running on this computer right now but it still has a lot of free cores so should be close to the same result from a proper benchmark)

akubera 3y ago

I'd bet they have very similar performance-metrics, but the yield syntax is more extensible (i.e. you're not limited to one expression) and debug-able (you can put breakpoints within the function). Also the name and the generator is nicer (for some definition of nice): >>> def square_vals(x : list): ... return (v * v for v in x) ... >>> square_vals([1,2,3]) <generator object square_vals.<locals>.<genexpr> at 0x786b8511f5e0> >>> def square_vals_yields(x: list): ... for v in x: ... yield v * v ... >>> square_vals_yields([1,2,3]) <generator object square_vals_yields at 0x786b851f5ff0> I think it's more idiomatic to pass generator-comprehensions into functions rather than return them from functions >>> sum((v*v for v in x))

iamcreasy 3y ago
I think the second parenthesis is not needed to create the generator expression. Source [1] 6.2.8. Generator expressions ... The parentheses can be omitted on calls with only one argument. [1] https://docs.python.org/3/reference/expressions.html#generator-expressions https://docs.python.org/3/reference/expressions.html#generat...

igsomething 3y ago

They are different. def square_vals(x : list): return (val * val for val in x) is a function that takes a list and returns a generator. While def square_vals(x : list): for val in x: yield val * val is a generator itself. The first case creates an anonymous generator. When in doubt you can always use the dis module: https://docs.python.org/3/library/dis.html https://docs.python.org/3/library/dis.html

wfitz 3y ago

Don't forget you can chain multiple generators together too. This can have huge memory and compute time benefits with lazy evaluation depending how you structure your code. A very crude example below... # Really big generator that would probably crash as a list divmods = (divmod(x,5) for x in range(100000000000000000000000000000000)) # Create a generator from a generator remains = (remainder for n, remainder in divmods) # Runs instantly next(remains)