6 ms·

> Does this really count as mutability though? Not really. The string would be thrown away anyway so this is just an optimization to reuse the memory. The impor

by js2 6y ago

> Does this really count as mutability though? Not really. The string would be thrown away anyway so this is just an optimization to reuse the memory. The important takeaway here is that you are not allocating a new string every single time like the internet says. This is an undocumented optimization, so you should assume you're allocating a new sting every single time like the internet says. I've been coding Python since 1.5.2 days and so I'll continue to use lists as intermediate string builders and then join afterwards because I know this works in past, present, a likely future versions of Python.

nine_k 6y ago
This does play a role if for some reason you use object IDs, e.g. when deserializing data with cross-links. Keeping a real second reference becomes important.
- deleted 6y ago
  [deleted]
- wizzwizz4 6y ago
  You should never use object IDs that way; that's what weak references / dictionaries are for. (See pickle for details.)
- heavenlyblue 6y ago
  Python docs explicitly say that object IDs are only unique for objects with overlapping lifetimes. Congrats, by trying to be smart you just shot yourself in the foot.
lostcolony 6y ago
Yeah; I'm pretty sure this isn't anything you can take advantage of. But I think the point of the post is a "isn't this neat?" rather than "because of an optimization one approach isn't as bad as it could be and you should take advantage of that".
- masklinn 6y ago
  > Yeah; I'm pretty sure this isn't anything you can take advantage of. You probably can but should not rely on it: * it was a very official part of the Python 2.4 release notes * it's unlikely the devs would remove it as they know how implementation details tend to leak into the software (see: dict ordering) * but it is an optimisation so there's no guarantee, a simple trace function could break it * and obviously there's no guarantee (and likely no way) other implementations can implement it
  - dmw_ng 6y ago
    The optimization got moved to unicode objects during the great Python 3 upheaval and AFAIK still remains unimplemented for what are now bytes objects. The paramiko SSH library relied on it heavily, as a result its runtime is (AFAIK still) horrific on Python 3
  - lostcolony 6y ago
    Right, but my thought is "at what times would it make sense to join, and not append, but this optimization would be faster than joining"? I'm guessing not very many. Because, yeah, the fact that str += "?" is faster isn't a big deal; that's the natural, ergonomic way to deal with it, because you're appending all of one string. Likewise foo + " " + bar + "?" is probably easier to write that way, than to drop them into a list and join (but even if not, I'd be curious if it's actually any faster; this article doesn't measure). By the time you get to joining large amounts together, concatenating a CSV or something, you're going to use join, naturally, and join is going to be more performant. So to my mind it's kind of an open question, at what points is the non-ergonomic thing going to be the faster thing? That's what "taking advantage of (an optimization)" feels like; otherwise you're just writing code and letting the performance fall where it may.
    - masklinn 6y ago
      > Right, but my thought is "at what times would it make sense to join, and not append, but this optimization would be faster than joining"? I'm guessing not very many. I’m guessing all of them. Because what it does is what join will do internally anyway, but without the overhead of the list itself, and with operations which have lower overhead than the list’s.
orf 6y ago
Some interesting results. Joining on a generator is slower than joining on a list because the list has a known size beforehand, which lets Python perform some optimizations. Even though it's more memory intensive. Appending a string is surprisingly fast. Building an intermediate list is not. While I do get your point, Python likes making optimizations. They rarely, if ever, make patterns slower by choice. There's nothing in any spec that says it _has_ to be slow: that's as much as an implementation detail as "joining lists are fast". In [18]: sys.version Out[18]: '3.9.1 (default, Feb 3 2021, 07:38:02) \n[Clang 12.0.0 (clang-1200.0.32.29)]' In [6]: strings = ["".join(random.choice(string.hexdigits) for _ in range(10)) for _ in range(10)] In [9]: def one(): ...: "".join(s for s in strings) ...: In [10]: def two(): ...: "".join([s for s in strings]) ...: In [11]: def three(): ...: x = [] ...: for s in strings: ...: x.append(s) ...: "".join(x) ...: In [12]: def four(): ...: x = "" ...: for s in strings: ...: x += s ...: In [13]: %timeit one() 753 ns ± 9 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) In [14]: %timeit two() 521 ns ± 5.63 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) In [15]: %timeit three() 696 ns ± 3.94 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) In [16]: %timeit four() 620 ns ± 4.82 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
- ufo 6y ago
  I agree with you that newer versions don't tend to get rid of optimizations. However, other python implementations might not optimize the concatenation. IIRC, pypy doesn't.
  - dr_zoidberg 6y ago
    You can write python to be performant for either CPython or for PyPy. It's very difficult to write code that is performant on both. If you are targetting PyPy, be explicit about it and go full RPython on it. If you are targetting CPython, try to be "pythonic"[0] and then drop to C/++ (through numba, cython, nuitka, pure C or whatever way you like best) if your performance requirements need it. I'm not familiar with Jython and IronPython innards to know what's the best way to handle performance for them. Most probably it's very different than CPython and PyPy. So like you said, keep things reasonable and don't depend on implementation details--unless you are explicit in your target/supported interpreter and you absolutely need to squeeze the performance out of the code.
Dylan16807 6y ago
Does python document anything about performance? I assume my array sort won't be n^3, but how do we draw the line on what to assume and what not to assume?
- jshprentz 6y ago
  Python uses Timsort[1]. They provide benchmarks and details at [2]. [1] https://en.wikipedia.org/wiki/Timsort https://en.wikipedia.org/wiki/Timsort [2] https://svn.python.org/projects/python/trunk/Objects/listsort.txt https://svn.python.org/projects/python/trunk/Objects/listsor...
  - Dylan16807 6y ago
    They also give details about the string implementation. I mean a guarantee that it meets some baseline efficiency that you can depend on across time and different implementations. Does that exist?
- rovr138 6y ago
  If you're implementing it on python, check `dis.dis()` If builtin, then you'll need to check the code for cpython or whatever you're using.
- dmurray 6y ago
  The closest thing Python has to a specification is the Language Reference. It shies well away from performance details and aims to describe "syntax and core semantics" while acknowledging that some parts of what it specifies may just be implementation details of CPython [0]. To draw the line on what you can assume re performance, you need a combination of background knowledge, folk wisdom and common sense. (You could say "or read the code" but that could change in the next minor version: you still need common sense to know if that's likely). This is unlike, e.g. C++ where there is a specification and standards-compliant compilers must implement certain functions with the specified runtime complexity. [0] https://docs.python.org/3/reference/introduction.html#alternate-implementations https://docs.python.org/3/reference/introduction.html#altern...
- 6y ago
mannykannot 6y ago
>> Does this really count as mutability though? Not really. The string would be thrown away anyway so this is just an optimization to reuse the memory. The important takeaway here is that you are not allocating a new string every single time like the internet says. The important takeaway is that semantic immutability does not require or depend on implementation immutability. It's a 'levels of abstraction' thing. Once you let the users peek behind the curtain, however, it becomes much more difficult to be sure they cannot shoot themselves in the foot by so doing.
helsinkiandrew 6y ago
> Does this really count as mutability though? No. The strings having the same address doesn't mean they're the same strings. This is an interesting optimization but one Python string is being replaced by another string at the same address. Even without this optimization the address will be reused eventually.

by js2 6y ago

nine_k 6y ago

This does play a role if for some reason you use object IDs, e.g. when deserializing data with cross-links. Keeping a real second reference becomes important.

deleted 6y ago
[deleted]
wizzwizz4 6y ago
You should never use object IDs that way; that's what weak references / dictionaries are for. (See pickle for details.)
heavenlyblue 6y ago
Python docs explicitly say that object IDs are only unique for objects with overlapping lifetimes. Congrats, by trying to be smart you just shot yourself in the foot.

lostcolony 6y ago

Yeah; I'm pretty sure this isn't anything you can take advantage of. But I think the point of the post is a "isn't this neat?" rather than "because of an optimization one approach isn't as bad as it could be and you should take advantage of that".

masklinn 6y ago
> Yeah; I'm pretty sure this isn't anything you can take advantage of. You probably can but should not rely on it: * it was a very official part of the Python 2.4 release notes * it's unlikely the devs would remove it as they know how implementation details tend to leak into the software (see: dict ordering) * but it is an optimisation so there's no guarantee, a simple trace function could break it * and obviously there's no guarantee (and likely no way) other implementations can implement it
- dmw_ng 6y ago
  The optimization got moved to unicode objects during the great Python 3 upheaval and AFAIK still remains unimplemented for what are now bytes objects. The paramiko SSH library relied on it heavily, as a result its runtime is (AFAIK still) horrific on Python 3
- lostcolony 6y ago
  Right, but my thought is "at what times would it make sense to join, and not append, but this optimization would be faster than joining"? I'm guessing not very many. Because, yeah, the fact that str += "?" is faster isn't a big deal; that's the natural, ergonomic way to deal with it, because you're appending all of one string. Likewise foo + " " + bar + "?" is probably easier to write that way, than to drop them into a list and join (but even if not, I'd be curious if it's actually any faster; this article doesn't measure). By the time you get to joining large amounts together, concatenating a CSV or something, you're going to use join, naturally, and join is going to be more performant. So to my mind it's kind of an open question, at what points is the non-ergonomic thing going to be the faster thing? That's what "taking advantage of (an optimization)" feels like; otherwise you're just writing code and letting the performance fall where it may.
  - masklinn 6y ago
    > Right, but my thought is "at what times would it make sense to join, and not append, but this optimization would be faster than joining"? I'm guessing not very many. I’m guessing all of them. Because what it does is what join will do internally anyway, but without the overhead of the list itself, and with operations which have lower overhead than the list’s.

orf 6y ago

Some interesting results. Joining on a generator is slower than joining on a list because the list has a known size beforehand, which lets Python perform some optimizations. Even though it's more memory intensive. Appending a string is surprisingly fast. Building an intermediate list is not. While I do get your point, Python likes making optimizations. They rarely, if ever, make patterns slower by choice. There's nothing in any spec that says it _has_ to be slow: that's as much as an implementation detail as "joining lists are fast". In [18]: sys.version Out[18]: '3.9.1 (default, Feb 3 2021, 07:38:02) \n[Clang 12.0.0 (clang-1200.0.32.29)]' In [6]: strings = ["".join(random.choice(string.hexdigits) for _ in range(10)) for _ in range(10)] In [9]: def one(): ...: "".join(s for s in strings) ...: In [10]: def two(): ...: "".join([s for s in strings]) ...: In [11]: def three(): ...: x = [] ...: for s in strings: ...: x.append(s) ...: "".join(x) ...: In [12]: def four(): ...: x = "" ...: for s in strings: ...: x += s ...: In [13]: %timeit one() 753 ns ± 9 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) In [14]: %timeit two() 521 ns ± 5.63 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) In [15]: %timeit three() 696 ns ± 3.94 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) In [16]: %timeit four() 620 ns ± 4.82 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

ufo 6y ago
I agree with you that newer versions don't tend to get rid of optimizations. However, other python implementations might not optimize the concatenation. IIRC, pypy doesn't.
- dr_zoidberg 6y ago
  You can write python to be performant for either CPython or for PyPy. It's very difficult to write code that is performant on both. If you are targetting PyPy, be explicit about it and go full RPython on it. If you are targetting CPython, try to be "pythonic"[0] and then drop to C/++ (through numba, cython, nuitka, pure C or whatever way you like best) if your performance requirements need it. I'm not familiar with Jython and IronPython innards to know what's the best way to handle performance for them. Most probably it's very different than CPython and PyPy. So like you said, keep things reasonable and don't depend on implementation details--unless you are explicit in your target/supported interpreter and you absolutely need to squeeze the performance out of the code.

Dylan16807 6y ago

Does python document anything about performance? I assume my array sort won't be n^3, but how do we draw the line on what to assume and what not to assume?

jshprentz 6y ago
Python uses Timsort[1]. They provide benchmarks and details at [2]. [1] https://en.wikipedia.org/wiki/Timsort https://en.wikipedia.org/wiki/Timsort [2] https://svn.python.org/projects/python/trunk/Objects/listsort.txt https://svn.python.org/projects/python/trunk/Objects/listsor...
- Dylan16807 6y ago
  They also give details about the string implementation. I mean a guarantee that it meets some baseline efficiency that you can depend on across time and different implementations. Does that exist?
rovr138 6y ago
If you're implementing it on python, check `dis.dis()` If builtin, then you'll need to check the code for cpython or whatever you're using.
dmurray 6y ago
The closest thing Python has to a specification is the Language Reference. It shies well away from performance details and aims to describe "syntax and core semantics" while acknowledging that some parts of what it specifies may just be implementation details of CPython [0]. To draw the line on what you can assume re performance, you need a combination of background knowledge, folk wisdom and common sense. (You could say "or read the code" but that could change in the next minor version: you still need common sense to know if that's likely). This is unlike, e.g. C++ where there is a specification and standards-compliant compilers must implement certain functions with the specified runtime complexity. [0] https://docs.python.org/3/reference/introduction.html#alternate-implementations https://docs.python.org/3/reference/introduction.html#altern...
6y ago

mannykannot 6y ago

>> Does this really count as mutability though? Not really. The string would be thrown away anyway so this is just an optimization to reuse the memory. The important takeaway here is that you are not allocating a new string every single time like the internet says. The important takeaway is that semantic immutability does not require or depend on implementation immutability. It's a 'levels of abstraction' thing. Once you let the users peek behind the curtain, however, it becomes much more difficult to be sure they cannot shoot themselves in the foot by so doing.

helsinkiandrew 6y ago

> Does this really count as mutability though? No. The strings having the same address doesn't mean they're the same strings. This is an interesting optimization but one Python string is being replaced by another string at the same address. Even without this optimization the address will be reused eventually.