7 ms·
If I have a string x = "this is a very long string..." and do y = x[:10], then it's a whole new string? If x is near my memory limits, and I do y = x[:-1] will
by ianhorn 6y ago
If I have a string x = "this is a very long string..." and do y = x[:10], then it's a whole new string? If x is near my memory limits, and I do y = x[:-1] will it basically double my memory usage? Is that what you meant by every string is a new string?
- deleted 6y ago[deleted]
- sadfklsjlkjwt 6y agoYes.
- ORioN63 6y agoYes. Although you can use `islice` from itertools to get around this problem, when a problem.
- BiteCode_dev 6y agoIf x is near your memory limits, and you do y = x[:-1], you will get a MemoryError :) For those situations, bytes() + memoryview() or bytearray() can be used, but then you are on your own.
- ianhorn 6y agoHuh, I've had a wrong understanding of that for over a decade! TIL, thanks.
- BiteCode_dev 6y agoHey! https://xkcd.com/1053/ https://xkcd.com/1053/ And honestly, I would be rich if I got a dollar every time a student does this: msg.upper() Instead of: msg = msg.upper() And then call me to say it doesn't work.
- cesarb 6y agoIMO, this is a defect in the language: the lack of a "must_use" annotation or similar. If that annotation existed, and the .upper() method was annotated with it, the compiler could warn in that situation.
- diarrhea 6y agoBut you are free to do if title == user_input.upper(): That is, you convert a string to upper without binding the result to a name. You just use it in-place and discard the result, which is fine. With compiler, you mean mypy or linters?
- TkTech 6y agoThat's still "using" the resulting value for a comparison. CPython isn't an optimizing compiler, or it would completely remove the call to upper(). >>> def up(v): ... v.upper() ... >>> dis.dis(up) 2 0 LOAD_FAST 0 (v) 2 LOAD_METHOD 0 (upper) 4 CALL_METHOD 0 6 POP_TOP 8 LOAD_CONST 0 (None) 10 RETURN_VALUE >>> def up(v): ... if v.upper() == "HelloWorld": ... return True ... >>> dis.dis(up) 2 0 LOAD_FAST 0 (v) 2 LOAD_METHOD 0 (upper) 4 CALL_METHOD 0 6 LOAD_CONST 1 ('HelloWorld') 8 COMPARE_OP 2 (==) 10 POP_JUMP_IF_FALSE 16 3 12 LOAD_CONST 2 (True) 14 RETURN_VALUE >> 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Notice in the first example, right after CALL_METHOD the return value on the stack is just immediately POP'd away. The parent is saying that when you run `python example.py` CPython should see that the return value is never used and emit a warning. This would only happen because `upper()` was manually marked using the suggested `must_use` annotation.
- delaaxe 6y agoHe meant that writing a line of code with only contents: msg.upper() should trigger a warning as this clearly doesn't do anything.
- Erlangen 6y agoSlicing in Python always create a new object. You can test it with a list of integers..
- ianhorn 6y agoIt'll always create a new object but my understanding is that at least in numpy the new and old object will share memory. Am I wrong there too?
- dialamac 6y agoCPython is pretty terrible. Numpy has the concept of views, cpython doesn’t do anything sophisticated.
- aidos 6y agoCorrect. In Numpy the slices are views on the underlying memory. That’s why they’re so fast, there’s no copying involved. Incidentally that’s also why freeing up the original variable doesn’t release the memory (the slices are still using it).
- masklinn 6y ago> Slicing in Python always create a new object. It always creates a new object but it doesn't necessarily copy the contents (even shallowly). For instance slicing a `memoryview` creates a subview which shares storage with its parent.
- cecilpl2 6y agoMy favorite example of something similar to this, since you brought it up: >>> a = [254, 255, 256, 257, 258] >>> b = [254, 255, 256, 257, 258] >>> for i in range(5): print(a[i] is b[i]) ... True True True False False In Python, integers in the range [-5, 256] are statically constructed in the interpreter and refer to fixed instances of objects. All other integers are created dynamically and refer to a new object each time they are created.
- mark-r 6y ago
- masklinn 6y ago> If I have a string x = "this is a very long string..." and do y = x[:10], then it's a whole new string? Yes. And doing otherwise is pretty risky as the Java folks discovered, ultimately deciding to revert the optimisation of substring sharing storage rather than copying its data. The issue is that while data-sharing substringing is essentially free, it also keeps the original string alive, so if you slurp in megabytes, slice out a few bytes you keep around and "throw away" the original string, that one is still kept alive by the substringing you perform, and you basically have a hard to diagnose memory leak due to completely implicit behaviour. Languages which perform this sharing explicitly — and especially statically (e.g. Rust) — don't have this issue, but it's a risky move when you only have one string type. Incidentally, Python provides for opting into that behaviour for bytes using memory views.
- a1369209993 6y ago> it also keeps the original string alive, so if you slurp in megabytes, slice out a few bytes you keep around and "throw away" the original string, that one is still kept alive by the substringing you perform, and you basically have a hard to diagnose memory leak due to completely implicit behaviour. You can get around that with a smarter garbage collector, though. On every mark-sweep pass (which you need for cycle detection even if you use refcounts for primary cleanup), add up the number of bytes of distinct string objects using the buffer. If it's less than the size of the buffer, you can save memory by transparently mutating each substring to use it's own buffer. If it's not, then you actually are saving memory by sharing storage, so you should probably keep doing that.
- Doxin 6y agoThe cpython mark-and-sweep garbage collector does very little on purpose. It basically only gets involved for reference cycles. Anything else is dealt with by reference counting. This way you prevent long GC pauses.
- a1369209993 6y agoTrue, but that's by no means a inherent characteristic of garbage collectors, or even garbage collectors operating on objects of a python implementation in particular.