3 ms·

Can you link to any article about this? I haven't been able to find a good read on this subject.

by jamra 13y ago

aaronblohowiak 13y ago
if you have a very large slice of Structs and then you end up with only a pointer to one of the Structs, the entire slice will be retained. This is also true for slices of Byte. For non-gophers, a slice is roughly a bounds-checked Vector.
- jamra 13y ago
  I get the idea, but I am wondering more on what kind of code will cause too much data to be retained and how to code around it. What you just described is how memory is allocated, but can this cause it to be retained outside a function body?
  - Someone 13y ago
    A typical use case in Java is/was (1) reading a huge file line by line or parsing it as XML, and filtering it. Java's substring is like go's slices in that it is constant-time, but retains the entire string. Consequently, your code that filters out 1‰ of a 1GB XML file may keep hold of the entire file's content. In Java, you code around it by doing new String( line.substring( x, y); String's constructor will copy the characters to a new buffer. I guess go will have something similar. Problem, of course, is that that means copying the objects. That can especially be an issue if the objects are mutable. An alternative in Java is to do s = s.intern(); But that comes with its own problems. 'intern()' decreases memory usage, but is slow, and library writers cannot know whether it is a good idea to use it or not because they cannot know what is more important to the code calling the library: speed or memory. (1) Oracle is going to change/has changed this after measuring its impact. It turns out that the two extra fields that each string needs to keep track of what part of a buffer contains its characters more or less compensate for the decrease in memory usage that sharing strings brings.
    - MBCook 13y ago
      Interning had other problems. Until some of the more recent JVMs, interning strings was permenant. The memory could never be reclaimed until you restarted the JVM. It was basically a programmer defined memory leak. Why? My understanding is that when you interned a string you were given something like a constant token to reference it. This did two things: 1. Equality was trivial. Instead of taking to strings and comparing each character, it could just see if the constants were equal. For strings you know you'll see often this could be great source of speedups. 2. You could end up saving a ton of memory. Let's say that string you're interning is 250 character long. Instead of taking up 250 unicode characters plus overhead, it's now... let's say... 8 bytes. Maybe you saved a few hundred bytes. Now multiply a few hundred bytes by a few fields times 10 million records in memory and that's nothing to sneeze at. Interning is definitely optimization and not part of normal Java development.

Can you link to any article about this? I haven't been able to find a good read on this subject.

by jamra 13y ago

Can you link to any article about this? I haven't been able to find a good read on this subject.

if you have a very large slice of Structs and then you end up with only a pointer to one of the Structs, the entire slice will be retained. This is also true for slices of Byte. For non-gophers, a slice is roughly a bounds-checked Vector.

jamra 13y ago
I get the idea, but I am wondering more on what kind of code will cause too much data to be retained and how to code around it. What you just described is how memory is allocated, but can this cause it to be retained outside a function body?
- Someone 13y ago
  A typical use case in Java is/was (1) reading a huge file line by line or parsing it as XML, and filtering it. Java's substring is like go's slices in that it is constant-time, but retains the entire string. Consequently, your code that filters out 1‰ of a 1GB XML file may keep hold of the entire file's content. In Java, you code around it by doing new String( line.substring( x, y); String's constructor will copy the characters to a new buffer. I guess go will have something similar. Problem, of course, is that that means copying the objects. That can especially be an issue if the objects are mutable. An alternative in Java is to do s = s.intern(); But that comes with its own problems. 'intern()' decreases memory usage, but is slow, and library writers cannot know whether it is a good idea to use it or not because they cannot know what is more important to the code calling the library: speed or memory. (1) Oracle is going to change/has changed this after measuring its impact. It turns out that the two extra fields that each string needs to keep track of what part of a buffer contains its characters more or less compensate for the decrease in memory usage that sharing strings brings.
  - MBCook 13y ago
    Interning had other problems. Until some of the more recent JVMs, interning strings was permenant. The memory could never be reclaimed until you restarted the JVM. It was basically a programmer defined memory leak. Why? My understanding is that when you interned a string you were given something like a constant token to reference it. This did two things: 1. Equality was trivial. Instead of taking to strings and comparing each character, it could just see if the constants were equal. For strings you know you'll see often this could be great source of speedups. 2. You could end up saving a ton of memory. Let's say that string you're interning is 250 character long. Instead of taking up 250 unicode characters plus overhead, it's now... let's say... 8 bytes. Maybe you saved a few hundred bytes. Now multiply a few hundred bytes by a few fields times 10 million records in memory and that's nothing to sneeze at. Interning is definitely optimization and not part of normal Java development.