2 ms·

Newbie question: What happen with the huge number of strings created?, byte sequences are cleaned with gc but the strings take memory.

• tgv 5y ago
  First, a string object is a small object with a pointer to the actual character sequence. The tokenizer produces the same number of these for all approaches. Second, in the new approach, that character sequence is part of the original input. All string objects point to a slice of the same "backing store", unlike the old approach where each string object had its own memory with a copy from a small part of the input.
  • newbye2 5y ago
    Two questions: first, how cache friendly is that backing store? and second how the backing store is updated when the strings are garbage collected? At first sight it seems that a stack allocated byte sequence can be easily removed but a backing store allocated sequence is not easily handle nor cache friendly. The general problem seems to be about dynamic dispatch of methods, in go using fat pointers, so the author avoid the dynamic dispatch by modifing the code.
    • tgv 5y ago
      > how cache friendly is that backing store It's the original input buffer. It's less fragmented than creating small copies for every token. The backing store stays alive as long as there's a single token pointing to it. It's a bit inefficient if there's a lot of whitespace, comment, etc. in it, or if you're only interested in a very small portion of the tokens, but it doesn't have the overhead of a memory block per string (which is at least 16 bytes). So it depends, I guess. I don't think dynamic dispatch comes into play in go. The strings are represented in the same way.