3 ms·

It would be interesting to see a comparison of Java (and Python) string processing with Go. In Go you use slices refer to underlying string data, which are bas

by chubot 14y ago

It would be interesting to see a comparison of Java (and Python) string processing with Go. In Go you use slices refer to underlying string data, which are basically (pointer, length) pairs. So you may still create garbage in terms of these small slice objects, but that should cause less of a problem than creating tons of temporary immutable strings.

brazzy 14y ago
Actually, Java strings work exactly like that. And if you use them correctly, string processing is no problem at all on a modern VM using a generational GC.
- chubot 14y ago
  Ah OK I didn't know that (not a Java programmer). What happens if you create a huge 10MB string, then a 10 byte substring of it, and then remove all references to the 10MB string. Does that memory ever get reclaimed? I guess the difference in Go is that the string abstraction and the slice abstraction is exposed to the programmer, so it will be apparent if that happens. There is some info here but I was unable to find a good explanation: http://stackoverflow.com/questions/2909848/how-does-java-implement-flyweight-pattern-for-string-under-the-hood http://stackoverflow.com/questions/2909848/how-does-java-imp...
  - brazzy 14y ago
    Nope, that's one way to have a memory leak. But you can avoid the problem by using the new String(oldString) constructor - it copies only the part of the underlying char array that the String uses. You're absolutely right that it would be much better if all that were part of the API rather than an implementation detail. The SO question and answers are confusing because they talk about two competely separate issues: the above and the interning mechanism.
skybrian 14y ago
I believe in Go, slices get passed by value so you shouldn't get garbage from passing them around. Also, it's a triple rather than a pair due to the capacity.
- carbocation 14y ago
  In Go, a slice is a reference to a segment of an underlying array.
  - skybrian 14y ago
    Yes, but the reference (along with size information) is passed by value, so no garbage.
mauvehaus 14y ago
The trick with doing this in a language without reference counting (like C) is keeping track of who owns the buffer all your slices point to. Windows uses UNICODE_STRING[1] internally for a great many things, paths being one of them. There are some functions[2][3] in the FilterManager's API for parsing paths that take advantage of this so as to avoid allocating extra memory for path components that already exist in the original path. Note also the proscription against modifying the parsed path components[3] because they're shared. Since there's no support for immutable strings, you're counting on everybody to read the documentation and follow it. Admittedly, this specific restriction comes about from getting the file name information from the FilterManager in the first place, which has the explicit purpose of reducing the number of copies of paths floating around in kernel memory, but it still demonstrates the original point of needing to know who owns the memory at the end of the day. [1]http://msdn.microsoft.com/en-us/library/windows/desktop/aa380518%28v=vs.85%29.aspx http://msdn.microsoft.com/en-us/library/windows/desktop/aa38... [2]http://msdn.microsoft.com/en-us/library/windows/hardware/ff543413%28v=vs.85%29.aspx http://msdn.microsoft.com/en-us/library/windows/hardware/ff5... [3]http://msdn.microsoft.com/en-us/library/windows/hardware/ff543417%28v=vs.85%29.aspx http://msdn.microsoft.com/en-us/library/windows/hardware/ff5...

by chubot 14y ago

brazzy 14y ago

Actually, Java strings work exactly like that. And if you use them correctly, string processing is no problem at all on a modern VM using a generational GC.

chubot 14y ago
Ah OK I didn't know that (not a Java programmer). What happens if you create a huge 10MB string, then a 10 byte substring of it, and then remove all references to the 10MB string. Does that memory ever get reclaimed? I guess the difference in Go is that the string abstraction and the slice abstraction is exposed to the programmer, so it will be apparent if that happens. There is some info here but I was unable to find a good explanation: http://stackoverflow.com/questions/2909848/how-does-java-implement-flyweight-pattern-for-string-under-the-hood http://stackoverflow.com/questions/2909848/how-does-java-imp...
- brazzy 14y ago
  Nope, that's one way to have a memory leak. But you can avoid the problem by using the new String(oldString) constructor - it copies only the part of the underlying char array that the String uses. You're absolutely right that it would be much better if all that were part of the API rather than an implementation detail. The SO question and answers are confusing because they talk about two competely separate issues: the above and the interning mechanism.

skybrian 14y ago

I believe in Go, slices get passed by value so you shouldn't get garbage from passing them around. Also, it's a triple rather than a pair due to the capacity.

carbocation 14y ago
In Go, a slice is a reference to a segment of an underlying array.
- skybrian 14y ago
  Yes, but the reference (along with size information) is passed by value, so no garbage.

mauvehaus 14y ago

The trick with doing this in a language without reference counting (like C) is keeping track of who owns the buffer all your slices point to. Windows uses UNICODE_STRING[1] internally for a great many things, paths being one of them. There are some functions[2][3] in the FilterManager's API for parsing paths that take advantage of this so as to avoid allocating extra memory for path components that already exist in the original path. Note also the proscription against modifying the parsed path components[3] because they're shared. Since there's no support for immutable strings, you're counting on everybody to read the documentation and follow it. Admittedly, this specific restriction comes about from getting the file name information from the FilterManager in the first place, which has the explicit purpose of reducing the number of copies of paths floating around in kernel memory, but it still demonstrates the original point of needing to know who owns the memory at the end of the day. [1]http://msdn.microsoft.com/en-us/library/windows/desktop/aa380518%28v=vs.85%29.aspx http://msdn.microsoft.com/en-us/library/windows/desktop/aa38... [2]http://msdn.microsoft.com/en-us/library/windows/hardware/ff543413%28v=vs.85%29.aspx http://msdn.microsoft.com/en-us/library/windows/hardware/ff5... [3]http://msdn.microsoft.com/en-us/library/windows/hardware/ff543417%28v=vs.85%29.aspx http://msdn.microsoft.com/en-us/library/windows/hardware/ff5...