4 ms·

Can someone explain me why something dealing with graphics needs to read: "several megabytes worth of text files"? Why aren't these binary data? (I'm not fami

Can someone explain me why something dealing with graphics needs to read: "several megabytes worth of text files"? Why aren't these binary data? (I'm not familiar with graphical / texture code) But I am familiar with fast text processing. From Java the problem is even worse: it's not so much that the methods are inefficiently written in themsevles. The issue is that string processing tends to create lots of garbage which then needs to be garbage collected. The optimization when all these strings processing methods are the bottleneck? Do not create hundreds of thousands or millions of object. Instead of doing, say, "s2 = removeLeadingAndTrailingStuff();" and then "parse(s2);" you do write your own method directly: "parseButDoNotTakeIntoAccountLeadingAndTrailingStuff()". It's oversimplified and crazy camel case is used above to make a point ; ) It's always the same : when objects creation become the bottleneck, make it so you're not creating that many objects. It's exactly what the LMAX Disruptor "pattern" is about: using gigantic arrays of primitives and hardly any object creation at all (to process 12 million events per second in Java on a single core!). That said in TFA I still don't understand why several megabytes of Unicode text files have to be processed...

skrebbel 14y ago
> Why aren't these binary data? Just guessing, but it's very coder-friendly to have particular kinds of data available as text. For example the (old, old) Wavefront .obj format is a text based 3d model format that's ridiculously easy to parse. I could imagine someone would choose to do the same with, say, entire game levels. That said, there's another class of programs that often need to read several megabytes worth of text files: compilers. Who's to say they're not parsing something programming-languagey?
jncraton 14y ago
I don't know much about DirectX, so this is from an OpenGL specific standpoint. 3d graphics applications include shaders, which are programs that get compiled by the graphics drivers and loaded directly into the graphics pipeline on the video card. The source code is standardized, but the binaries vary from card to card and driver to driver, so applications have to include the source and let the driver build it into working shaders. The text being processes is possibly shader source code.
chubot 14y ago
It would be interesting to see a comparison of Java (and Python) string processing with Go. In Go you use slices refer to underlying string data, which are basically (pointer, length) pairs. So you may still create garbage in terms of these small slice objects, but that should cause less of a problem than creating tons of temporary immutable strings.
- brazzy 14y ago
  Actually, Java strings work exactly like that. And if you use them correctly, string processing is no problem at all on a modern VM using a generational GC.
  - chubot 14y ago
    Ah OK I didn't know that (not a Java programmer). What happens if you create a huge 10MB string, then a 10 byte substring of it, and then remove all references to the 10MB string. Does that memory ever get reclaimed? I guess the difference in Go is that the string abstraction and the slice abstraction is exposed to the programmer, so it will be apparent if that happens. There is some info here but I was unable to find a good explanation: http://stackoverflow.com/questions/2909848/how-does-java-implement-flyweight-pattern-for-string-under-the-hood http://stackoverflow.com/questions/2909848/how-does-java-imp...
    - brazzy 14y ago
      Nope, that's one way to have a memory leak. But you can avoid the problem by using the new String(oldString) constructor - it copies only the part of the underlying char array that the String uses. You're absolutely right that it would be much better if all that were part of the API rather than an implementation detail. The SO question and answers are confusing because they talk about two competely separate issues: the above and the interning mechanism.
- skybrian 14y ago
  I believe in Go, slices get passed by value so you shouldn't get garbage from passing them around. Also, it's a triple rather than a pair due to the capacity.
malkia 14y ago
That's a topic that comes once in a while in our studio. Artists, generally non-technical people like to look what's inside the file. This way they can spot an extra bone in an animation, or material that should've not been used in an model. Sometimes changing the model allows for merging, or doing diff in the source control system. A more elaborate 3D diffing is what's needed, like the one shown here: http://3drepo.org/ http://3drepo.org/ - but it's not an easy thing, and not very easy to integrate either. Sometimes binary files might be bigger than text, not often but possible - for example if double is used for number it always takes 8 bytes, while as text it might be just two (0 and space). Also certain source control systems (like perforce I think) can store differences only if the files are text, otherwise they have to store full copies (that's really not a big game changer, but one more little detail to be aware). Artists, coders often would look for a material/bone by just using Find in Files/grep/etc, and although this might work for binary files, it's harder to show the data.
ww520 14y ago
That is probably similar to source code vs compiled object code. The textual graphic code is the human readable source code for describing the graphic. Reading and parsing them are similar to interpret them. To make things faster, you can read them (compile) ahead of time and write the binary objects to binary files. In runtime loading them will be fast.
aras_p 14y ago
In this case, he's working with external code (Intel's example code) that happened to use data in that way. Why - good question. I guess someone at Intel thought "hey this is just example code how to do software rasterizer for occlusion, who cares what is the data format" and went with that. Badly ;)
yock 14y ago
I would think that string immutability has much to do with the object creation. That might be why regular expressions was such a focus of the Java 1.4 release.

skrebbel 14y ago

> Why aren't these binary data? Just guessing, but it's very coder-friendly to have particular kinds of data available as text. For example the (old, old) Wavefront .obj format is a text based 3d model format that's ridiculously easy to parse. I could imagine someone would choose to do the same with, say, entire game levels. That said, there's another class of programs that often need to read several megabytes worth of text files: compilers. Who's to say they're not parsing something programming-languagey?

jncraton 14y ago

I don't know much about DirectX, so this is from an OpenGL specific standpoint. 3d graphics applications include shaders, which are programs that get compiled by the graphics drivers and loaded directly into the graphics pipeline on the video card. The source code is standardized, but the binaries vary from card to card and driver to driver, so applications have to include the source and let the driver build it into working shaders. The text being processes is possibly shader source code.

chubot 14y ago

It would be interesting to see a comparison of Java (and Python) string processing with Go. In Go you use slices refer to underlying string data, which are basically (pointer, length) pairs. So you may still create garbage in terms of these small slice objects, but that should cause less of a problem than creating tons of temporary immutable strings.

brazzy 14y ago
Actually, Java strings work exactly like that. And if you use them correctly, string processing is no problem at all on a modern VM using a generational GC.
- chubot 14y ago
  Ah OK I didn't know that (not a Java programmer). What happens if you create a huge 10MB string, then a 10 byte substring of it, and then remove all references to the 10MB string. Does that memory ever get reclaimed? I guess the difference in Go is that the string abstraction and the slice abstraction is exposed to the programmer, so it will be apparent if that happens. There is some info here but I was unable to find a good explanation: http://stackoverflow.com/questions/2909848/how-does-java-implement-flyweight-pattern-for-string-under-the-hood http://stackoverflow.com/questions/2909848/how-does-java-imp...
  - brazzy 14y ago
    Nope, that's one way to have a memory leak. But you can avoid the problem by using the new String(oldString) constructor - it copies only the part of the underlying char array that the String uses. You're absolutely right that it would be much better if all that were part of the API rather than an implementation detail. The SO question and answers are confusing because they talk about two competely separate issues: the above and the interning mechanism.
skybrian 14y ago
I believe in Go, slices get passed by value so you shouldn't get garbage from passing them around. Also, it's a triple rather than a pair due to the capacity.

malkia 14y ago

That's a topic that comes once in a while in our studio. Artists, generally non-technical people like to look what's inside the file. This way they can spot an extra bone in an animation, or material that should've not been used in an model. Sometimes changing the model allows for merging, or doing diff in the source control system. A more elaborate 3D diffing is what's needed, like the one shown here: http://3drepo.org/ http://3drepo.org/ - but it's not an easy thing, and not very easy to integrate either. Sometimes binary files might be bigger than text, not often but possible - for example if double is used for number it always takes 8 bytes, while as text it might be just two (0 and space). Also certain source control systems (like perforce I think) can store differences only if the files are text, otherwise they have to store full copies (that's really not a big game changer, but one more little detail to be aware). Artists, coders often would look for a material/bone by just using Find in Files/grep/etc, and although this might work for binary files, it's harder to show the data.

ww520 14y ago

That is probably similar to source code vs compiled object code. The textual graphic code is the human readable source code for describing the graphic. Reading and parsing them are similar to interpret them. To make things faster, you can read them (compile) ahead of time and write the binary objects to binary files. In runtime loading them will be fast.

aras_p 14y ago

In this case, he's working with external code (Intel's example code) that happened to use data in that way. Why - good question. I guess someone at Intel thought "hey this is just example code how to do software rasterizer for occlusion, who cares what is the data format" and went with that. Badly ;)

yock 14y ago

I would think that string immutability has much to do with the object creation. That might be why regular expressions was such a focus of the Java 1.4 release.