4 ms·
Clickbait title ;) The problem wasn't strlen(), that just showed up at the bottom of the call stack when profiling. IIRC the actual problem was an exceptional
by flohofwoe 4y ago
Clickbait title ;)
The problem wasn't strlen(), that just showed up at the bottom of the call stack when profiling.
IIRC the actual problem was an exceptionally dumb JSON parser combined with parsing a pretty big JSON file (which probably also grew much bigger than the original developer working on that feature anticipated).
Edit: the title has been edited in the meantime, originally it was something about strlen() being responsible for the slow loading time.
- ConceptJunkie 4y agoIt's funny though. Several years ago a coworker did some profiling on one of our server software components and found that it was spending up to 10% of the CPU time in strlen()... for very similar reasons.
- sfteus 4y agoHad something similar happen with low-level components in a PHP application I inherited repeatedly calling trim() and/or str_pad() on values that had already been processed. Refactoring those out led to a 10-15% performance boost in our test suites.
- Sesse__ 4y agoI've seen PHP applications use 30% of their CPU in the “realpath cache”, which at the time was a super-slow (O(n²) in this case) home-grown hash table for caching the result of realpath(), which was called all the time because someone decided include_once() needed to run its paths through realpath() in case someone included a file multiple time through symlinks. And I guess they couldn't trust inode numbers or Windows didn't have them or something… I hope it's gone. I really do hope it's gone :-)
- ddtaylor 4y agostrlen() was a major component of the slowdown. Yes, the JSON file was large and strlen() was indirectly caused by sscanf() being used. But it's not accurate to say it's a clickbait title when a bulk of the slowdown was caused by constantly recalculating the length of the C string. The article shows in detail how he created a monkey-patched strlen() that cached the length to avoid recalculating it over and over which was attributed to the speedup. The speedup gains were not caused by reducing the size of the JSON file or doing any exceptionally better JSON parsing. The speedups were a direct result of simply not calling strlen() over and over in an O(N^2) fashion.
- flohofwoe 4y agoAll true, but replacing the strlen() with a caching version is just fixing the symptoms (of calling strlen too often... but the only realistic solution without the ability to recompile the game - so kudos to the author to come up with this hack). I bet the "official" patch by Rockstar simply dropped in a less embarassing JSON parser library :)
- ddtaylor 4y agoThis article was written before Rockstar released a patch and the 70% speedup is from his strlen() caching.
- flohofwoe 4y agoYes I'm aware, that article basically triggered the official patch. Would be interesting to know if the Rockstar patch provided a further speedup though.
- whizzter 4y agoYou don't think that it's a bigger WTF that sscanf implementation calls strlen when parsing a number (when it iirc even ignores any non-digit characters for the actual number parsing so no reason it couldn't terminate that part w/o the extra call)?
- gbraad 4y agoThe title is still not right as the poster is not the 'I' as in the title.
- Joker_vD 4y agoThe JSON parser was not "exceptionally dumb": it just used sscanf(s, "%d", ...) for parsing numbers. But most implementations of C's standard library (yes, including BSD and Linux's) implement sscanf() by converting the source string into a dummy FILE* object (which involves calling strlen() to set the stream's size) and calling fscanf() on it. Apparently most people who use sscanf() are unaware of this "lovely" implementation quirk and I can't really blame them.
- AlexanderTheGr8 4y agoIs C's std impl slow because of calling strlen() or due to converting into a dummy FILE* object or due to calling fscanf?
- unwind 4y agoAll of the above, I guess. It wants to reuse code in fscanf(), but for that it needs a FILE * interface. But to create that, it needs to implement EOF-handling, and to do that it needs to provide a size, so it has to call strlen(). I had no idea. :|