5 ms·

Piggybacking on this, can anyone recommend an approach for finding the cause of memory issues in v8? In my case I'm tuning a 3D game, and the heap usage is a s

by fenomas 10y ago

Piggybacking on this, can anyone recommend an approach for finding the cause of memory issues in v8? In my case I'm tuning a 3D game, and the heap usage is a sawtooth pattern with GCs about once per second. Not awful, but it would be nice to smooth it out. But playing with Chrome's various memory profiling tools I've never been able to discover where the allocations are coming from - the results always seem to be dominated by apparently internal stuff (like entries called "(code deopt data") or such). Does anyone know any good techniques for such things?

tehlike 10y ago
Is the game open to public? I would like to give it a look, i have done some amount of optimizations as a hobby.
- alooPotato 10y ago
  hey tehlike - I work on streak.com and we have some pretty interesting memory challenges being a full blown web app running inside of another very complex web app (gmail). Any interest in poking around and helping us with memory issues?
- fenomas 10y ago
  It's not yet, but I'm pretty sure the allocations are happening in the 3D engine I'm using. (I mean the chrome tools aren't telling me that, but when I let the game run without rendering the sawtooth goes away.) Actually now that I take a second look, some relatively simple demos of the same engine (Babylon.js) show the same sort of behavior. Some rather trivial three.js demos do as well. I might be dealing with something that's just a fact of life for webGL rendering. Random example (not mine) showing a vaguely similar sawtooth of heap memory usage: http://gleborgne.github.io/molvwr/#1GCN http://gleborgne.github.io/molvwr/#1GCN
RKoutnik 10y ago
Usually that's because you're creating and disposing of a lot of objects in your game loop. Look into initializing a lot of objects at the start and reusing 'em (usually called object pooling). I'd love to take a look deeper - contact info is in my profile if you're interested.
- fenomas 10y ago
  Sorry if I wasn't clear, but the problem is finding the allocations, not fixing them. (They are probably happening in the 3D engine, i.e. code I didn't write.) One suspects that DevTools' memory profiling should be the place to start, but I haven't found any way to get it to shed any light on where allocations are occurring. So that's what I'm asking about here.
  - duncanawoods 10y ago
    You are going to hate this one but its effective - divide and conquer. Just strip out features in a binary-chop fashion and rerun the memory profile. Depending on the code, it doesn't take as long as you think it will to get to small units of functionality e.g. sounds like just commenting out the 3d engine render step would be informative to you.
    - fenomas 10y ago
      As I commented to a different reply, skipping the render step does indeed remove the sawtooth, which is why I think the problem is in the 3D engine. Past that, it's not so easy to jump into a three.js-sized chunk of 3rd party code and find atomic things that can be turned off in such a way that everything else still functions. (I mean, not to suggest that what you describe isn't useful - I'm just hoping to use profiling tools to attack from a different direction.)
      - duncanawoods 10y ago
        Understood. >> not so easy to jump into a three.js-sized chunk of 3rd party code and find atomic things that can be turned off Incidentally, doing a dissection like this is actually an engaging way to learn a big chunk of 3rd party code compared to just staring at it. By stubbing out various pieces you learn where the joints are where things are entangled nests of sorrow.
  - ofrobots 10y ago
    Recent versions of Chrome DevTools have a new profiling feature called 'Record allocation profile' that may help. Enable this around a few of the sawtooth and it will give you a profile based on a sampling of allocations that happen during that period. The profile will include the stack-trace at the time of the allocation which should help you figure out where the allocations are coming from.
natorion 10y ago
There are tools with more depths: You can try about:tracing or the heap visualizer mentioned in the article.
iamstef 10y ago
You may want to give the follow script a try: * https://gist.github.com/krisselden/d3ce3cbb37cc6035b0927fdbf4117b3c https://gist.github.com/krisselden/d3ce3cbb37cc6035b0927fdbf... It flips some handy flags providing useful output, this output can quickly illuminate issues the regular tools do not (yet). Running this on the example you linked to bellow, shows that a series of functions are deopting and optimizing repeatedly. most likely causing at-least some of the sawtooth pattern you see. example output (likely related to the problem): ``` removing optimized code for: r.getViewMatrix] [removing optimized code for: r._isSynchronizedViewMatrix] [removing optimized code for: r._computeViewMatrix] [removing optimized code for: r._isSynchronized] [removing optimized code for: r._computeViewMatrix] [removing optimized code for: r._isSynchronizedViewMatrix] [removing optimized code for: r._isSynchronized] [removing optimized code for: r._computeViewMatrix] [removing optimized code for: r._isSynchronizedViewMatrix] [removing optimized code for: r._isSynchronized] [removing optimized code for: r.getViewMatrix] [removing optimized code for: r._isSynchronizedViewMatrix] [removing optimized code for: r._computeViewMatrix] [removing optimized code for: r._isSynchronized] [removing optimized code for: r._computeViewMatrix] [removing optimized code for: r._isSynchronizedViewMatrix] [removing optimized code for: r._isSynchronized] [removing optimized code for: r._computeViewMatrix] [removing optimized code for: r._isSynchronizedViewMatrix] [removing optimized code for: r._isSynchronized] [removing optimized code for: r.getViewMatrix] ``` --- note: Credit for this should go to @krisselden the author of the above gist not me.
- fenomas 10y ago
  Hrm. I've looked at deopts before, and I may be wrong but I think they aren't the issue. As I understand v8, it's normal at startup for hot functions to go through the opt/deopt cycle several times as the engine learns about them - and once a function does it too many times, it gets deopted permanently. For this reason I always let the game run for 10-15 seconds before I profile, figuring that by that time most of the opt/deopt churn will be finished. And this (very useful) script seems to back this up - I get output like yours at startup, but if I wait a while and then rm the output files, further output is relatively minimal. So I'm inclined to think that opt/deopt stuff isn't the issue, and there really are lots of JS heap objects getting allocated somewhere. At the same time though, when I used Chrome's built-in memory profiling I see a bunch of deopt-related strings, so maybe I'm way off base. If anyone sees what I'm missing please do clue me in. (Also: great tip on the script!)

by fenomas 10y ago

tehlike 10y ago

Is the game open to public? I would like to give it a look, i have done some amount of optimizations as a hobby.

alooPotato 10y ago
hey tehlike - I work on streak.com and we have some pretty interesting memory challenges being a full blown web app running inside of another very complex web app (gmail). Any interest in poking around and helping us with memory issues?
fenomas 10y ago
It's not yet, but I'm pretty sure the allocations are happening in the 3D engine I'm using. (I mean the chrome tools aren't telling me that, but when I let the game run without rendering the sawtooth goes away.) Actually now that I take a second look, some relatively simple demos of the same engine (Babylon.js) show the same sort of behavior. Some rather trivial three.js demos do as well. I might be dealing with something that's just a fact of life for webGL rendering. Random example (not mine) showing a vaguely similar sawtooth of heap memory usage: http://gleborgne.github.io/molvwr/#1GCN http://gleborgne.github.io/molvwr/#1GCN

RKoutnik 10y ago

Usually that's because you're creating and disposing of a lot of objects in your game loop. Look into initializing a lot of objects at the start and reusing 'em (usually called object pooling). I'd love to take a look deeper - contact info is in my profile if you're interested.

fenomas 10y ago
Sorry if I wasn't clear, but the problem is finding the allocations, not fixing them. (They are probably happening in the 3D engine, i.e. code I didn't write.) One suspects that DevTools' memory profiling should be the place to start, but I haven't found any way to get it to shed any light on where allocations are occurring. So that's what I'm asking about here.
- duncanawoods 10y ago
  You are going to hate this one but its effective - divide and conquer. Just strip out features in a binary-chop fashion and rerun the memory profile. Depending on the code, it doesn't take as long as you think it will to get to small units of functionality e.g. sounds like just commenting out the 3d engine render step would be informative to you.
  - fenomas 10y ago
    As I commented to a different reply, skipping the render step does indeed remove the sawtooth, which is why I think the problem is in the 3D engine. Past that, it's not so easy to jump into a three.js-sized chunk of 3rd party code and find atomic things that can be turned off in such a way that everything else still functions. (I mean, not to suggest that what you describe isn't useful - I'm just hoping to use profiling tools to attack from a different direction.)
    - duncanawoods 10y ago
      Understood. >> not so easy to jump into a three.js-sized chunk of 3rd party code and find atomic things that can be turned off Incidentally, doing a dissection like this is actually an engaging way to learn a big chunk of 3rd party code compared to just staring at it. By stubbing out various pieces you learn where the joints are where things are entangled nests of sorrow.
- ofrobots 10y ago
  Recent versions of Chrome DevTools have a new profiling feature called 'Record allocation profile' that may help. Enable this around a few of the sawtooth and it will give you a profile based on a sampling of allocations that happen during that period. The profile will include the stack-trace at the time of the allocation which should help you figure out where the allocations are coming from.

natorion 10y ago

There are tools with more depths: You can try about:tracing or the heap visualizer mentioned in the article.

iamstef 10y ago

You may want to give the follow script a try: * https://gist.github.com/krisselden/d3ce3cbb37cc6035b0927fdbf4117b3c https://gist.github.com/krisselden/d3ce3cbb37cc6035b0927fdbf... It flips some handy flags providing useful output, this output can quickly illuminate issues the regular tools do not (yet). Running this on the example you linked to bellow, shows that a series of functions are deopting and optimizing repeatedly. most likely causing at-least some of the sawtooth pattern you see. example output (likely related to the problem): ``` removing optimized code for: r.getViewMatrix] [removing optimized code for: r._isSynchronizedViewMatrix] [removing optimized code for: r._computeViewMatrix] [removing optimized code for: r._isSynchronized] [removing optimized code for: r._computeViewMatrix] [removing optimized code for: r._isSynchronizedViewMatrix] [removing optimized code for: r._isSynchronized] [removing optimized code for: r._computeViewMatrix] [removing optimized code for: r._isSynchronizedViewMatrix] [removing optimized code for: r._isSynchronized] [removing optimized code for: r.getViewMatrix] [removing optimized code for: r._isSynchronizedViewMatrix] [removing optimized code for: r._computeViewMatrix] [removing optimized code for: r._isSynchronized] [removing optimized code for: r._computeViewMatrix] [removing optimized code for: r._isSynchronizedViewMatrix] [removing optimized code for: r._isSynchronized] [removing optimized code for: r._computeViewMatrix] [removing optimized code for: r._isSynchronizedViewMatrix] [removing optimized code for: r._isSynchronized] [removing optimized code for: r.getViewMatrix] ``` --- note: Credit for this should go to @krisselden the author of the above gist not me.

fenomas 10y ago
Hrm. I've looked at deopts before, and I may be wrong but I think they aren't the issue. As I understand v8, it's normal at startup for hot functions to go through the opt/deopt cycle several times as the engine learns about them - and once a function does it too many times, it gets deopted permanently. For this reason I always let the game run for 10-15 seconds before I profile, figuring that by that time most of the opt/deopt churn will be finished. And this (very useful) script seems to back this up - I get output like yours at startup, but if I wait a while and then rm the output files, further output is relatively minimal. So I'm inclined to think that opt/deopt stuff isn't the issue, and there really are lots of JS heap objects getting allocated somewhere. At the same time though, when I used Chrome's built-in memory profiling I see a bunch of deopt-related strings, so maybe I'm way off base. If anyone sees what I'm missing please do clue me in. (Also: great tip on the script!)