3 ms·

Point of order on the methodology here. This benchmark gets harder the longer it runs. Which is not in itself terribly unusual. It is pretty smooth for the f

Point of order on the methodology here. This benchmark gets harder the longer it runs. Which is not in itself terribly unusual. It is pretty smooth for the first bit, where the benchmark runs, but right about where things get hard on my iPad Air is when the test stops. So this raises a question about how the test endpoint was decided. Because of course if you run longer, the performance would be worse, if you run shorter, the performance will be better, etc. Now according to the previous article [1], this endpoint is exactly 60 seconds, and an average is taken over that entire period except some time at the beginning for JIT. Now 60 is a suspiciously round number to arrive at by trying to hit X many sprites on the screen or numerically deriving some benchmark objective. Probably, somebody just eyeballed what a Space Blaster game looked like and one minute of runtime was about right. That probably is a reasonable predictor for Space Blaster, but it is quite the leap to use these results to predict "most other HTML5 games" as asserted in this article. Some games will have half the sprites and others will have double, some games will be more spikey, and others will be more well-behaved, etc. If I was going to use this benchmark to predict my game's performance, I would do what the benchmark designer probably did. I would ignore the results table, run the benchmark until the sprite count/size looked about right, and take my own average of the FPS indicator around that time. [1] https://www.scirra.com/blog/107/boosting-mobile-html5-game-performance-with-webgl https://www.scirra.com/blog/107/boosting-mobile-html5-game-p...

kevingadd 12y ago
I'd also be concerned that the benchmark doesn't present a realistic performance profile, either in CPU or GPU terms. It appears to have a low entity count up until the end, and not have any considerable amount of collision detection or other cpu-intensive entity behaviors. This means it's not really stressing a JS runtime and could probably perform great under an interpreter (as demonstrated by Ejecta getting a good framerate) It also doesn't appear to produce a taxing GPU profile even on a canvas implementation. It draws very few objects, and most of them share the same texture and rendering state, ensuring that even a simple rendering implementation can batch them up into a single draw call with very few vertices. The framebuffer resolution is low and the overdraw is low as well. A real-world game, even one that isn't 'too complicated' for mobile, would have a dramatically different performance profile. I think this is a solid benchmark, but it's irresponsible to make claims like 'if sbperftest runs well, most other HTML5 games should run decently as well' and 'The next most common case is usually someone designing or porting such a complex game that even a native engine would struggle on the limited, battery-powered mobile hardware.' You simply can't make those claims without a broader benchmark suite (which I should note most of the professional mobile/desktop benchmarking suites come closer to offering, with a handful of different benchmarks based on different workloads).
- AshleysBrain 12y ago
  Author here. Perhaps the "if sbperftest runs well, most other HTML5 games should run decently as well" statement should have been inverted, so it reads "if sbperftest doesn't run well, few other HTML5 games will". We develop a major HTML5 game engine, and this statement is backed by quite a bit of testing and profiling of real-world games. The test is deliberately based on a real-world game (it was adapted from an interactive game) so it's not a synthetic benchmark. No it is not enormously onerous for the CPU or GPU, but if you look at the results many mobile devices still can't quite manage a rock-solid 60 FPS, so I think it serves pretty well to see if mobile devices are fast enough for relatively simple games. There is actually a lot of collision testing in the laser-to-enemy intersection test - all of the objects in the game have custom collision polygons, it's not a simple bounding box test. The explosions use additive blending (or "lighter" composite mode) which actually seems to stress many canvas2d implementations but generally is a lot smoother with WebGL. So it's not that simple a test. I'm not sure why the 60 second period is important; the most intensive part is the waves of green sprites which should come well within the measured period, and it gets quieter after that when the test ends, so it's not skipping the hard bit.
sp332 12y ago
There are constant-FPS tests, that create more and more sprites until your FPS falls below a certain target. Your score is the number of sprites you can handle above that target.

kevingadd 12y ago

I'd also be concerned that the benchmark doesn't present a realistic performance profile, either in CPU or GPU terms. It appears to have a low entity count up until the end, and not have any considerable amount of collision detection or other cpu-intensive entity behaviors. This means it's not really stressing a JS runtime and could probably perform great under an interpreter (as demonstrated by Ejecta getting a good framerate) It also doesn't appear to produce a taxing GPU profile even on a canvas implementation. It draws very few objects, and most of them share the same texture and rendering state, ensuring that even a simple rendering implementation can batch them up into a single draw call with very few vertices. The framebuffer resolution is low and the overdraw is low as well. A real-world game, even one that isn't 'too complicated' for mobile, would have a dramatically different performance profile. I think this is a solid benchmark, but it's irresponsible to make claims like 'if sbperftest runs well, most other HTML5 games should run decently as well' and 'The next most common case is usually someone designing or porting such a complex game that even a native engine would struggle on the limited, battery-powered mobile hardware.' You simply can't make those claims without a broader benchmark suite (which I should note most of the professional mobile/desktop benchmarking suites come closer to offering, with a handful of different benchmarks based on different workloads).

AshleysBrain 12y ago
Author here. Perhaps the "if sbperftest runs well, most other HTML5 games should run decently as well" statement should have been inverted, so it reads "if sbperftest doesn't run well, few other HTML5 games will". We develop a major HTML5 game engine, and this statement is backed by quite a bit of testing and profiling of real-world games. The test is deliberately based on a real-world game (it was adapted from an interactive game) so it's not a synthetic benchmark. No it is not enormously onerous for the CPU or GPU, but if you look at the results many mobile devices still can't quite manage a rock-solid 60 FPS, so I think it serves pretty well to see if mobile devices are fast enough for relatively simple games. There is actually a lot of collision testing in the laser-to-enemy intersection test - all of the objects in the game have custom collision polygons, it's not a simple bounding box test. The explosions use additive blending (or "lighter" composite mode) which actually seems to stress many canvas2d implementations but generally is a lot smoother with WebGL. So it's not that simple a test. I'm not sure why the 60 second period is important; the most intensive part is the waves of green sprites which should come well within the measured period, and it gets quieter after that when the test ends, so it's not skipping the hard bit.

sp332 12y ago

There are constant-FPS tests, that create more and more sprites until your FPS falls below a certain target. Your score is the number of sprites you can handle above that target.