4 ms·
That test is relevant for the end user, but it doesn't make sense as a hardware test. It's a different system, on different hardware, with different apps (that
by viraptor 10y ago
That test is relevant for the end user, but it doesn't make sense as a hardware test. It's a different system, on different hardware, with different apps (that happen to share the same name), and essentially what you're comparing is resource loading strategies. iPhone is better at that - great.
But for a hardware test, I'd expect a single running app, executing shared native codebase, in no disruptions airplane mode, measuring specific part of the hardware. And that's not even close to measuring before/after improved branch prediction, which could be useful in both phones.
- swinglock 10y agoWhy measure and therefore optimize for anything but how the product is being used?
- girvo 10y agoIn general: because how a piece of hardware (or set of hardware) is used today isn't necessarily how it's going to be used tomorrow. Faster hardware allows for use-cases that weren't possible prior.
- kingnothing 10y agoHowever, when we're talking about phones or consumer devices, how it's used today is the only thing that's relevant. It will be obsolete in just a couple of years.
- comboy 10y agoI think the parent point is still valid. Look at Pokemon Go. Average user wouldn't care about AR performance, all that high gps accuracy or gfx performance in unity. Suddenly, there are millions of players who are choosing their hardware based on that. If you are planning for today's usage you you may be lagging.
- viraptor 10y agoI didn't say it's not valid, or that it shouldn't have been done. I'm just saying "NN still doesn't help them beat iPhone" is not a relevant comment regarding this test. To go with car analogy it's close to an article about an slightly improved design of one part of fuel injection in new ABC and someone commenting "look, an XYZ from last year still can go faster without it". The test is a valid observation, but is irrelevant to this article, or to the technology. Why measure something that doesn't show up in a standard user's test? Here are some examples: lower power usage, lower latency of small operations (rather than throughput of large actions), smaller design (saving chip space), etc. And finally - if this is a good technology, Apple can use it too and get even faster.
- aokyler 10y agoThat is a great analogy. That test in the video didn't really mean anything towards the overall speed of the device. It's interesting to see, but I how this test would change over n trials.
- adrianN 10y agoYou should do that, but not only that. If you just test the way the product is being used, you have a million possible reasons why it could be slow. If you just want to know whether slapping a neural network on the chip is a good idea or not, it's best to test that in such a way as to exclude as many confounding factors as possible.
- BinaryIdiot 10y ago> But for a hardware test, I'd expect a single running app, executing shared native codebase, in no disruptions airplane mode, measuring specific part of the hardware. And that's not even close to measuring before/after improved branch prediction, which could be useful in both phones. I would argue that would be a completely useless test unless you're explicitly testing something for development purposes only. Users use phones. Nothing else matters but the user experience. If you can make your phone beat another phone in your limited, no disruptions, network traffic off example that doesn't mean anything. I feel this is a fair test. You test the phones at what they're supposed to be able to do. If one does it better it's legit to point that out regardless of what the underlying structure looks like.
- viraptor 10y agoI think you either misread my comment or missed the point of it. Overall tests matter, specific tests matter, unittests matter, microbenchmarks matter, user experience matters, perceived performance matters. But they don't always intersect. Bringing up user experience when talking about branch predictors is meaningless. Just as bringing up branch predictors when talking about perceived performance. (unless you're proving that this exact feature being available/not available, with other things being controlled for, makes a major difference in the test)
- zmb_ 10y ago> Bringing up user experience when talking about branch predictors is meaningless. If working on branch predictors does not impact the user experience, then arguably that work is meaningless in the context of building real products -- it may still be interesting for academic purposes.
- fasouto 10y agoBranch predictors made the phones faster and this affects the user experience. But if you want to test the performance of branch predictors you have to compare similar things, you cannot benchmark two different OS and apps because you get and OS and app test not an processor test.
- ddebernardy 10y ago> That test is relevant for the end user, but it doesn't make sense as a hardware test. Err? It seems to me that the only tests that make any sense are those that count for end users. Everything else is pointless. It doesn't matter if it's the same system or code base or whatever other engineering fetish. Users who compare their phones to see which goes faster look at how the same app opens and runs on two separate phones. Which incidentally is what this test is about. It's the only sensible test you can do.
- ajdlinux 10y agoIf you're an engineer who's working on branch predictors, or a developer working on compiler optimisations, then such a test isn't exactly pointless...
- golergka 10y agoThis os a test for an entire phone, including the OS. The article focuses on the chip alone.
- dspillett 10y ago> Err? It seems to me that the only tests that make any sense are those that count for end users. Everything else is pointless. If you are specifically comparing two specific phones against each other then yes this is the only sort of test that matters, and if you are comparing manufacture's flagship models then that is what you are doing. From every other point of view, either comparing performance of individual components or comparing like-for-like performance of devices with very similar spec it is less valid though: the screen resolution is a huge variable for a game so it isn't a "fair" test to compare performance of two devices like that. You could equally state that the higher resolution produces a better result because of the resolution difference and you would similarly be called to task by for not comparing like for like. Maybe the a device with the same hardware other than the better screen would outperform the iDevice on the test, or at least not appear to underperform by as much, this test can't tell us that. This is difficult of course because few phone models are practically identical and few vary by just one factor, and can make a developers life more difficult when trying to support a varied market like Android based devices.
- 10y ago
- coldtea 10y ago>That test is relevant for the end user, but it doesn't make sense as a hardware test. It's a different system, on different hardware, with different apps (that happen to share the same name), and essentially what you're comparing is resource loading strategies. iPhone is better at that - great. Anything besides the concrete end-user experience of a system doesn't matter at all. (In theory it could matter for cpu/compiler designers etc -- but even what matters to them is inconsequential if in the end of the line it doesn't matter to the end-user).