3 ms·

I work in mobile performance and have built out an automated testing workflow for our CI system to try and prevent big breaking changes to these types of phones

by bm1362 8y ago

I work in mobile performance and have built out an automated testing workflow for our CI system to try and prevent big breaking changes to these types of phones. The issue with testing these phones is the device introduces lots of variance due to the low hardware standards. We typically test for relative changes in key metrics and want to move to testing different types of phones for usability. This is generally hard because the noise makes the test inconclusive.

toredash 8y ago
How do you this testing? Any public service?
- bm1362 8y ago
  We have a lab environment with multiple phones plugged into a single controller. The controller runs a set of Jenkins agents which advertises the different devices as targets for Jenkins jobs. Plus, lots of tweaking and perf related fixes to the phone. If you're looking for an off-the-shelf solution, check out Headspin or the Google Cloud offerings.
  - toredash 8y ago
    Thanks for the feedback. I'll look into headspin. I'm not into mobile performance or testing so not sure how I would automate and log the performance on different devices. Honestly I'm not sure what tools I would use on the device to run the tests.
    - bm1362 8y ago
      We are trying to build something similar to CT-scan; except it’s a 1-man team so it’s been a longer journey: https://code.fb.com/developer-tools/mobile-performance-tooling-infrastructure-at-facebook/ https://code.fb.com/developer-tools/mobile-performance-tooli... We run functional UI tests with a performance tracer wired up- we already had them setup by feature teams so it was easy to get started. Instead of a Zookeeper scheduler, it’s just Jenkins. Right now we’ve been able to catch 2-3% relative regressions in things like startup, opening a Google Maps view etc which is pretty good. It was easy to setup and then hard to optimize the variance down so we could get a strong signal. We’re hiring if any of this is interesting, it’s Golang/San Francisco/Full-Time.

by bm1362 8y ago

toredash 8y ago

How do you this testing? Any public service?

bm1362 8y ago
We have a lab environment with multiple phones plugged into a single controller. The controller runs a set of Jenkins agents which advertises the different devices as targets for Jenkins jobs. Plus, lots of tweaking and perf related fixes to the phone. If you're looking for an off-the-shelf solution, check out Headspin or the Google Cloud offerings.
- toredash 8y ago
  Thanks for the feedback. I'll look into headspin. I'm not into mobile performance or testing so not sure how I would automate and log the performance on different devices. Honestly I'm not sure what tools I would use on the device to run the tests.
  - bm1362 8y ago
    We are trying to build something similar to CT-scan; except it’s a 1-man team so it’s been a longer journey: https://code.fb.com/developer-tools/mobile-performance-tooling-infrastructure-at-facebook/ https://code.fb.com/developer-tools/mobile-performance-tooli... We run functional UI tests with a performance tracer wired up- we already had them setup by feature teams so it was easy to get started. Instead of a Zookeeper scheduler, it’s just Jenkins. Right now we’ve been able to catch 2-3% relative regressions in things like startup, opening a Google Maps view etc which is pretty good. It was easy to setup and then hard to optimize the variance down so we could get a strong signal. We’re hiring if any of this is interesting, it’s Golang/San Francisco/Full-Time.