6 ms·

AWS Device Farm adds support for iOS

66 points by impostervt 11y ago

gruez 11y ago
Did they work with apple to create a custom version of iOS to integrate with amazon's service, or did amazon jailbreak the devices and installed their own hooks?
- quisquous 11y ago
  From their FAQ: "AWS Device Farm tests are run on real, non-rooted devices. The devices are a mixture of OEM and carrier-branded devices."
- babababa 11y ago
  It's probably not rooted, but it probably also violates Apple's developer license agreement (which are needlessly restrictive) - sections 2 and 3 in this doc: https://developer.apple.com/programs/terms/ios/standard/ios_program_standard_agreement_20140909.pdf https://developer.apple.com/programs/terms/ios/standard/ios_...
- NateLawson 11y ago
  More importantly, what do they do to keep users from jailbreaking the device via software? Especially with Android, there are a lot of software root flaws and the privileged surface area is a lot bigger than the hypervisor AWS uses for EC2. The summary makes it sound like they only wipe your app from the device, not reinstall the OS between users.
  - jakozaur 11y ago
    I believe it doesn't matter for 95%+ of users. EC2 is used to run production services with sensitive data. Most Android/iOS testing is done with made up data, without privileged access to production. So even if it get compromised, it probably not much different from installing an app on jailbreaked device be end user.
AndrewKemendo 11y ago
We found it hard to find reliable testers for iOS without building your own team or doing it all on your own devices. Maybe Amazon will come through, but we'll see. I'd love to see a service where one super tech just tries to break your application on every supported device for a couple hours. A room full of toddlers might also work.
- i336_ 11y ago
  I think it would be a truly viable startup idea to connect app developers with "naive" people - soccer moms who simply haven't got the time to learn tech properly, 70-year-olds who've never used a smartphone, 3 year olds, etc. Being able to get your website or app (or even good ol' desktop program!) tested by people who have <specific metric> exposure to <granular list of options> wrt. technology would be invaluable, I think. I consider it sad that this either doesn't exist, or isn't yet a standard part of the common consciousness/culture.
  - didgeoridoo 11y ago
    Applause.com does this kind of "in the wild testing" with testers of all levels of expertise. Definitely worth checking out.
bitmapbrother 11y ago
I believe Google already provides this service to its developers for free.
- sinatra 11y ago
  Wait, Google doesn't support iOS devices (whereas, this announcement is specifically about iOS), right?
  - bitmapbrother 11y ago
    For Android devices only.
- i336_ 11y ago
  Where can I find more info about this?
  - dharma1 11y ago
    http://techcrunch.com/2015/05/28/googles-new-cloud-based-test-lab-helps-android-developers-test-their-apps-on-multiple-devices/ http://techcrunch.com/2015/05/28/googles-new-cloud-based-tes... It's in your Google Play developer console
JDeArte 11y ago
We created an Android AND iOS farm for automated game testing. You don't need to root iOS to get it done - it's just a PITA to do it right. You end up needing an attached Mac running XCode to send commands & you can't use the internet connection through the wire so we set up a local secure wifi network. The only devices we had to "root" (aka put in developer mode) were some android devices that wouldn't allow us to change the HOSTS file otherwise ;-) I'm glad that AWS is now offering this type of service, I can't wait to try it out.
i336_ 11y ago
This is admittedly definitely the kind of thing I'd love to see a "making of" about. Actual real devices?! I would really like to get a bit of a behind-the-scenes on how they managed that. :P
srikz 11y ago
It says first 250 device minutes free. How much does it cost after that? There doesn't seem to be a direct link to the pricing page.
- itsmeduncan 11y ago
  "Pricing is based on device minutes, which are determined by the duration of tests on each selected device. AWS Device Farm comes with a free trial of 250 device minutes. After that, customers are charged $0.17 per device minute. As your testing needs grow, you can opt for an unmetered testing plan, which allows unlimited testing for a flat monthly fee of $250 per device." From the FAQ[1] which albeit was a bit buried. edit: It is also at the bottom of the landing page now. 1: https://aws.amazon.com/device-farm/faq/ https://aws.amazon.com/device-farm/faq/
  - srikz 11y ago
    thanks!
  - sandis 11y ago
    > a flat monthly fee of $250 per device. That's a bit steep? You could recoup any iOS device in <= 3 months.

66 points by impostervt 11y ago

gruez 11y ago

Did they work with apple to create a custom version of iOS to integrate with amazon's service, or did amazon jailbreak the devices and installed their own hooks?

quisquous 11y ago
From their FAQ: "AWS Device Farm tests are run on real, non-rooted devices. The devices are a mixture of OEM and carrier-branded devices."
babababa 11y ago
It's probably not rooted, but it probably also violates Apple's developer license agreement (which are needlessly restrictive) - sections 2 and 3 in this doc: https://developer.apple.com/programs/terms/ios/standard/ios_program_standard_agreement_20140909.pdf https://developer.apple.com/programs/terms/ios/standard/ios_...
NateLawson 11y ago
More importantly, what do they do to keep users from jailbreaking the device via software? Especially with Android, there are a lot of software root flaws and the privileged surface area is a lot bigger than the hypervisor AWS uses for EC2. The summary makes it sound like they only wipe your app from the device, not reinstall the OS between users.
- jakozaur 11y ago
  I believe it doesn't matter for 95%+ of users. EC2 is used to run production services with sensitive data. Most Android/iOS testing is done with made up data, without privileged access to production. So even if it get compromised, it probably not much different from installing an app on jailbreaked device be end user.

AndrewKemendo 11y ago

We found it hard to find reliable testers for iOS without building your own team or doing it all on your own devices. Maybe Amazon will come through, but we'll see. I'd love to see a service where one super tech just tries to break your application on every supported device for a couple hours. A room full of toddlers might also work.

i336_ 11y ago
I think it would be a truly viable startup idea to connect app developers with "naive" people - soccer moms who simply haven't got the time to learn tech properly, 70-year-olds who've never used a smartphone, 3 year olds, etc. Being able to get your website or app (or even good ol' desktop program!) tested by people who have <specific metric> exposure to <granular list of options> wrt. technology would be invaluable, I think. I consider it sad that this either doesn't exist, or isn't yet a standard part of the common consciousness/culture.
- didgeoridoo 11y ago
  Applause.com does this kind of "in the wild testing" with testers of all levels of expertise. Definitely worth checking out.

bitmapbrother 11y ago

I believe Google already provides this service to its developers for free.

sinatra 11y ago
Wait, Google doesn't support iOS devices (whereas, this announcement is specifically about iOS), right?
- bitmapbrother 11y ago
  For Android devices only.
i336_ 11y ago
Where can I find more info about this?
- dharma1 11y ago
  http://techcrunch.com/2015/05/28/googles-new-cloud-based-test-lab-helps-android-developers-test-their-apps-on-multiple-devices/ http://techcrunch.com/2015/05/28/googles-new-cloud-based-tes... It's in your Google Play developer console

JDeArte 11y ago

We created an Android AND iOS farm for automated game testing. You don't need to root iOS to get it done - it's just a PITA to do it right. You end up needing an attached Mac running XCode to send commands & you can't use the internet connection through the wire so we set up a local secure wifi network. The only devices we had to "root" (aka put in developer mode) were some android devices that wouldn't allow us to change the HOSTS file otherwise ;-) I'm glad that AWS is now offering this type of service, I can't wait to try it out.

i336_ 11y ago

This is admittedly definitely the kind of thing I'd love to see a "making of" about. Actual real devices?! I would really like to get a bit of a behind-the-scenes on how they managed that. :P

srikz 11y ago

It says first 250 device minutes free. How much does it cost after that? There doesn't seem to be a direct link to the pricing page.

itsmeduncan 11y ago
"Pricing is based on device minutes, which are determined by the duration of tests on each selected device. AWS Device Farm comes with a free trial of 250 device minutes. After that, customers are charged $0.17 per device minute. As your testing needs grow, you can opt for an unmetered testing plan, which allows unlimited testing for a flat monthly fee of $250 per device." From the FAQ[1] which albeit was a bit buried. edit: It is also at the bottom of the landing page now. 1: https://aws.amazon.com/device-farm/faq/ https://aws.amazon.com/device-farm/faq/
- srikz 11y ago
  thanks!
- sandis 11y ago
  > a flat monthly fee of $250 per device. That's a bit steep? You could recoup any iOS device in <= 3 months.