4 ms·
Ask HN: how would you A/B test iPhone apps?
patio11's A/Bingo post got me thinking. How would you go about doing A/B testing for iPhone apps?
The issue, obviously, is the app store submission/approval cycle which makes changes slow to reach users. (even without that, you probably wouldn't want to make users update their apps every day).
If the app is mainly web-based, it's pretty obvious, but what about native apps?
- bdmac97 17y agoI know very little about iPhone programming so forgive me if this is ludicrous but just thinking out loud... We had an app launch on launchly awhile back that was basically trying to be the iPhone version of GetSatisfaction or UserVoice. You could plug in their code into your app and it would handle feedback and bug reports for you. Maybe there's a market for something similar for A/B testing on iPhones. If it's a tricky problem to solve, even better cause then people will pay you for it. Initial thought w/out knowing anything about the security model is can you set up config files or even remote code on a server somewhere and then access it from your code on the iPhone using an API key? If you can then you could maybe set up multiple config files on the server and have your code load different versions randomly and report back statistics... Of course that brings up an interesting point. What would you be A/B testing in an iPhone app? The user has to have already downloaded/installed your app. At that point there's really not any landing page or funnel to test like there is on a website. Obviously you can still A/B test internals just to improve the app which will affect ratings which can affect downloads (conversions). Anyways, just thinking out loud here.
- DougBTX 17y agoGuess: - Do fake A/B testing for your AppStore product page, toggle once a week and compare sales. - For the product itself, try usability testing face-to-face with potential users, using ad-hoc distribution.
- wmeredith 17y agoDesign an iPhone app that's a CMS pulling in data streams (XML) and displaying them to users. Set the first queried data stream to be the A/B switch and all other data is pulled into the app dependent on the state of that switch. This is all assuming you're talking about testing within the app itself after it has been downloaded. (I'm an SEM/SEO UX guy, so this is out of my league, but I think that makes sense.)
- frosty 17y agoIf the screen which you want to A/B test is not the first screen, you can make background calls to your server and based on result decide which screen to show. Write to user settings when you want to permanently disable it. Note: i) Reasoning behind not having it as first screen as it really slows the application launch. ii) You should make sure that if webservice call takes too long, just pick one default which is shown to user instead of waiting for result iii) if you really want to do it for first screen, load and save the result in user settings for next launch while the application is running
- ryanpetrich 17y agoIt's actually pretty simple to A/B test on App Store apps: 1. Prepare two versions of each feature you want to test 2. Generate random integer on first launch 3. Save to NSUserDefaults 4. Present A/B based on the value of a particular bit in the random integer 5. Collect statistics and send back anonymously via NSURLConnection Question is probably better suited to StackOverflow