6 ms·

Show HN – Apptimize: A/B testing for native Android, iOS apps

126 points by jorlow 13y ago

sync 13y ago
For folks interested in mobile A/B testing, also see an open-source alternative by Clutch: https://github.com/clutchio https://github.com/clutchio
- jorlow 13y ago
  For what it's worth, using Clutch was one of the reasons we ended up making Apptimize. We ran into several bugs including one that was crashing the app we were working on and issues with its probability distribution. And the reporting was very simplistic. Are you using Clutch now? If so, it'd be awesome if you could give us a try.
- siglesias 13y ago
  Additionally, incumbent mobile A/B testing service http://www.leanplum.com http://www.leanplum.com -- I've used them for a while now and it's a pretty robust system with good support. Worth checking out IMO.
- pbharrin 13y ago
  ClutchIO is not native, which is why something like this is needed.
jakestein 13y ago
Interesting, Artisan mobile is also in this space http://useartisan.com/ http://useartisan.com/
igorsyl 13y ago
Why should anyone choose to use Apptimize over a free open-source alternative?
- jorlow 13y ago
  One of the biggest reasons is probably the reason behind using any software as a service rather than doing it yourself: you can concentrate on what you do best and not spend time managing things yourself. A couple of our users switched from using open source alternatives for that reason. We've put a lot of time into our infrastructure and decoupling things so we're very resilient against outages so you don't have to. We've also put a lot of time into making our library more robust and well tested than any alternatives we've seen to date.
- chrisalvino 13y ago
  I'm guessing because it's better than what they could integrate and build up with an open-source alternative in a short amount of time and thus more cost effective.
philbarr 13y ago
This looks interesting - although having never really considered using A/B testing on native apps before I did spend a few minutes looking around the site for examples of things to track. Maybe you could show an example project on the site showing what was changed and how that affected sales, etc.?
- nancyhua 13y ago
  Part of apptimize team here. We want to do that! We will add more to our site; currently it looks deceptively slick bc our designer is awesome but it's really a first iteration. Right now if you sign in, you can see an example experiment. How does it look to you?
  - philbarr 13y ago
    Yes the example gives a much better idea of what's going on. Although I did have to create an account to look at it... Would be nice to have that accessible from the main page.
sirpillage 13y ago
Apptimize is far better than anything currently available!
chrisalvino 13y ago
Our organization often comes up with its own secondary metrics based on user data (not just CTR or usage). Is this possible when using Apptimize?
- nancyhua 13y ago
  Part of apptimize team here! We let you track whatever goals you want, so people measure stuff like "user returned within a week" and "user used app." What goals do you want to track?
  - chrisalvino 13y ago
    That really depends on what we find to be important metrics. Flexibility is important since we need to be flexible too.
xoail 13y ago
I would like to see some code examples.
- jorlow 13y ago
  If you register (takes 2 seconds) there's an example project where you can see the integration steps with the exact code you need to paste in. You're right though, we should definitely add examples to the main site.
  - tjbiddle 13y ago
    I'd recommend not forcing people to register to see generic information like this - I know HN is biased towards "Connect with Facebook to login", I would assume most are also biased towards registering to see content - I know I am.
    - nancyhua 13y ago
      Ok you guys are right. We're adding this to our site right now.
      - gailees 13y ago
        Thanks :)
      - jorlow 13y ago
        Updated: https://newton.apptimize.com/faqs.html https://newton.apptimize.com/faqs.html
        theelfismike 13y ago
        It'd be nice if the text were selectable instead of an image.
        MaxGabriel 13y ago
pr0filer__ 13y ago
I really like the flat design. Nice and subtle. Also, Optimizely reference?
huhtenberg 13y ago
I've got some karma points to burn, so here comes something. A list of your clients would be nice to see. So that I'd know to never install any of their apps. I hate apps that phone home. I hate apps that phone to some random analytics companies even more. I don't like companies that enable, entice and profit from this sort of behavior, a behavior that every single user would opt out from if they were asked upfront. So, unfortunately, the only thing I can wish you at this point is a speedy demise so that you'd be free to focus on projects that are less ethically questionable. /rant
- Rezo 13y ago
  How does the anonymous gathering of statistics harm you? As an app developer, this data allows me to measurably improve my apps. I have several free (no ads either) apps in the Android market, and even in the most trivial of apps I will include Google Analytics to receive crash reports, see what screens are most popular (so I can improve my apps where it matters most), where the app is being used (for some reason one of my apps was really popular in France, I then added a French translation and the adoption really took off), what devices are most/least popular etc. If I add a new feature, I want to know if its being used or if I wasted my time, if I need to improve the discoverability of the feature etc. Last but not least, the usage data is really motivating to me as a developer of free apps. If I release an app, and I see it being used daily, that makes me happy and motives me to keep updating it. As far as I know, the GA data is anonymous and only shown to the developer in aggregates.
  - lobotryas 13y ago
    I agree with you 100%. What it ultimately boils down to is the need for feedback. Publishing an app is a conversation with your users instead of throwing a message in a bottle into the ocean. You need all kinds of data and metrics not only to see how well your business is doing, but also how you can help users have a better experience with your app. For better or worse, until telepathic feedback is invented, phone-home is all we have.
  - nileshtrivedi 13y ago
    If he is using your app, it implies that he somewhat trusts you. I think it's Google that he doesn't trust. Using an app shouldn't require trusting a third party with personal behavior data.
Rezo 13y ago
I couldn't find any pricing information even after registering. It only says its free to try, but no further clarification in the docs or the FAQ. While it looks promising, I can't go and integrate this into my app without knowing if, when, and how much you're going to be charging for it.
- nancyhua 13y ago
  We haven't announced our pricing yet because we're constantly adding new features our users request. Email me at nancy at apptimize dot com and we'll give you a special deal we've been giving to early adopters.
  - hrabago 13y ago
    The lack of information on pricing is an instant back-button-trigger for me. I have no idea if you (eventually) plan to charge $5 per month or $200 per month. If it's in the lower end, then it can be worth evaluating. If it's a few or several hundreds per month, then I have to pass until I'm earning enough to afford it.
jayxni 13y ago
Great team working on Apptimize! Will definitely try it for our mobile app when we make that transition from web.
gailees 13y ago
Do people actually do A/B testing on iOS apps? I feel like the two-week development cycle really hinders your ability to conduct valuable A/B tests with your users in time to actually make any change.
- nancyhua 13y ago
  You're right the app store is a barrier but people still do it. That's one of the advantages of our product, that you can start and stop experiments so you can put in as many variations as you want. Currently people just suffer through the really slow time frame.
- kunle 13y ago
  Some of the more forward thinking folks (Socialcam and HauteLook for example) have rolled in house versions of these, but Apptimize is one of the first end to end solutions I've seen. But A/B testing in mobile is definitely something a lot of folks are thinking about.
- michaelmior 13y ago
  We build in DB-driven tests into our app so we can make changes at will once we find a winner. Then we just clean up the code to remove the test in the next iteration.
- jasonlotito 13y ago
  Take a look at http://useartisan.com/ http://useartisan.com/ Basically, it removes the 2-week development cycle, and you can essentially make changes in real time. I've seen it demo'ed, and it's much more impressive than the website makes it out to be.
  - SupremumLimit 13y ago
    It sounds interesting, but how do they get around the app store approval process?
    - jasonlotito 13y ago

126 points by jorlow 13y ago

sync 13y ago

For folks interested in mobile A/B testing, also see an open-source alternative by Clutch: https://github.com/clutchio https://github.com/clutchio

jorlow 13y ago
For what it's worth, using Clutch was one of the reasons we ended up making Apptimize. We ran into several bugs including one that was crashing the app we were working on and issues with its probability distribution. And the reporting was very simplistic. Are you using Clutch now? If so, it'd be awesome if you could give us a try.
siglesias 13y ago
Additionally, incumbent mobile A/B testing service http://www.leanplum.com http://www.leanplum.com -- I've used them for a while now and it's a pretty robust system with good support. Worth checking out IMO.
pbharrin 13y ago
ClutchIO is not native, which is why something like this is needed.

jakestein 13y ago

Interesting, Artisan mobile is also in this space http://useartisan.com/ http://useartisan.com/

igorsyl 13y ago

Why should anyone choose to use Apptimize over a free open-source alternative?

jorlow 13y ago
One of the biggest reasons is probably the reason behind using any software as a service rather than doing it yourself: you can concentrate on what you do best and not spend time managing things yourself. A couple of our users switched from using open source alternatives for that reason. We've put a lot of time into our infrastructure and decoupling things so we're very resilient against outages so you don't have to. We've also put a lot of time into making our library more robust and well tested than any alternatives we've seen to date.
chrisalvino 13y ago
I'm guessing because it's better than what they could integrate and build up with an open-source alternative in a short amount of time and thus more cost effective.

philbarr 13y ago

This looks interesting - although having never really considered using A/B testing on native apps before I did spend a few minutes looking around the site for examples of things to track. Maybe you could show an example project on the site showing what was changed and how that affected sales, etc.?

nancyhua 13y ago
Part of apptimize team here. We want to do that! We will add more to our site; currently it looks deceptively slick bc our designer is awesome but it's really a first iteration. Right now if you sign in, you can see an example experiment. How does it look to you?
- philbarr 13y ago
  Yes the example gives a much better idea of what's going on. Although I did have to create an account to look at it... Would be nice to have that accessible from the main page.

sirpillage 13y ago

Apptimize is far better than anything currently available!

chrisalvino 13y ago

Our organization often comes up with its own secondary metrics based on user data (not just CTR or usage). Is this possible when using Apptimize?

nancyhua 13y ago
Part of apptimize team here! We let you track whatever goals you want, so people measure stuff like "user returned within a week" and "user used app." What goals do you want to track?
- chrisalvino 13y ago
  That really depends on what we find to be important metrics. Flexibility is important since we need to be flexible too.

xoail 13y ago

I would like to see some code examples.

jorlow 13y ago
If you register (takes 2 seconds) there's an example project where you can see the integration steps with the exact code you need to paste in. You're right though, we should definitely add examples to the main site.
- tjbiddle 13y ago
  I'd recommend not forcing people to register to see generic information like this - I know HN is biased towards "Connect with Facebook to login", I would assume most are also biased towards registering to see content - I know I am.
  - nancyhua 13y ago
    Ok you guys are right. We're adding this to our site right now.
    - gailees 13y ago
      Thanks :)
    - jorlow 13y ago
      Updated: https://newton.apptimize.com/faqs.html https://newton.apptimize.com/faqs.html
      - theelfismike 13y ago
        It'd be nice if the text were selectable instead of an image.
        MaxGabriel 13y ago

pr0filer__ 13y ago

I really like the flat design. Nice and subtle. Also, Optimizely reference?

huhtenberg 13y ago

I've got some karma points to burn, so here comes something. A list of your clients would be nice to see. So that I'd know to never install any of their apps. I hate apps that phone home. I hate apps that phone to some random analytics companies even more. I don't like companies that enable, entice and profit from this sort of behavior, a behavior that every single user would opt out from if they were asked upfront. So, unfortunately, the only thing I can wish you at this point is a speedy demise so that you'd be free to focus on projects that are less ethically questionable. /rant

Rezo 13y ago
How does the anonymous gathering of statistics harm you? As an app developer, this data allows me to measurably improve my apps. I have several free (no ads either) apps in the Android market, and even in the most trivial of apps I will include Google Analytics to receive crash reports, see what screens are most popular (so I can improve my apps where it matters most), where the app is being used (for some reason one of my apps was really popular in France, I then added a French translation and the adoption really took off), what devices are most/least popular etc. If I add a new feature, I want to know if its being used or if I wasted my time, if I need to improve the discoverability of the feature etc. Last but not least, the usage data is really motivating to me as a developer of free apps. If I release an app, and I see it being used daily, that makes me happy and motives me to keep updating it. As far as I know, the GA data is anonymous and only shown to the developer in aggregates.
- lobotryas 13y ago
  I agree with you 100%. What it ultimately boils down to is the need for feedback. Publishing an app is a conversation with your users instead of throwing a message in a bottle into the ocean. You need all kinds of data and metrics not only to see how well your business is doing, but also how you can help users have a better experience with your app. For better or worse, until telepathic feedback is invented, phone-home is all we have.
- nileshtrivedi 13y ago
  If he is using your app, it implies that he somewhat trusts you. I think it's Google that he doesn't trust. Using an app shouldn't require trusting a third party with personal behavior data.

Rezo 13y ago

I couldn't find any pricing information even after registering. It only says its free to try, but no further clarification in the docs or the FAQ. While it looks promising, I can't go and integrate this into my app without knowing if, when, and how much you're going to be charging for it.

nancyhua 13y ago
We haven't announced our pricing yet because we're constantly adding new features our users request. Email me at nancy at apptimize dot com and we'll give you a special deal we've been giving to early adopters.
- hrabago 13y ago
  The lack of information on pricing is an instant back-button-trigger for me. I have no idea if you (eventually) plan to charge $5 per month or $200 per month. If it's in the lower end, then it can be worth evaluating. If it's a few or several hundreds per month, then I have to pass until I'm earning enough to afford it.

jayxni 13y ago

Great team working on Apptimize! Will definitely try it for our mobile app when we make that transition from web.

gailees 13y ago

Do people actually do A/B testing on iOS apps? I feel like the two-week development cycle really hinders your ability to conduct valuable A/B tests with your users in time to actually make any change.

nancyhua 13y ago
You're right the app store is a barrier but people still do it. That's one of the advantages of our product, that you can start and stop experiments so you can put in as many variations as you want. Currently people just suffer through the really slow time frame.
kunle 13y ago
Some of the more forward thinking folks (Socialcam and HauteLook for example) have rolled in house versions of these, but Apptimize is one of the first end to end solutions I've seen. But A/B testing in mobile is definitely something a lot of folks are thinking about.
michaelmior 13y ago
We build in DB-driven tests into our app so we can make changes at will once we find a winner. Then we just clean up the code to remove the test in the next iteration.
jasonlotito 13y ago
Take a look at http://useartisan.com/ http://useartisan.com/ Basically, it removes the 2-week development cycle, and you can essentially make changes in real time. I've seen it demo'ed, and it's much more impressive than the website makes it out to be.
- SupremumLimit 13y ago
  It sounds interesting, but how do they get around the app store approval process?
  - jasonlotito 13y ago