I'm a cofounder of a small (and growing :-)) startup called Mito [1]. We don't do A/B testing currently, for two main reasons: 1. We're a locally installable p

I'm a cofounder of a small (and growing :-)) startup called Mito [1]. We don't do A/B testing currently, for two main reasons: 1. We're a locally installable product with opt-in updates. 2. We don't get enough new users per week to make the turnaround time on experiments quick enough. We're big proponents for local-first software, so though we might have a hosted offering at some point for users who prefer that, the locally-installable + opt-in updates are gonna be around for a while. These make A/B testing hard for obvious reasons - it's no longer just flipping a switch to get both sets of users on the same branch after the experiment terminates, nor is it as easy to randomize people into two different groups. We also just don't get enough new users a week to make the experiments make sense. For the effect sizes we're hoping to measure, we'd have to wait a few weeks to to draw conclusions - and a given that it's even more complicated to run more than one experiment at once (given the above local install + opt-in upgrading mentioned), it becomes really expensive to do any sort of A/B testing. There's also the question that this post leaves out: what changes are worth A/B testing in the first place? I know folks at a gaming company that A/B tests every single change they make to their games for the effect it has on rev/user. They are a mature company, and the changes they make to their apps are, on balance, fairly small. This makes sense to me, especially given the tooling they have built to facilitate this - although it is clear they are just trying to extract value from their existing user base rather than dramatically improve their product and grow. For early-stage startups, IMO often the best bet is to be a user of your own product, and to just test your product changes on yourself - it's usually pretty obvious which direction you should take things. We recently overhauled our graphing capabilities to add about 5x more graph types and actual graph configuration options. Given how limited our previous graphing capabilities were, was pretty much a no brainer and obviously better. A/B testing would have just been a waste of time, methinks! Feedback and thoughts on the above greatly appreciated. We're always looking for ways to improve our product/technical processes! [1] https://trymito.io https://trymito.io