Some pretty easy rules of thumb, assuming you have a decent grasp on your economics. Look at it as a "low hanging fruit" optimization problem -- do you put res

Some pretty easy rules of thumb, assuming you have a decent grasp on your economics. Look at it as a "low hanging fruit" optimization problem -- do you put resources into running a test (+ opportunity cost for lost sales), or into something else? Suppose you have 10k monthly sessions with a 0.5% conversion rate (50 conversions). How many more customers would you need in order to prioritize running a test? If 55 conversions in a given month means you crush important KPIs, then that's probably worth testing -- you just need a 10% lift.* Also keep in mind that running A/B tests (1 control, 1 treatment) is suboptimal. That tests, "does this beat what I have now?" The more important question is "what is my best option?". OTOH, if other things like messaging and product are stable, you can test a smaller traffic site by leaving it running longer. My rough estimate is 100 conversion events in the time the test runs. So if I have 100 conversion events in 1 month, it may make sense to run a 2-3 option + 1 control test for 1 month. (You can also test much larger things than buttons. For startups, I like to suggest trying out positioning or value statements and seeing how visitors respond!) * however, it'll take a long time for you to reach statistical confidence for a 10% lift in rate, with only 50 conversion events across all tests.