I think the multi-armed bandit problem would assume that the probability distribution, for each choice, would remain a constant for the duration of the problem.

I think the multi-armed bandit problem would assume that the probability distribution, for each choice, would remain a constant for the duration of the problem. Diminishing returns would imply that some subset of those choices would have decreasing probability of a positive pay-off the more they are used. This would be more realistic, and much harder to model. With the constant probabilities with the payoff being for a single score, it makes sense to just search for the highest expected value score. But that expected value relies on the probability distribution remaining the same!