Statisticians have spent time thinking about the right way to deal with these sorts of problems for a long time: https://en.wikipedia.org/wiki/Sequential_analys

Statisticians have spent time thinking about the right way to deal with these sorts of problems for a long time: https://en.wikipedia.org/wiki/Sequential_analysis https://en.wikipedia.org/wiki/Sequential_analysis. Funnily enough, the page they reference for calculating the right sample size actually talks about sequential analysis, but AirBnB doesn't mention this in describing their solution...