I presume he means he puts all users who arrive on an even day into the control and all users who arrive on an odd day into the "show a phone number" bucket. Y

I presume he means he puts all users who arrive on an even day into the control and all users who arrive on an odd day into the "show a phone number" bucket. You then use cookies to track which user is in what bucket, because you have to know which treatment the user was in when they converted. As you said, that may or may not be on the same day they were treated. That's still kind of weird, though. You should just randomly assign each user as they arrive to one group or the other. If he's doing what you're suggesting that's bad methodology for all the reasons you said, and I wouldn't trust the results.