Is there something a candidate can do to get out of the lunch interview? It seems like it might be one of those things that both sides put up with in order to

Is there something a candidate can do to get out of the lunch interview? It seems like it might be one of those things that both sides put up with in order to seem polite, but neither is actually interested in. I've been in the lunch interviewer situation before and would have been ecstatic if the candidate said, "I'm actually unable to go to lunch due to [some plausible excuse], can we reschedule for 1pm?" but generally, the schedule was already set at that point.