Well, it is assumed they are speaking the truth, and it is assumed that they are good mathematicians (i.e. if they could know what x and y are at some point, th

Well, it is assumed they are speaking the truth, and it is assumed that they are good mathematicians (i.e. if they could know what x and y are at some point, they would know). The idea is that if Bob says that he doesn't know what x and y are, then Alice can deduce some things from that, because she knows for which x and y he would know what they are. For example, after the first time Bob says he doesn't know Alice can deduce that x=1 y=1 is not the solution, because then Bob would know (because Bob gets told 1^2 + 1^2 = 2, and 2 = x^2 + y^2 implies x=1 y=1).