It's not that hard to give it problems it hasn't seen - you can take a classic description of a logical thinking exercise the text for which does occur online,

It's not that hard to give it problems it hasn't seen - you can take a classic description of a logical thinking exercise the text for which does occur online, then mix it up in ways that doesn't change the underlying pattern of reasoning necessary to solve it, and at least from the tests I've done it will confidently tell you the incorrect answer (along with some semi-plausible but fatally flawed description of the reasoning it used to come up with the answer). In at least one case it was certain the answer was exactly the answer given in the common online statement of the problem, despite the fact I'd changed all the inputs such that the given answer was obviously not an option (it was a guess so-and-so's birthday type problem). Even for simple arithmetic just choosing sufficiently large numbers will bring it unstuck.