I dug deeper in another comment: https://news.ycombinator.com/item?id=37517597 https://news.ycombinator.com/item?id=37517597 — it's mentioned in many books, but

I dug deeper in another comment: https://news.ycombinator.com/item?id=37517597 https://news.ycombinator.com/item?id=37517597 — it's mentioned in many books, but Newton's very thorough biographer Richard Westfall authoritatively says “According to a story that rests solely on anecdotal authority, he spoke only once”. So in fact we can be fairly confident it's only an anecdote and nothing more.