6 ms·

Terence Tao Uses GPT-4 to Study Mathematics

irthomasthomas 3y ago
Sorry for the twitter link, but the original article[0] was already submitted 5 days ago and didn't get picked up, so I can't repost it. Thought HN would be interested in Tao using GPT-4 at work. [0] https://unlocked.microsoft.com/ai-anthology/terence-tao/ https://unlocked.microsoft.com/ai-anthology/terence-tao/ Edit: Some choice quotes from Tao. I could feed GPT-4 the first few PDF pages of a recent math preprint and get it to generate a half-dozen intelligent questions that an expert attending a talk on the preprint could ask. Strangely, even nonsensical LLM-generated math often references relevant concepts. With effort, human experts can modify ideas that do not work as presented into a correct and original argument. The 2023-level AI can already generate suggestive hints and promising leads to a working mathematician and participate actively in the decision-making process. When integrated with tools such as formal proof verifiers, internet search, and symbolic math packages, I expect, say, 2026-level AI, when used properly, will be a trustworthy co-author in mathematical research, and in many other fields as well
- supriyo-biswas 3y ago
  (Deleted)
  - irthomasthomas 3y ago
    His first use case > I could feed GPT-4 the first few PDF pages of a recent math preprint and get it to generate a half-dozen intelligent questions that an expert attending a talk on the preprint could ask. Is that not part of the process of studying math? Also, Strangely, even nonsensical LLM-generated math often references relevant concepts. With effort, human experts can modify ideas that do not work as presented into a correct and original argument. And, The 2023-level AI can already generate suggestive hints and promising leads to a working mathematician and participate actively in the decision-making process. When integrated with tools such as formal proof verifiers, internet search, and symbolic math packages, I expect, say, 2026-level AI, when used properly, will be a trustworthy co-author in mathematical research, and in many other fields as well.
- defrost 3y ago
  > Sorry for the twitter link, Try: https://publish.twitter.com/?query=https%3A%2F%2Ftwitter.com%2Fblader%2Fstatus%2F1670578014508433410&widget=Tweet https://publish.twitter.com/?query=https%3A%2F%2Ftwitter.com... The publish.twitter.com link generates publishable embed code for magazines, etc. I don't think they'll rate limit | login protect that one. Also, Terry on mathstodon on GPT-4 (3 days ago) https://mathstodon.xyz/@tao/110601051375142142 https://mathstodon.xyz/@tao/110601051375142142
  - irthomasthomas 3y ago
    Thanks, that's good to know for future.
- Loic 3y ago
  Thank you for the original link. Here is the point I like (my emphasis): > Current large language models (LLM) can often persuasively mimic correct expert response in a given knowledge domain (such as my own, research mathematics). But as is infamously known, the response often consists of nonsense when inspected closely. Both humans and AI need to develop skills to analyze this new type of text. The stylistic signals that I traditionally rely on to “smell out” a hopelessly incorrect math argument are of little use with LLM-generated mathematics. I found the same problem/challenge when I am using GPT-4 to dig into subjects which are tangential to my main expertise. The good thing, is that as I know the LLM can provide answers which are totally wrong, I am forced to be more critical of the answer than just reading a book on the subject. I am more active in exploring. Usually I am ending up with a chat and many open Wikipedia tabs and scientific papers.
  - irthomasthomas 3y ago
    Absolutely nailed it. I love articles by people like Tao and Wolfram as they tend cut the B.S. and get down to the real utility of it, rather quickly. I was also pleased, in a schadenfreude kind of way, that Tao followed the same dead-end paths as me, and I presume most others, when learning GPT-4. Like, starting out by trying to be very precise and descriptive, before throwing caution to the wind and embracing the non-deterministic nature of the thing, and just throwing a ton of keywords and loosely worded requests at it. The good thing, is that as I know the LLM can provide answers which are totally wrong, I am forced to be more critical of the answer than just reading a book on the subject. Yep, having to fact-check it's hallucinations has been far less detrimental than I expected. I find, often, if it's a subject I am vaguely familiar with, that the surprises jump out at me, then I can fact-check them and learn something. Actually, many times I was convinced it was hallucinating some cli tool or option flag, and it actually turned out to be correct. And, those times when we are embarrassingly wrong, tend to be the most instructive. Browsing mode, when used right, was a huge boost for this. I found the optimal use, was to structure a prompt like normal, as if targeting GPT-4 WITHOUT the browser mode, and then tack on to the end a carefully crafted search request, or two. This way, it writes a response first before performing the web searches and augmenting the answer. This acts like chain-of-thought reasoning by expanding the information in the initial prompt. And, as a bonus, it meant you had something to read while waiting for it to finish browsing.
- amadeuspagel 3y ago
  I think it's better to add a hash like #resubmit in this case.
aloe_falsa 3y ago
> One of my colleagues was moved to tears by a GPT-4-generated letter of condolence to a relative who had recently received a devastating medical diagnosis. Nice to know that Theodore Twombly's job from the movie "Her" has already been automated away. More to the point, I find it hilarious that in the near future AIs will be more creative, convincing, and compassionate than us, and humans will likely be relegated to menial jobs. This is not what sci-fi promised us.
- v64 3y ago
  A relevant comic from SMBC: https://www.smbc-comics.com/comic/sad-2 https://www.smbc-comics.com/comic/sad-2
- captainbland 3y ago
  I think most people will eventually find such AI generated materials to be "cheap". But since you won't be able to escape them in digital media, it may return value to handwritten notes and face to face conversations that social media has diluted over the previous decades.
  - jart 3y ago
    That's like saying people will eventually find the printing press cheap because our paperback novels weren't laboriously drawn by monks. People prefer the clarity and uniformity of mechanized typesetting over the artisticness of blackletter scribbled in cursive with a bird feather. CGI in movies is a similar story. People want to hear stories told by computers rather than puppets. People don't play dungeons and dragons anymore because it's now cheap to have AI generate realistic looking 3d fantasy worlds on the fly thanks to video games. Technology wins, every single time, because it gives us what we want and it solves the greatest problem people have always had, which is other people. Anyone who says differently is kidding themselves.
    - logicprog 3y ago
      > People don't play dungeons and dragons anymore You sure about that? Also, the examples you're giving aren't remotely analogous to AI-generated letters, because your examples are about the means of communication — the medium and presentation — changing, but the underlying meaningful content being communicated is still fundamentally generated by human beings, whereas with AI generated letters there's no underlying meaning or intentionality at all, because no human generated it. So it isn't just the medium that's being changed fundamentally this time, but what's being communicated with it. And we already have an equivalent of AI-generated letters: those premade cards for all occasions you can find at grocery stores. And sure enough those cards aren't valued that highly in comparison to handwritten notes or talking in person
jaapz 3y ago
Twitter links are pretty useless now that non-users can't see the tweets any more
- irthomasthomas 3y ago
  I know, I'm sorry. Another user shared this link that can be viewed logged out https://publish.twitter.com/?query=https%3A%2F%2Ftwitter.com%2Fblader%2Fstatus%2F1670578014508433410&widget=Tweet https://publish.twitter.com/?query=https%3A%2F%2Ftwitter.com... And the main article is here: https://unlocked.microsoft.com/ai-anthology/terence-tao/ https://unlocked.microsoft.com/ai-anthology/terence-tao/
wrp 3y ago
About 15 years ago, I spent a while on a project to train a language model on a corpus of math, with idea to create a math exploration assistant. The project foundered on my inability extract the typeset math and proofs from print materials into a consistent, machine-readable form. How did OpenAI do it? Don't tell me they OCRed Google Books.
- theGnuMe 3y ago
  Probably in combination with mechanical turks.

Sorry for the twitter link, but the original article[0] was already submitted 5 days ago and didn't get picked up, so I can't repost it. Thought HN would be interested in Tao using GPT-4 at work. [0] https://unlocked.microsoft.com/ai-anthology/terence-tao/ https://unlocked.microsoft.com/ai-anthology/terence-tao/ Edit: Some choice quotes from Tao. I could feed GPT-4 the first few PDF pages of a recent math preprint and get it to generate a half-dozen intelligent questions that an expert attending a talk on the preprint could ask. Strangely, even nonsensical LLM-generated math often references relevant concepts. With effort, human experts can modify ideas that do not work as presented into a correct and original argument. The 2023-level AI can already generate suggestive hints and promising leads to a working mathematician and participate actively in the decision-making process. When integrated with tools such as formal proof verifiers, internet search, and symbolic math packages, I expect, say, 2026-level AI, when used properly, will be a trustworthy co-author in mathematical research, and in many other fields as well

supriyo-biswas 3y ago
(Deleted)
- irthomasthomas 3y ago
  His first use case > I could feed GPT-4 the first few PDF pages of a recent math preprint and get it to generate a half-dozen intelligent questions that an expert attending a talk on the preprint could ask. Is that not part of the process of studying math? Also, Strangely, even nonsensical LLM-generated math often references relevant concepts. With effort, human experts can modify ideas that do not work as presented into a correct and original argument. And, The 2023-level AI can already generate suggestive hints and promising leads to a working mathematician and participate actively in the decision-making process. When integrated with tools such as formal proof verifiers, internet search, and symbolic math packages, I expect, say, 2026-level AI, when used properly, will be a trustworthy co-author in mathematical research, and in many other fields as well.
defrost 3y ago
> Sorry for the twitter link, Try: https://publish.twitter.com/?query=https%3A%2F%2Ftwitter.com%2Fblader%2Fstatus%2F1670578014508433410&widget=Tweet https://publish.twitter.com/?query=https%3A%2F%2Ftwitter.com... The publish.twitter.com link generates publishable embed code for magazines, etc. I don't think they'll rate limit | login protect that one. Also, Terry on mathstodon on GPT-4 (3 days ago) https://mathstodon.xyz/@tao/110601051375142142 https://mathstodon.xyz/@tao/110601051375142142
- irthomasthomas 3y ago
  Thanks, that's good to know for future.
Loic 3y ago
Thank you for the original link. Here is the point I like (my emphasis): > Current large language models (LLM) can often persuasively mimic correct expert response in a given knowledge domain (such as my own, research mathematics). But as is infamously known, the response often consists of nonsense when inspected closely. Both humans and AI need to develop skills to analyze this new type of text. The stylistic signals that I traditionally rely on to “smell out” a hopelessly incorrect math argument are of little use with LLM-generated mathematics. I found the same problem/challenge when I am using GPT-4 to dig into subjects which are tangential to my main expertise. The good thing, is that as I know the LLM can provide answers which are totally wrong, I am forced to be more critical of the answer than just reading a book on the subject. I am more active in exploring. Usually I am ending up with a chat and many open Wikipedia tabs and scientific papers.
- irthomasthomas 3y ago
  Absolutely nailed it. I love articles by people like Tao and Wolfram as they tend cut the B.S. and get down to the real utility of it, rather quickly. I was also pleased, in a schadenfreude kind of way, that Tao followed the same dead-end paths as me, and I presume most others, when learning GPT-4. Like, starting out by trying to be very precise and descriptive, before throwing caution to the wind and embracing the non-deterministic nature of the thing, and just throwing a ton of keywords and loosely worded requests at it. The good thing, is that as I know the LLM can provide answers which are totally wrong, I am forced to be more critical of the answer than just reading a book on the subject. Yep, having to fact-check it's hallucinations has been far less detrimental than I expected. I find, often, if it's a subject I am vaguely familiar with, that the surprises jump out at me, then I can fact-check them and learn something. Actually, many times I was convinced it was hallucinating some cli tool or option flag, and it actually turned out to be correct. And, those times when we are embarrassingly wrong, tend to be the most instructive. Browsing mode, when used right, was a huge boost for this. I found the optimal use, was to structure a prompt like normal, as if targeting GPT-4 WITHOUT the browser mode, and then tack on to the end a carefully crafted search request, or two. This way, it writes a response first before performing the web searches and augmenting the answer. This acts like chain-of-thought reasoning by expanding the information in the initial prompt. And, as a bonus, it meant you had something to read while waiting for it to finish browsing.
amadeuspagel 3y ago
I think it's better to add a hash like #resubmit in this case.

aloe_falsa 3y ago

> One of my colleagues was moved to tears by a GPT-4-generated letter of condolence to a relative who had recently received a devastating medical diagnosis. Nice to know that Theodore Twombly's job from the movie "Her" has already been automated away. More to the point, I find it hilarious that in the near future AIs will be more creative, convincing, and compassionate than us, and humans will likely be relegated to menial jobs. This is not what sci-fi promised us.

v64 3y ago
A relevant comic from SMBC: https://www.smbc-comics.com/comic/sad-2 https://www.smbc-comics.com/comic/sad-2
captainbland 3y ago
I think most people will eventually find such AI generated materials to be "cheap". But since you won't be able to escape them in digital media, it may return value to handwritten notes and face to face conversations that social media has diluted over the previous decades.
- jart 3y ago
  That's like saying people will eventually find the printing press cheap because our paperback novels weren't laboriously drawn by monks. People prefer the clarity and uniformity of mechanized typesetting over the artisticness of blackletter scribbled in cursive with a bird feather. CGI in movies is a similar story. People want to hear stories told by computers rather than puppets. People don't play dungeons and dragons anymore because it's now cheap to have AI generate realistic looking 3d fantasy worlds on the fly thanks to video games. Technology wins, every single time, because it gives us what we want and it solves the greatest problem people have always had, which is other people. Anyone who says differently is kidding themselves.
  - logicprog 3y ago
    > People don't play dungeons and dragons anymore You sure about that? Also, the examples you're giving aren't remotely analogous to AI-generated letters, because your examples are about the means of communication — the medium and presentation — changing, but the underlying meaningful content being communicated is still fundamentally generated by human beings, whereas with AI generated letters there's no underlying meaning or intentionality at all, because no human generated it. So it isn't just the medium that's being changed fundamentally this time, but what's being communicated with it. And we already have an equivalent of AI-generated letters: those premade cards for all occasions you can find at grocery stores. And sure enough those cards aren't valued that highly in comparison to handwritten notes or talking in person

jaapz 3y ago

Twitter links are pretty useless now that non-users can't see the tweets any more

irthomasthomas 3y ago
I know, I'm sorry. Another user shared this link that can be viewed logged out https://publish.twitter.com/?query=https%3A%2F%2Ftwitter.com%2Fblader%2Fstatus%2F1670578014508433410&widget=Tweet https://publish.twitter.com/?query=https%3A%2F%2Ftwitter.com... And the main article is here: https://unlocked.microsoft.com/ai-anthology/terence-tao/ https://unlocked.microsoft.com/ai-anthology/terence-tao/

wrp 3y ago

About 15 years ago, I spent a while on a project to train a language model on a corpus of math, with idea to create a math exploration assistant. The project foundered on my inability extract the typeset math and proofs from print materials into a consistent, machine-readable form. How did OpenAI do it? Don't tell me they OCRed Google Books.

theGnuMe 3y ago
Probably in combination with mechanical turks.