16 ms·

Terence Tao on GPT-4

207 points by admp 4y ago

endisneigh 4y ago
wow, the site is so slow. getting "This resource could not be found" errors on reloads. anyway, here's the text: > Today was the first day that I could definitively say that #GPT4 has saved me a significant amount of tedious work. As part of my responsibilities as chair of the ICM Structure Committee, I needed to gather various statistics on the speakers at the previous ICM (for instance, how many speakers there were for each section, taking into account that some speakers were jointly assigned to multiple sections). The raw data (involving about 200 speakers) was not available to me in spreadsheet form, but instead in a number of tables in web pages and PDFs. In the past I would have resigned myself to the tedious taks of first manually entering the data into a spreadsheet and then looking up various spreadsheet functions to work out how to calculate exactly what I needed; but both tasks were easily accomplished in a few minutes by GPT4, and the process was even somewhat enjoyable (with the only tedious aspect being the cut-and-paste between the raw data, GPT4, and the spreadsheet). > Am now looking forward to native integration of AI into the various software tools that I use, so that even the cut-and-paste step can be omitted. (Just being able to resolve >90% of LaTeX compilation issues automatically would be wonderful...) Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? In this case GPT-4 is used to solve a problem that shouldn't have even been one to begin with. The administrators of the ICM could've simply exported the raw data as a Google Sheet (for example) and his problem could've been trivially solved even without GPT-4.
- sebzim4500 4y ago
  Yeah bold prediction I know but I think basically everyone will move back to twitter by the end of the year
  - bantunes 4y ago
    > Yeah bold prediction I know but I think basically everyone will move back to twitter by the end of the year You do know Mastodon is a federation of servers, right? I predict most people will move off this particular one to a faster one.
    - sebzim4500 4y ago
      Sure, but on twitter you don't have to think about which federation you are on
- aphit 4y ago
  Yeah, I waited a good minute of it trying to load before I gave up. Anyone that got in can share what it says?
  - yellow_lead 4y ago
    Check the comment to which you replied (maybe it was added after)
  - kzrdude 4y ago
    I accessed the message through a mastodon client instead. They are a lot better at federation than the various instance sites IMO, a more consistent interface when there is no option of sending you out to a non home instance page.
    - SkyMarshal 4y ago
      I didn’t even know there were mastodon clients, though I should have assumed. Know and good ones for Linux and Mac?
      - kzrdude 4y ago
        I use Tusky/Android and that's the limit of my knowledge
- red_trumpet 4y ago
  HackerNews' hug of death hitting a site with normally 4k active users. Not that surprising...
  - endisneigh 4y ago
    it's plain text. it can be cached. it's embarrassing.
    - blowski 4y ago
      Perhaps they've optimised for the normal levels of traffic they receive, not unanticpated spikes. That's hardly an embarrassing choice.
      - endisneigh 4y ago
        I'm not talking about the administrator of this instance, I'm talking about how mastodon is fundamentally designed. Even an iPhone could probably serve the (static) text of Tao's post to a million people on a 5G connection. Computers are fast.
        blowski 4y ago
        I run Mastodon on a $5 VM. If I hit HN homepage, my site will probably be slow as well. I don’t see how that reflects on Mastodon though.
        endisneigh 4y ago
        what am I saying is that there's a way to design a site on a $5 VM that serves static text that wouldn't be slow even if it was #1 on HN. if you disagree with that then we'll have to agree to disagree.
    - shakow 4y ago
      Feel free to share your know-how https://github.com/mastodon/mastodon https://github.com/mastodon/mastodon
- webdood90 4y ago
  > Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. incredible what mental hoops people will jump through to disqualify AI! the reality is that a lot of things have a terrible UI with unorganized data. that's why this tool is so amazing - because it doesn't matter anymore.
  - endisneigh 4y ago
    > the reality is that a lot of things have a terrible UI with unorganized data. that's why this tool is so amazing - because it doesn't matter anymore. how naive. how do you know it's right? ah, you have to manually do the calculation anyway to confirm, this is what Tao ended up saying in a reply asking as much. AI is great, but it's not a silver bullet, since its correctness can never be 100% under the current LLM framework.
    - webdood90 4y ago
      I'm convinced people like you are in for such a rude awakening. I don't understand how you can hold this position with AI considering it's only the beginning.
      - endisneigh 4y ago
        ChatGPT is impressive, but I do not think it is a silver bullet, even for data cleansing and processing tasks. I would use your words and say people who think (that ChatGPT solves the problem of data cleansing once and for all) so are the ones who will be in for a rude awakening.
      - ipaddr 4y ago
        I can't wait for 3d printers to take over manufacturing or self driving cars to self drive. Still waiting for the metaverse. Or cryto to take over banking. We are not even at those points for AI. You might see the future.. let's get a 3d printer in every home first.
        4y ago
- hn_throwaway_99 4y ago
  > Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Umm, and I want a pony? The world doesn't come in perfectly structured data formats. I was pretty amazed when I pasted in some of my medical lab test results and ChatGPT was accurately able to parse everything out and summarize my data. It worked extremely well.
  - endisneigh 4y ago
    > The world doesn't come in perfectly structured data formats. In Tao's case, the data was organized already, simply not given to him. In any case, Tao, per the post, had to manually calculate to confirm that GPT-4 was correct in any case (he implies as much in a comment asking if he checked for correctness).
    - WastingMyTime89 4y ago
      Checking a table for correctness and filling up a table are not the same thing. I also check for correctness when I manually count entries. It's an extremely error prone task.
      - endisneigh 4y ago
        How exactly could you check for the correctness without effectively finding out the answer? I mean this discrepancy is fundamentally the problem of P = NP or not.
        wetmore 4y ago
        NP is the class of problems for which it's hard to find the answer but easy to verify. The fact such a class even exists should hint to you that finding the answer and checking correctness can be two different things.
        endisneigh 4y ago
        the point of my comment is that whether P = NP to begin with is still to be determined. in the case of ChatGPT, it's easy for it to respond, but how difficult is it for you to verify that it was correct, or not? how does that compare with the difficulty in doing the task to begin with?
- fauxpause_ 4y ago
  Solving for every last mile problem in UX is hard though. A lot of AI value is being a flexible, generic interface to software automation.
  - SkyMarshal 4y ago
    We can probably even think of AI as a generalized meta-interface to everything - there’s probably no better UI/UX than natural human language that we’ve all been speaking since childhood.
    - fauxpause_ 4y ago
      I don’t know that it’s a better UX, but it is a viable UX, and that is very valuable
- rvz 4y ago
  > wow, the site is so slow. getting "This resource could not be found" errors on reloads. Oh dear. Federation at its finest /s If the instance is too small, it will easily fall over under heavy traffic. If it is too big, it is highly centralized and even then it cannot scale at a maximum of 300,000+ users at the same time. Eventually, they would all be re-centralizing back to Cloudflare. Once that goes down, all of the biggest Mastodon instances go down at once. As for GPT-4 in mathematics (LLMs in particular), in goes inline on what I said before. Fundamentally, these LLMs cannot reason transparently, nor can it directly bring up its own mathematical proofs with thorough explanations that allows mathematicians to work with. Only low hanging fruit of summarization of existing text. If Mr. Tao can see its limitations, surely it puts the hype of unexplainable magical black-box LLMs to rest as being great sophists and bullshit generators.
  - williamstein 4y ago
    I would not discount the potential of LLM's to support research level mathematics. There's interesting research going into combining the LEAN theorem prover with LLMs, e.g., this recent paper: https://arxiv.org/abs/2202.01344 https://arxiv.org/abs/2202.01344 "... automatically solving multiple challenging problems drawn from high school olympiads." This combination of GPT and LEAN is the first thing that has finally got me really interested in the potential of LEAN. And LEAN very directly addresses the "bullshit" issue, since it formally verifies whether or not a proof is correct (it's like a compiler like Rust, but for mathematical proofs).
- 77pt77 4y ago
  https://archive.is/wbwUa https://archive.is/wbwUa
- HelloMcFly 4y ago
  > In this case GPT-4 is used to solve a problem that shouldn't have even been one to begin with. But the problem DID exist, and problems like that are likely to continue to exist for the foreseeable future.
- helen___keller 4y ago
  > Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? I’m fairly convinced that for many problems (including this), AI is not the best solution but will become the preferred solution. “Best” doesnt get in the way of “good enough” when convenience is at stake (and making a good UX is often very very inconvenient)
  - teleforce 4y ago
    I think it's not the case of AI or ChatGPT is the best solution because it can be a local optima. But it's currently the "best" solution available to the average Joe. The fact in this OP case study the average Joe is probably one of the best mathematicians of our time, makes the situation much more interesting and intriguing. I'm in the middle of reading Steve Jobs' very own words, Make Something Wonderful book. Apparently throughout his whole life he frequently mentioned and discussed a lot about providing and creating easy access to the computer because ultimately people do not want to program but want to use computer instead. He did mention about how Morse code was not popular as much of the later telephone technology because people refused to learn and use the unintuitive Morse code even though it only takes 40 hours for average person to learn the entire Morse code. As a trained communication engineers I can very much relate to this because even though we learnt much of the underlying technology of communication that enable the Morse code or the telephone to work across the Atlantic ocean most of us can't be bothered to learn the Morse code and use it if we can help it because the telephone make it redundant and it's counter intuitive to use. Granted, now we know that telephone or circuit switching in general is a suboptimal in providing and solving the human communication problems and needs. As of now the Internet, packet switching and the multimedia approaches are probably the best solutions but telephone has served us a potent and still one of the best solutions for communication until very recently.
- poulpy123 4y ago
  > Why was the data so disjoint to begin with? Because real life is messy. > Why is Latex so hard to work with? Because people that use latex want to be able to brag they use latex (just joking for the second one.... unless ?)
  - endisneigh 4y ago
    > Because real life is messy. This doesn't make sense - the data inherently is centralized. Tao implied as much.
- hzay 4y ago
  Imagine Terrence Tao manually entering data into spreadsheets!!
- SkyMarshal 4y ago
  > Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? > In this case GPT-4 is used to solve a problem that shouldn't have even been one to begin with. Friction. These problems shouldn’t exist, but they do anyway, and they’re everywhere. Anything human is inevitably going to be imperfect and messy to some greater or lesser degree, introducing friction into dealing with it. Especially as we produce more and more unstructured or semi-structured data, which AI is particularly good at wrangling. If AI can help us cut through that friction significantly faster and more accurately, that’s a win.
  - moffkalast 4y ago
    Let's just hope that someone doesn't task the AGI with eliminating friction and it realizes that humans are the problem.
  - layer8 4y ago
    Ironically, interacting with ChatGPT still comes with a lot of human-like friction and imperfection.
    - dwringer 4y ago
      IME most of that is of the form "I'm sorry, but as a large language model I..." and seems to have been drilled into it with training to dodge tricky subjects.
- Jeff_Brown 4y ago
  Better source data is indeed a better solution, but the source data is often not under one's control.
- dpcan 4y ago
  They should have done it right the first time? That's not the world we live in. Coworkers, associates, clients, family - everyone sends us funky data. Sometimes it's a PDF of text instead of a text file, or a picture of their computer screen with a paragraph in it, or copy-pasted text with unwanted formatting, etc, etc. And the point of this article wasn't really about the tedious job of copy/pasting, it was that once the data was ready, instead of hunkering down and having to work through a problem with an unknown number of variables, GPT-4 can drop an answer in seconds flat. Maybe a few more if you have to tweak your prompt a few times. Part 2 here is that we have a problem to solve, and that's how data is sent and received, and that it all needs to end up in a structural format that's usable by our AI tools. I am thinking that writing a script that can dump textual data from literally anything I'm looking at so it's AI-ready makes a lot of sense, then we have step 1 taken care of as well for my own personal GPT-solvable situations.
- PurpleRamen 4y ago
  > Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Because the world is not a database. His source were formats meant for Human consumption, not machines. This will never change, people are lazy, greedy, or fear "data leaks", so we will never have a machine-first-format that everyone will use. I mean, that guy is using latex, others have markdown, or org-mode, or Microsoft Word, those all are not meant for easy scripting. This is why AI will be such a dealbreaker, because it will close this gap, make human-formats to machine-formats when neccessary.
  - rhn_mk1 4y ago
    I find the "human consumption" here funny, because it's the "human" part which led the author to turn to machines which can do it better.
    - pegasus 4y ago
      Well, yes, this is kinda the whole point of the argument. That's what LLMs bring to the table - machines that can better deal with data and documents meant for human consumption.
- belugacat 4y ago
  And here’s an exchange about how he validated the data was correct - which to me seems like the most important part: > Out of curiosity, are you sure that GPT did it correctly? If yes, is it because you were able to "spot check" it in a few places? Or have you used GPT enough that you trust it with a task like this? Or is this for some kind of internal use where a few small errors is unimportant, and you only need the broad strokes to be correct? > Yes, yes, and yes. (There were some obvious checksums, for instance by computing the total number of speakers several different ways.) Still cool, but the circumstances above aren’t always true. In fact, in a lot of meaningful work, none of them probably are. If GPT is like an assistant whose work you always have to double check in order to be sure of the validity of the results, that seems to be a pretty big caveat to me. In that case it’s probably better to just write a script whose output you know to be deterministic, even if it takes slightly longer (and the bigger your dataset to verify, the more likely it is that it’ll take less time to write a script than to validate results one by one). The scary part is if/when companies/bureaucracies/governments/etc start using GPT for all sorts of Important Tasks and skip the validation part because they assume the machine will always get it right.
  - humanistbot 4y ago
    > If GPT is like an assistant whose work you always have to double check in order to be sure of the validity of the results, that seems to be a pretty big caveat to me. Sounds like every junior developer fresh out of college I've had to work with. Interns are even worse. Even senior devs mess up from time to time. That's why we do code review and other QA. Trust no one.
    - belugacat 4y ago
      True, but one obvious difference is that a senior contributor can typically validate the work of a more junior one in less time than it took them to produce, that’s why the overall equation works. And if you decide to keep the more junior person, it’s probably because you feel like they’re improving and becoming more trustworthy over time. Another key difference is that a deterministic script that has determined to be valid will be valid for all subsequent runs, regardless of input size. That is not the case for ChatGPT, which might be correct on some runs and not others, or do well on small input sizes but start messing up when thousands (or more) of data items are involved. Again, I still think ChatGPT is really cool, but thinking about all these nuances about the nature of the work that is actually being done when you use ChatGPT vs writing a script vs handing it off to an intern seems crucial to the debate.
- 6gvONxR4sf7o 4y ago
  > AI, though amazing, isn't really the solution. Better UX and data quality is. In this case, AI was the tool used to have a better UX. People build something with use-case X in mind, and use the best tool for that job. Piping that into use-case Y requires some duct tape and plumbing. It turns out that AI is great at that sort of repurposing. It's great if someone using the best tool for their job is nearly the best tool for the flexible infinity of other use cases, which is what flexible enough duct-tape gets us.
- sdfghswe 4y ago
  Wait.... is it possible to tell chat gpt to grab data from a page?
- ugh123 4y ago
  >Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? Do you mean build better, specific, UX and normalize data for only this task? Why spend that time, even as a developer of whatever improved app/system you're thinking of, when he can just turn to AI? I'd rather lean on GPT4's ability to generalize highly specific technical work rather than ruminate on each individual app's lackings and how it could be better than using AI if we just... put more effort into it?
- gvhst 4y ago
  And AI is a way to enable better UX and cleaner data...
- jacquesm 4y ago
  And how does he know the results are correct without verifying each and every entry?
  - tjr 4y ago
    I find this a real concern when it comes to the proposition of "replace every programmer with GPT". I think the reality is, there are cases where it matters, and cases where it doesn't matter (at least, not very much). I work on aerospace software systems. I suspect that most software developers would be surprised at the process and rigor involved. That does not mean that humans are flawless; of course not. But there are going to be a lot more hurdles to replacing all of the humans with non-deterministic machines. There are numerous software components on a modern airplane. How good is good enough? If the overall system works correctly 99% of the time, is that good enough? There are about 16 million flights annually in the United States alone. If 160,000 of them crash, is that good enough? I mean, like, if Microsoft Word failed to save your document 99% of the time, you might be irked. If Professor Tao's data was only 99% accurate, he probably still has a reasonably good picture of the information he needs. Regardless of if a human or a machine is writing the code, there are different levels of software criticality. As humans, we don't apply avionics software development process to iOS games because it's not worth the extra time and expense. Likewise, there will be a spectrum of where it makes good enough sense to automate tasks with AI and where it doesn't. Which is not to dismiss AI! It's amazingly useful technology. But there really are places where above average accuracy is needed, and AI that works great for handling some tasks might not suffice for other tasks. If you truly need all of the results correct, then a nondeterministic neural network is probably not the best path to be on. There are other methods; even other automated methods.
    - jacquesm 4y ago
      > I work on aerospace software systems. I suspect that most software developers would be surprised at the process and rigor involved. I did this once, a fuel estimation program for 747's, the degree of understanding of the problem required to create something that would pass review was off the scale compared to any other program that I've ever written. It is also the only time that the customer wanted to get it right rather than that they were looking at what it cost (because the savings would pay for the development many times over this was probably less of an issue anyway). I'd love to be able to always work like that.
- csomar 4y ago
  > Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? AI is just that. A UI/UX for any/your data.
fifilura 4y ago
https://web.archive.org/web/20230412131029/https://mathstodon.xyz/@tao/110172426733603359 https://web.archive.org/web/20230412131029/https://mathstodo...
Rescis 4y ago
Archive Link: https://web.archive.org/web/20230412131029/https://mathstodon.xyz/@tao/110172426733603359 https://web.archive.org/web/20230412131029/https://mathstodo...
deleted 4y ago
[deleted]
jacknews 4y ago
I mean, one of the leading mathematicians at the moment, is expected to spent his time collating conference statistics? Does he not have an assistant for this kind of thing?
- jostmey 4y ago
  It’s academia. Only admins get secretaries. I’m only joking a little bit, as I once was a professor
  - robertlagrant 4y ago
    I was going to make the same joke! If he were an Assistant Vice Under-Dean he'd have 10 assistants, but sadly he's just an academic.
- kevmo314 4y ago
  Seems like his assistant is GPT-4 :)
- deleted 4y ago
  [deleted]
- NhanH 4y ago
  Terence Tao makes about only half a million or so every year (his salary at UCLA is public). Not sure about other income, but it is pretty much just the same ballpark as good doctors, lawyers, accountants at big 4 etc. At that range of income, it’s job-dependent on whether they would have an assistant or not, so no surprise here either way.
  - jacknews 4y ago
    So you think 'good accountant' and 'world class mathematician' are equivalent, based on their salary?
    - NhanH 4y ago
      No I don’t, but they have to pay for assistant all the same. I was describing how things are, not how they should be. University could have paid for Terence Tao’s assistant, but then they could have paid him more too.
amadvance 4y ago
Terence Tao doing a data entry job ? The Peter Principle strikes again! https://en.wikipedia.org/wiki/Peter_principle https://en.wikipedia.org/wiki/Peter_principle
- archgoon 4y ago
  [dead]
- nequo 4y ago
  Wouldn’t the Peter principle in this situation be more like a very good data enterer getting the job of Terence Tao?
  - squokko 4y ago
    LOL, this is funny. I'm imagining some bushy tailed assistant finishing all the data entry and asking for another task and being told to prove the Collatz Conjecture.
sampo 4y ago
Site took 37 seconds to load.
m_mueller 4y ago
how do you upload PDFs to GPT-4, is it available through the ChatGPT interface?
commandlinefan 4y ago
I used GPT-3 yesterday for the first time myself - I wanted some code to list the DynamoDB tables available to me (I had to run it from an EC2 instance and I wanted to see what that particular EC2 instance could see). I was pretty impressed - it put the code together so I asked it for a POM and it put that together. The POM failed with an error because it was trying to load from Maven central using HTTP, so I asked ChatGPT how to fix that error and it told me how to update my settings.xml file (it did make a mistake, but it was an easy one to fix). Like - I knew how to do all of that, but it probably would have taken a half hour vs about 5 minutes with ChatGPT.
- RugnirViking 4y ago
  exactly. It makes mistakes, but so do I. It makes mistakes far faster however, because it takes about 5 seconds to finish each attempt, so they seem far more prevalent. However, I can fix it's mistakes almost as fast as I can fix my own, so this speeds up iteration nicely. It's not a "perfect tool", but nothing is
- elboru 4y ago
  ChatGPT is great with well known tools and libraries. I’ve been able to build HTML/JS apps in minutes. Then I tried building a MAUI app, it started pretty good, but then when I tried to add charts it failed (it’s important to mention this was my first MAUI app), I tried different approaches, in the end I went back to HTML/JS. That led me to a realization, this could mean that people will likely discard less known libraries or technologies more easily than before. Popular technologies will become even more popular since we’ll be 10x more productive with them compared to less known ones since we won’t have LLMs super powers for those.
  - burlesona 4y ago
    I agree and have had the same thought. This is gas on the fire to winner-take-all dynamics. It’s already hard for new languages and systems to catch up fast enough to technically weaker existing options bc they have mature tooling. But when the robot is fluent in Java and can’t understand $newlang it’s going to be even harder.
throwsalj2rttw 4y ago
I'd really like to see sessions of people using things like ChatGPT for non-trivial things. People keep saying it saves them time and look forward to the future. For the life of me I can't figure out how to use it.
- webdood90 4y ago
  complicated tasks are just a series of trivial tasks, wouldn't you agree?
  - bumby 4y ago
    It depends on the context. A complicated spider web is just a series of connected strands, but pulling on one impacts all the others to various degrees. The point being, a contextual understanding of the systemic effects becomes important when deciding what to do on each "trivial" task. To the OPs point, I'm not sure it's been convincingly shown that ChatGPT has a strong contextual understanding. (in fact, that's also a major shortcoming of humans when we over simply complex models)
  - kgwgk 4y ago
    Heart surgery is just a series of simple cuts and stitches!
- laichzeit0 4y ago
  Depends on what you do. I had some Pandas code that I wanted to be slightly faster. I pasted it in and asked it to optimize the code. It did exactly that, with an explanation at the bottom of why it was doing each particular thing. It was correct, the code was faster. I’ve used it for a bunch of things like that.
  - Jeff_Brown 4y ago
    How much text was the code you fed it?
- golol 4y ago
  I used GPT-4 to write a simple chat website so I can talk to it without having to do manual API calls. It created a working version on the first try and me telling it about a few errors. In an hour I had improved it to a really polished look. I recently used GPT-4 for matplotlib. I wrote a imple PDE solver and wanted it to create a function that saves a simple 3d array as an animatied 2d plot. It did it right away. I could ask it for improvements and it did it too. Both of these tasks are easy, and if you are working every day in web development or with matplotlib I am sure you can do them in 5 minutes. But in my case, each of them might have taken half a day. And even if I could do it in 1 hour, that would be 1 hour of furrowed-brows staring at stackoverflow. Using GPT-4 is just extremely easy. From my experience I claim that GPT-4 can also solve more complex problems. I think if I iteratively ask for features on top of what it has given me, I can get up to 3-4 times the amount of features before the whole code becomes too complex to handle. This is just a guess.
pgt 4y ago
How are folks parsing PDFs with GPT4? Every 3rd party tool that supposedly uses GPT has failed for me on even the most regular of PDF bank statements. Currently I am parsing the text contents manually using pdfbox to reconstruct historical bank statements, but this is error-prone due to page breaks and run-on lines.
- zachwill 4y ago
  Note: I haven’t worked with PDF bank statements. My current solution is pdfplumber → GPT-3 API. I played around with a few different options, and this is personally what’s worked best for my use cases.
- MilStdJunkie 4y ago
  Same deal for me, I've tried a few AI PDF pipelines with no good output. If I feed it cleaned up delimited text then everything's good[1], but by the time I get to that point I might as well be using Tabula + Orange / R. I'll keep trying though, because damn, PDF files need to go die under a rock. [1] Well . . kinda. The model has some weird ideas about what an edge table is.
- Jeff_Brown 4y ago
  I'm guessing that manually feeding it a table at a time, with prompts tailored to the shapes you see in that table, is more effective than using a tool that tries to do that for you. Automated tools on top of GPT will be great when they work but they introduce a new communication joint where things can fail.
- ragazzina 4y ago
  I think he just copy-pasted everything from the PDF to GPT4? He says: > with the only tedious aspect being the cut-and-paste between the raw data, GPT4, and the spreadsheet
- Ductapemaster 4y ago
  I had success with having it guess the formatting, which was a pretty cool experiment. I had a table in a PDF of registers and associated information, and I copied and pasted the text directly into ChatGPT as a big block, and asked it to structure the data as a table again based on its best understanding of the data, knowing it came from a table. To my surprise, it did a really good job. A couple small edits here and there were needed to change some formatting and it missed a couple values, but overall it took me a couple minutes to edit and I was on my way.
agnosticmantis 4y ago
Unrelated: Is there a stats/ml mastodon instance that’s comparable to Mathstodon? Hopefully populated with technical people rather than marketing and hype-oriented posts. I find the site a pleasure to use compared to other social media. So much less annoyance with pop-ups and banners and engagement traps everywhere, albeit it’s slow to load right now.
dangerwill 4y ago
It is so depressing that we are using chatGPT for data collation. You can never ever know if it is going to silently alter your data unless you check it for correctness afterwards which no one is doing. The tech won't seem so awesome when someone is left off the credits for a talk (in this case). We are adding a new point of failure and no one should believe any curated stats from openAI about GPT4 being X% more accurate for a second.
- sebzim4500 4y ago
  Yes, that's why I get people to do these repetitive tasks instead. Humans are well known to have a 0% failure rate, especially when they are bored out of their minds.
- kurisufag 4y ago
  >You can never ever know if it is going to silently alter your data unless you check it for correctness afterwards which no one is doing. The tech won't seem so awesome when someone is left off the credits for a talk (in this case). terry mentions running some checksums after processing on the linked page. I'd imagine most people working on serious tasks do something similar.
  - dangerwill 4y ago
    https://mathstodon.xyz/@tao/110172819887751038 https://mathstodon.xyz/@tao/110172819887751038 I'm glad that he spot checked but the checksum thing is nonsense. To be able to fully check that the output was correct (and not just the correct raw count of speakers but apportioned properly) then you need to compare the collation of the data as provided by another means (ie manually going through and building a spreadsheet in this case), which would be doing the work that this is claiming to avoid doing.
deleted 4y ago
[deleted]
Chinjut 4y ago
Looking at the actual input/output prompts he shared (https://mathstodon.xyz/@tao/110178324935054737 https://mathstodon.xyz/@tao/110178324935054737), this seems like not very much. These tasks seem really trivial regardless. Entering line breaks that were removed when copy and pasting from a PDF, and looking up the most basic functionality in the spreadsheet software he was already using.
m3kw9 4y ago
A lot of this stuff will be built into OS or software pretty soon in a year. Likely as a subscription because you really can’t run gpt4 quality models locally
yewenjie 4y ago
It is kind of a shame of our academia (society?) that a mathematician of Terence Tao's caliber had to do data entry jobs like this manually before.
mihaic 4y ago
The most shocking thing for me here is that one of the best mathematicians of our times has to do fucking random data to Excel gruntwork. What AI seems to be replacing is mostly bullshit jobs in this case, that should have been handled by someone else.
- acuozzo 4y ago
  This doesn't surprise me. Our species has proven itself to be exceptionally bad at finding the middle ground between giving individuals the freedom to self-direct and allocating resources efficiently at scale. These resource allocation problems exist at the national level, but also within even some of the smallest organizations and certainly everywhere between. Outside of FAANG, how many development teams really have a dedicated DevOps teammate? A DBA? A tester? A sysadmin? Hell, a sysop?! SWEs outside of FAANG are so accustomed to wearing dozens of hats that many would still consider the wisdom in "The Mythical Man-Month" a pipedream.
  - Abroszka 4y ago
    Even in some of the FAANG that's not the case. Not sure about all of them though.
- thaw13579 4y ago
  There's quite a bit of drudgery in academic positions, organizing/running meetings/conferences, editing journals, scrounging for reviewers, grant writing, and navigating all of the red tape of universities.
- dan_mctree 4y ago
  Someone ought to start a charity to hire personal assistants for people like Terence Tao. Or get some government tech grant working on that. That would be probably one of the most efficient investments in progress you can get
amelius 4y ago
Two questions: Does the data fit in the "active memory" of gpt-4, or do they now have a workaround for that? In what form was the input prompt given (or, how does gpt-4 know when the raw data starts and stops)?
tintedfireglass 4y ago
this mastodon instance is not a fan of HN's HOD
squokko 4y ago
What's really the most astonishing thing about all of this is that one of the top 10 brains in the United States is doing this administrative bullshit in the first place. A well run society should find a way to hire someone to do this for him.
- maxdoop 4y ago
  … the point is he doesn’t need to hire someone at all . He can do this task with GPT4 in minutes .
maxdoop 4y ago
I honestly can’t even read HN AI threads anymore. The constant downplaying and denial is unreal. “Why doesn’t a person like him have better data? This isn’t cool for AI; it’s simply a bullshit problem that should’ve never existed.” “How does he know the data was accurate? These things are just hallucinating parrots.” Isn't a normal human just as likely to be inaccurate or make mistakes? The world isn't filled with perfect data in every format. Good grief. Perhaps I get worked up and comment here in hopes that the AI overlords will view me favorably based on my online comments. My real frustration, though, is with the significant number of "usually smart" people who continue to downplay such incredibly cool technology.
- sebzim4500 4y ago
  It's a massive overcorrection against the ten guys on twitter (now mastodon probably) saying that the singularity is two weeks away.
- dang 4y ago
  This looks like a case of the contrarian dynamic: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=%22contrarian%20dynamic%22%20objection%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor... (The parent comment was upvoted to the top of the thread. I've downweighted it now. It's standard moderation practice to do that with generic and/or indignant comments because they tend to attract upvotes and get stuck at the top of threads, making the threads less interesting.)
- evouga 4y ago
  The AI boom has been 6 years of massively-overhyped vaporware followed by 6 months of massively-overhyped services that occasionally deliver the goods. People need some time to recalibrate.
rubenfiszel 4y ago
Am I the only one to wonder why one of the greatest mind of this generation doesn't have an assistant to free his time for some more worthwhile use of it?
- strikelaserclaw 4y ago
  you assume that just because some one is brilliant, they use 100% of their time trying to make discoveries?
BogdanPetre 4y ago
[flagged]
WolfOliver 4y ago
Just use MonsterWriter

207 points by admp 4y ago

wow, the site is so slow. getting "This resource could not be found" errors on reloads. anyway, here's the text: > Today was the first day that I could definitively say that #GPT4 has saved me a significant amount of tedious work. As part of my responsibilities as chair of the ICM Structure Committee, I needed to gather various statistics on the speakers at the previous ICM (for instance, how many speakers there were for each section, taking into account that some speakers were jointly assigned to multiple sections). The raw data (involving about 200 speakers) was not available to me in spreadsheet form, but instead in a number of tables in web pages and PDFs. In the past I would have resigned myself to the tedious taks of first manually entering the data into a spreadsheet and then looking up various spreadsheet functions to work out how to calculate exactly what I needed; but both tasks were easily accomplished in a few minutes by GPT4, and the process was even somewhat enjoyable (with the only tedious aspect being the cut-and-paste between the raw data, GPT4, and the spreadsheet). > Am now looking forward to native integration of AI into the various software tools that I use, so that even the cut-and-paste step can be omitted. (Just being able to resolve >90% of LaTeX compilation issues automatically would be wonderful...) Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? In this case GPT-4 is used to solve a problem that shouldn't have even been one to begin with. The administrators of the ICM could've simply exported the raw data as a Google Sheet (for example) and his problem could've been trivially solved even without GPT-4.

sebzim4500 4y ago
Yeah bold prediction I know but I think basically everyone will move back to twitter by the end of the year
- bantunes 4y ago
  > Yeah bold prediction I know but I think basically everyone will move back to twitter by the end of the year You do know Mastodon is a federation of servers, right? I predict most people will move off this particular one to a faster one.
  - sebzim4500 4y ago
    Sure, but on twitter you don't have to think about which federation you are on
aphit 4y ago
Yeah, I waited a good minute of it trying to load before I gave up. Anyone that got in can share what it says?
- yellow_lead 4y ago
  Check the comment to which you replied (maybe it was added after)
- kzrdude 4y ago
  I accessed the message through a mastodon client instead. They are a lot better at federation than the various instance sites IMO, a more consistent interface when there is no option of sending you out to a non home instance page.
  - SkyMarshal 4y ago
    I didn’t even know there were mastodon clients, though I should have assumed. Know and good ones for Linux and Mac?
    - kzrdude 4y ago
      I use Tusky/Android and that's the limit of my knowledge
red_trumpet 4y ago
HackerNews' hug of death hitting a site with normally 4k active users. Not that surprising...
- endisneigh 4y ago
  it's plain text. it can be cached. it's embarrassing.
  - blowski 4y ago
    Perhaps they've optimised for the normal levels of traffic they receive, not unanticpated spikes. That's hardly an embarrassing choice.
    - endisneigh 4y ago
      I'm not talking about the administrator of this instance, I'm talking about how mastodon is fundamentally designed. Even an iPhone could probably serve the (static) text of Tao's post to a million people on a 5G connection. Computers are fast.
      - blowski 4y ago
        I run Mastodon on a $5 VM. If I hit HN homepage, my site will probably be slow as well. I don’t see how that reflects on Mastodon though.
        endisneigh 4y ago
        what am I saying is that there's a way to design a site on a $5 VM that serves static text that wouldn't be slow even if it was #1 on HN. if you disagree with that then we'll have to agree to disagree.
  - shakow 4y ago
    Feel free to share your know-how https://github.com/mastodon/mastodon https://github.com/mastodon/mastodon
webdood90 4y ago
> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. incredible what mental hoops people will jump through to disqualify AI! the reality is that a lot of things have a terrible UI with unorganized data. that's why this tool is so amazing - because it doesn't matter anymore.
- endisneigh 4y ago
  > the reality is that a lot of things have a terrible UI with unorganized data. that's why this tool is so amazing - because it doesn't matter anymore. how naive. how do you know it's right? ah, you have to manually do the calculation anyway to confirm, this is what Tao ended up saying in a reply asking as much. AI is great, but it's not a silver bullet, since its correctness can never be 100% under the current LLM framework.
  - webdood90 4y ago
    I'm convinced people like you are in for such a rude awakening. I don't understand how you can hold this position with AI considering it's only the beginning.
    - endisneigh 4y ago
      ChatGPT is impressive, but I do not think it is a silver bullet, even for data cleansing and processing tasks. I would use your words and say people who think (that ChatGPT solves the problem of data cleansing once and for all) so are the ones who will be in for a rude awakening.
    - ipaddr 4y ago
      I can't wait for 3d printers to take over manufacturing or self driving cars to self drive. Still waiting for the metaverse. Or cryto to take over banking. We are not even at those points for AI. You might see the future.. let's get a 3d printer in every home first.
      - 4y ago
hn_throwaway_99 4y ago
> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Umm, and I want a pony? The world doesn't come in perfectly structured data formats. I was pretty amazed when I pasted in some of my medical lab test results and ChatGPT was accurately able to parse everything out and summarize my data. It worked extremely well.
- endisneigh 4y ago
  > The world doesn't come in perfectly structured data formats. In Tao's case, the data was organized already, simply not given to him. In any case, Tao, per the post, had to manually calculate to confirm that GPT-4 was correct in any case (he implies as much in a comment asking if he checked for correctness).
  - WastingMyTime89 4y ago
    Checking a table for correctness and filling up a table are not the same thing. I also check for correctness when I manually count entries. It's an extremely error prone task.
    - endisneigh 4y ago
      How exactly could you check for the correctness without effectively finding out the answer? I mean this discrepancy is fundamentally the problem of P = NP or not.
      - wetmore 4y ago
        NP is the class of problems for which it's hard to find the answer but easy to verify. The fact such a class even exists should hint to you that finding the answer and checking correctness can be two different things.
        endisneigh 4y ago
        the point of my comment is that whether P = NP to begin with is still to be determined. in the case of ChatGPT, it's easy for it to respond, but how difficult is it for you to verify that it was correct, or not? how does that compare with the difficulty in doing the task to begin with?
fauxpause_ 4y ago
Solving for every last mile problem in UX is hard though. A lot of AI value is being a flexible, generic interface to software automation.
- SkyMarshal 4y ago
  We can probably even think of AI as a generalized meta-interface to everything - there’s probably no better UI/UX than natural human language that we’ve all been speaking since childhood.
  - fauxpause_ 4y ago
    I don’t know that it’s a better UX, but it is a viable UX, and that is very valuable
rvz 4y ago
> wow, the site is so slow. getting "This resource could not be found" errors on reloads. Oh dear. Federation at its finest /s If the instance is too small, it will easily fall over under heavy traffic. If it is too big, it is highly centralized and even then it cannot scale at a maximum of 300,000+ users at the same time. Eventually, they would all be re-centralizing back to Cloudflare. Once that goes down, all of the biggest Mastodon instances go down at once. As for GPT-4 in mathematics (LLMs in particular), in goes inline on what I said before. Fundamentally, these LLMs cannot reason transparently, nor can it directly bring up its own mathematical proofs with thorough explanations that allows mathematicians to work with. Only low hanging fruit of summarization of existing text. If Mr. Tao can see its limitations, surely it puts the hype of unexplainable magical black-box LLMs to rest as being great sophists and bullshit generators.
- williamstein 4y ago
  I would not discount the potential of LLM's to support research level mathematics. There's interesting research going into combining the LEAN theorem prover with LLMs, e.g., this recent paper: https://arxiv.org/abs/2202.01344 https://arxiv.org/abs/2202.01344 "... automatically solving multiple challenging problems drawn from high school olympiads." This combination of GPT and LEAN is the first thing that has finally got me really interested in the potential of LEAN. And LEAN very directly addresses the "bullshit" issue, since it formally verifies whether or not a proof is correct (it's like a compiler like Rust, but for mathematical proofs).
77pt77 4y ago
https://archive.is/wbwUa https://archive.is/wbwUa
HelloMcFly 4y ago
> In this case GPT-4 is used to solve a problem that shouldn't have even been one to begin with. But the problem DID exist, and problems like that are likely to continue to exist for the foreseeable future.
helen___keller 4y ago
> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? I’m fairly convinced that for many problems (including this), AI is not the best solution but will become the preferred solution. “Best” doesnt get in the way of “good enough” when convenience is at stake (and making a good UX is often very very inconvenient)
- teleforce 4y ago
  I think it's not the case of AI or ChatGPT is the best solution because it can be a local optima. But it's currently the "best" solution available to the average Joe. The fact in this OP case study the average Joe is probably one of the best mathematicians of our time, makes the situation much more interesting and intriguing. I'm in the middle of reading Steve Jobs' very own words, Make Something Wonderful book. Apparently throughout his whole life he frequently mentioned and discussed a lot about providing and creating easy access to the computer because ultimately people do not want to program but want to use computer instead. He did mention about how Morse code was not popular as much of the later telephone technology because people refused to learn and use the unintuitive Morse code even though it only takes 40 hours for average person to learn the entire Morse code. As a trained communication engineers I can very much relate to this because even though we learnt much of the underlying technology of communication that enable the Morse code or the telephone to work across the Atlantic ocean most of us can't be bothered to learn the Morse code and use it if we can help it because the telephone make it redundant and it's counter intuitive to use. Granted, now we know that telephone or circuit switching in general is a suboptimal in providing and solving the human communication problems and needs. As of now the Internet, packet switching and the multimedia approaches are probably the best solutions but telephone has served us a potent and still one of the best solutions for communication until very recently.
poulpy123 4y ago
> Why was the data so disjoint to begin with? Because real life is messy. > Why is Latex so hard to work with? Because people that use latex want to be able to brag they use latex (just joking for the second one.... unless ?)
- endisneigh 4y ago
  > Because real life is messy. This doesn't make sense - the data inherently is centralized. Tao implied as much.
hzay 4y ago
Imagine Terrence Tao manually entering data into spreadsheets!!
SkyMarshal 4y ago
> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? > In this case GPT-4 is used to solve a problem that shouldn't have even been one to begin with. Friction. These problems shouldn’t exist, but they do anyway, and they’re everywhere. Anything human is inevitably going to be imperfect and messy to some greater or lesser degree, introducing friction into dealing with it. Especially as we produce more and more unstructured or semi-structured data, which AI is particularly good at wrangling. If AI can help us cut through that friction significantly faster and more accurately, that’s a win.
- moffkalast 4y ago
  Let's just hope that someone doesn't task the AGI with eliminating friction and it realizes that humans are the problem.
- layer8 4y ago
  Ironically, interacting with ChatGPT still comes with a lot of human-like friction and imperfection.
  - dwringer 4y ago
    IME most of that is of the form "I'm sorry, but as a large language model I..." and seems to have been drilled into it with training to dodge tricky subjects.
Jeff_Brown 4y ago
Better source data is indeed a better solution, but the source data is often not under one's control.
dpcan 4y ago
They should have done it right the first time? That's not the world we live in. Coworkers, associates, clients, family - everyone sends us funky data. Sometimes it's a PDF of text instead of a text file, or a picture of their computer screen with a paragraph in it, or copy-pasted text with unwanted formatting, etc, etc. And the point of this article wasn't really about the tedious job of copy/pasting, it was that once the data was ready, instead of hunkering down and having to work through a problem with an unknown number of variables, GPT-4 can drop an answer in seconds flat. Maybe a few more if you have to tweak your prompt a few times. Part 2 here is that we have a problem to solve, and that's how data is sent and received, and that it all needs to end up in a structural format that's usable by our AI tools. I am thinking that writing a script that can dump textual data from literally anything I'm looking at so it's AI-ready makes a lot of sense, then we have step 1 taken care of as well for my own personal GPT-solvable situations.
PurpleRamen 4y ago
> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Because the world is not a database. His source were formats meant for Human consumption, not machines. This will never change, people are lazy, greedy, or fear "data leaks", so we will never have a machine-first-format that everyone will use. I mean, that guy is using latex, others have markdown, or org-mode, or Microsoft Word, those all are not meant for easy scripting. This is why AI will be such a dealbreaker, because it will close this gap, make human-formats to machine-formats when neccessary.
- rhn_mk1 4y ago
  I find the "human consumption" here funny, because it's the "human" part which led the author to turn to machines which can do it better.
  - pegasus 4y ago
    Well, yes, this is kinda the whole point of the argument. That's what LLMs bring to the table - machines that can better deal with data and documents meant for human consumption.
belugacat 4y ago
And here’s an exchange about how he validated the data was correct - which to me seems like the most important part: > Out of curiosity, are you sure that GPT did it correctly? If yes, is it because you were able to "spot check" it in a few places? Or have you used GPT enough that you trust it with a task like this? Or is this for some kind of internal use where a few small errors is unimportant, and you only need the broad strokes to be correct? > Yes, yes, and yes. (There were some obvious checksums, for instance by computing the total number of speakers several different ways.) Still cool, but the circumstances above aren’t always true. In fact, in a lot of meaningful work, none of them probably are. If GPT is like an assistant whose work you always have to double check in order to be sure of the validity of the results, that seems to be a pretty big caveat to me. In that case it’s probably better to just write a script whose output you know to be deterministic, even if it takes slightly longer (and the bigger your dataset to verify, the more likely it is that it’ll take less time to write a script than to validate results one by one). The scary part is if/when companies/bureaucracies/governments/etc start using GPT for all sorts of Important Tasks and skip the validation part because they assume the machine will always get it right.
- humanistbot 4y ago
  > If GPT is like an assistant whose work you always have to double check in order to be sure of the validity of the results, that seems to be a pretty big caveat to me. Sounds like every junior developer fresh out of college I've had to work with. Interns are even worse. Even senior devs mess up from time to time. That's why we do code review and other QA. Trust no one.
  - belugacat 4y ago
    True, but one obvious difference is that a senior contributor can typically validate the work of a more junior one in less time than it took them to produce, that’s why the overall equation works. And if you decide to keep the more junior person, it’s probably because you feel like they’re improving and becoming more trustworthy over time. Another key difference is that a deterministic script that has determined to be valid will be valid for all subsequent runs, regardless of input size. That is not the case for ChatGPT, which might be correct on some runs and not others, or do well on small input sizes but start messing up when thousands (or more) of data items are involved. Again, I still think ChatGPT is really cool, but thinking about all these nuances about the nature of the work that is actually being done when you use ChatGPT vs writing a script vs handing it off to an intern seems crucial to the debate.
6gvONxR4sf7o 4y ago
> AI, though amazing, isn't really the solution. Better UX and data quality is. In this case, AI was the tool used to have a better UX. People build something with use-case X in mind, and use the best tool for that job. Piping that into use-case Y requires some duct tape and plumbing. It turns out that AI is great at that sort of repurposing. It's great if someone using the best tool for their job is nearly the best tool for the flexible infinity of other use cases, which is what flexible enough duct-tape gets us.
sdfghswe 4y ago
Wait.... is it possible to tell chat gpt to grab data from a page?
ugh123 4y ago
>Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? Do you mean build better, specific, UX and normalize data for only this task? Why spend that time, even as a developer of whatever improved app/system you're thinking of, when he can just turn to AI? I'd rather lean on GPT4's ability to generalize highly specific technical work rather than ruminate on each individual app's lackings and how it could be better than using AI if we just... put more effort into it?
gvhst 4y ago
And AI is a way to enable better UX and cleaner data...
jacquesm 4y ago
And how does he know the results are correct without verifying each and every entry?
- tjr 4y ago
  I find this a real concern when it comes to the proposition of "replace every programmer with GPT". I think the reality is, there are cases where it matters, and cases where it doesn't matter (at least, not very much). I work on aerospace software systems. I suspect that most software developers would be surprised at the process and rigor involved. That does not mean that humans are flawless; of course not. But there are going to be a lot more hurdles to replacing all of the humans with non-deterministic machines. There are numerous software components on a modern airplane. How good is good enough? If the overall system works correctly 99% of the time, is that good enough? There are about 16 million flights annually in the United States alone. If 160,000 of them crash, is that good enough? I mean, like, if Microsoft Word failed to save your document 99% of the time, you might be irked. If Professor Tao's data was only 99% accurate, he probably still has a reasonably good picture of the information he needs. Regardless of if a human or a machine is writing the code, there are different levels of software criticality. As humans, we don't apply avionics software development process to iOS games because it's not worth the extra time and expense. Likewise, there will be a spectrum of where it makes good enough sense to automate tasks with AI and where it doesn't. Which is not to dismiss AI! It's amazingly useful technology. But there really are places where above average accuracy is needed, and AI that works great for handling some tasks might not suffice for other tasks. If you truly need all of the results correct, then a nondeterministic neural network is probably not the best path to be on. There are other methods; even other automated methods.
  - jacquesm 4y ago
    > I work on aerospace software systems. I suspect that most software developers would be surprised at the process and rigor involved. I did this once, a fuel estimation program for 747's, the degree of understanding of the problem required to create something that would pass review was off the scale compared to any other program that I've ever written. It is also the only time that the customer wanted to get it right rather than that they were looking at what it cost (because the savings would pay for the development many times over this was probably less of an issue anyway). I'd love to be able to always work like that.
csomar 4y ago
> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? AI is just that. A UI/UX for any/your data.

fifilura 4y ago

https://web.archive.org/web/20230412131029/https://mathstodon.xyz/@tao/110172426733603359 https://web.archive.org/web/20230412131029/https://mathstodo...

Rescis 4y ago

Archive Link: https://web.archive.org/web/20230412131029/https://mathstodon.xyz/@tao/110172426733603359 https://web.archive.org/web/20230412131029/https://mathstodo...

deleted 4y ago

[deleted]

jacknews 4y ago

I mean, one of the leading mathematicians at the moment, is expected to spent his time collating conference statistics? Does he not have an assistant for this kind of thing?

jostmey 4y ago
It’s academia. Only admins get secretaries. I’m only joking a little bit, as I once was a professor
- robertlagrant 4y ago
  I was going to make the same joke! If he were an Assistant Vice Under-Dean he'd have 10 assistants, but sadly he's just an academic.
kevmo314 4y ago
Seems like his assistant is GPT-4 :)
deleted 4y ago
[deleted]
NhanH 4y ago
Terence Tao makes about only half a million or so every year (his salary at UCLA is public). Not sure about other income, but it is pretty much just the same ballpark as good doctors, lawyers, accountants at big 4 etc. At that range of income, it’s job-dependent on whether they would have an assistant or not, so no surprise here either way.
- jacknews 4y ago
  So you think 'good accountant' and 'world class mathematician' are equivalent, based on their salary?
  - NhanH 4y ago
    No I don’t, but they have to pay for assistant all the same. I was describing how things are, not how they should be. University could have paid for Terence Tao’s assistant, but then they could have paid him more too.

amadvance 4y ago

Terence Tao doing a data entry job ? The Peter Principle strikes again! https://en.wikipedia.org/wiki/Peter_principle https://en.wikipedia.org/wiki/Peter_principle

archgoon 4y ago
[dead]
nequo 4y ago
Wouldn’t the Peter principle in this situation be more like a very good data enterer getting the job of Terence Tao?
- squokko 4y ago
  LOL, this is funny. I'm imagining some bushy tailed assistant finishing all the data entry and asking for another task and being told to prove the Collatz Conjecture.

sampo 4y ago

Site took 37 seconds to load.

m_mueller 4y ago

how do you upload PDFs to GPT-4, is it available through the ChatGPT interface?

commandlinefan 4y ago

I used GPT-3 yesterday for the first time myself - I wanted some code to list the DynamoDB tables available to me (I had to run it from an EC2 instance and I wanted to see what that particular EC2 instance could see). I was pretty impressed - it put the code together so I asked it for a POM and it put that together. The POM failed with an error because it was trying to load from Maven central using HTTP, so I asked ChatGPT how to fix that error and it told me how to update my settings.xml file (it did make a mistake, but it was an easy one to fix). Like - I knew how to do all of that, but it probably would have taken a half hour vs about 5 minutes with ChatGPT.

RugnirViking 4y ago
exactly. It makes mistakes, but so do I. It makes mistakes far faster however, because it takes about 5 seconds to finish each attempt, so they seem far more prevalent. However, I can fix it's mistakes almost as fast as I can fix my own, so this speeds up iteration nicely. It's not a "perfect tool", but nothing is
elboru 4y ago
ChatGPT is great with well known tools and libraries. I’ve been able to build HTML/JS apps in minutes. Then I tried building a MAUI app, it started pretty good, but then when I tried to add charts it failed (it’s important to mention this was my first MAUI app), I tried different approaches, in the end I went back to HTML/JS. That led me to a realization, this could mean that people will likely discard less known libraries or technologies more easily than before. Popular technologies will become even more popular since we’ll be 10x more productive with them compared to less known ones since we won’t have LLMs super powers for those.
- burlesona 4y ago
  I agree and have had the same thought. This is gas on the fire to winner-take-all dynamics. It’s already hard for new languages and systems to catch up fast enough to technically weaker existing options bc they have mature tooling. But when the robot is fluent in Java and can’t understand $newlang it’s going to be even harder.

throwsalj2rttw 4y ago

I'd really like to see sessions of people using things like ChatGPT for non-trivial things. People keep saying it saves them time and look forward to the future. For the life of me I can't figure out how to use it.

webdood90 4y ago
complicated tasks are just a series of trivial tasks, wouldn't you agree?
- bumby 4y ago
  It depends on the context. A complicated spider web is just a series of connected strands, but pulling on one impacts all the others to various degrees. The point being, a contextual understanding of the systemic effects becomes important when deciding what to do on each "trivial" task. To the OPs point, I'm not sure it's been convincingly shown that ChatGPT has a strong contextual understanding. (in fact, that's also a major shortcoming of humans when we over simply complex models)
- kgwgk 4y ago
  Heart surgery is just a series of simple cuts and stitches!
laichzeit0 4y ago
Depends on what you do. I had some Pandas code that I wanted to be slightly faster. I pasted it in and asked it to optimize the code. It did exactly that, with an explanation at the bottom of why it was doing each particular thing. It was correct, the code was faster. I’ve used it for a bunch of things like that.
- Jeff_Brown 4y ago
  How much text was the code you fed it?
golol 4y ago
I used GPT-4 to write a simple chat website so I can talk to it without having to do manual API calls. It created a working version on the first try and me telling it about a few errors. In an hour I had improved it to a really polished look. I recently used GPT-4 for matplotlib. I wrote a imple PDE solver and wanted it to create a function that saves a simple 3d array as an animatied 2d plot. It did it right away. I could ask it for improvements and it did it too. Both of these tasks are easy, and if you are working every day in web development or with matplotlib I am sure you can do them in 5 minutes. But in my case, each of them might have taken half a day. And even if I could do it in 1 hour, that would be 1 hour of furrowed-brows staring at stackoverflow. Using GPT-4 is just extremely easy. From my experience I claim that GPT-4 can also solve more complex problems. I think if I iteratively ask for features on top of what it has given me, I can get up to 3-4 times the amount of features before the whole code becomes too complex to handle. This is just a guess.

pgt 4y ago

How are folks parsing PDFs with GPT4? Every 3rd party tool that supposedly uses GPT has failed for me on even the most regular of PDF bank statements. Currently I am parsing the text contents manually using pdfbox to reconstruct historical bank statements, but this is error-prone due to page breaks and run-on lines.

zachwill 4y ago
Note: I haven’t worked with PDF bank statements. My current solution is pdfplumber → GPT-3 API. I played around with a few different options, and this is personally what’s worked best for my use cases.
MilStdJunkie 4y ago
Same deal for me, I've tried a few AI PDF pipelines with no good output. If I feed it cleaned up delimited text then everything's good[1], but by the time I get to that point I might as well be using Tabula + Orange / R. I'll keep trying though, because damn, PDF files need to go die under a rock. [1] Well . . kinda. The model has some weird ideas about what an edge table is.
Jeff_Brown 4y ago
I'm guessing that manually feeding it a table at a time, with prompts tailored to the shapes you see in that table, is more effective than using a tool that tries to do that for you. Automated tools on top of GPT will be great when they work but they introduce a new communication joint where things can fail.
ragazzina 4y ago
I think he just copy-pasted everything from the PDF to GPT4? He says: > with the only tedious aspect being the cut-and-paste between the raw data, GPT4, and the spreadsheet
Ductapemaster 4y ago
I had success with having it guess the formatting, which was a pretty cool experiment. I had a table in a PDF of registers and associated information, and I copied and pasted the text directly into ChatGPT as a big block, and asked it to structure the data as a table again based on its best understanding of the data, knowing it came from a table. To my surprise, it did a really good job. A couple small edits here and there were needed to change some formatting and it missed a couple values, but overall it took me a couple minutes to edit and I was on my way.

agnosticmantis 4y ago

Unrelated: Is there a stats/ml mastodon instance that’s comparable to Mathstodon? Hopefully populated with technical people rather than marketing and hype-oriented posts. I find the site a pleasure to use compared to other social media. So much less annoyance with pop-ups and banners and engagement traps everywhere, albeit it’s slow to load right now.

dangerwill 4y ago

It is so depressing that we are using chatGPT for data collation. You can never ever know if it is going to silently alter your data unless you check it for correctness afterwards which no one is doing. The tech won't seem so awesome when someone is left off the credits for a talk (in this case). We are adding a new point of failure and no one should believe any curated stats from openAI about GPT4 being X% more accurate for a second.

sebzim4500 4y ago
Yes, that's why I get people to do these repetitive tasks instead. Humans are well known to have a 0% failure rate, especially when they are bored out of their minds.
kurisufag 4y ago
>You can never ever know if it is going to silently alter your data unless you check it for correctness afterwards which no one is doing. The tech won't seem so awesome when someone is left off the credits for a talk (in this case). terry mentions running some checksums after processing on the linked page. I'd imagine most people working on serious tasks do something similar.
- dangerwill 4y ago
  https://mathstodon.xyz/@tao/110172819887751038 https://mathstodon.xyz/@tao/110172819887751038 I'm glad that he spot checked but the checksum thing is nonsense. To be able to fully check that the output was correct (and not just the correct raw count of speakers but apportioned properly) then you need to compare the collation of the data as provided by another means (ie manually going through and building a spreadsheet in this case), which would be doing the work that this is claiming to avoid doing.

deleted 4y ago

[deleted]

Chinjut 4y ago

Looking at the actual input/output prompts he shared (https://mathstodon.xyz/@tao/110178324935054737 https://mathstodon.xyz/@tao/110178324935054737), this seems like not very much. These tasks seem really trivial regardless. Entering line breaks that were removed when copy and pasting from a PDF, and looking up the most basic functionality in the spreadsheet software he was already using.

m3kw9 4y ago

A lot of this stuff will be built into OS or software pretty soon in a year. Likely as a subscription because you really can’t run gpt4 quality models locally

yewenjie 4y ago

It is kind of a shame of our academia (society?) that a mathematician of Terence Tao's caliber had to do data entry jobs like this manually before.

mihaic 4y ago

The most shocking thing for me here is that one of the best mathematicians of our times has to do fucking random data to Excel gruntwork. What AI seems to be replacing is mostly bullshit jobs in this case, that should have been handled by someone else.

acuozzo 4y ago
This doesn't surprise me. Our species has proven itself to be exceptionally bad at finding the middle ground between giving individuals the freedom to self-direct and allocating resources efficiently at scale. These resource allocation problems exist at the national level, but also within even some of the smallest organizations and certainly everywhere between. Outside of FAANG, how many development teams really have a dedicated DevOps teammate? A DBA? A tester? A sysadmin? Hell, a sysop?! SWEs outside of FAANG are so accustomed to wearing dozens of hats that many would still consider the wisdom in "The Mythical Man-Month" a pipedream.
- Abroszka 4y ago
  Even in some of the FAANG that's not the case. Not sure about all of them though.
thaw13579 4y ago
There's quite a bit of drudgery in academic positions, organizing/running meetings/conferences, editing journals, scrounging for reviewers, grant writing, and navigating all of the red tape of universities.
dan_mctree 4y ago
Someone ought to start a charity to hire personal assistants for people like Terence Tao. Or get some government tech grant working on that. That would be probably one of the most efficient investments in progress you can get

amelius 4y ago

Two questions: Does the data fit in the "active memory" of gpt-4, or do they now have a workaround for that? In what form was the input prompt given (or, how does gpt-4 know when the raw data starts and stops)?

tintedfireglass 4y ago

this mastodon instance is not a fan of HN's HOD

squokko 4y ago

What's really the most astonishing thing about all of this is that one of the top 10 brains in the United States is doing this administrative bullshit in the first place. A well run society should find a way to hire someone to do this for him.

maxdoop 4y ago
… the point is he doesn’t need to hire someone at all . He can do this task with GPT4 in minutes .

maxdoop 4y ago

I honestly can’t even read HN AI threads anymore. The constant downplaying and denial is unreal. “Why doesn’t a person like him have better data? This isn’t cool for AI; it’s simply a bullshit problem that should’ve never existed.” “How does he know the data was accurate? These things are just hallucinating parrots.” Isn't a normal human just as likely to be inaccurate or make mistakes? The world isn't filled with perfect data in every format. Good grief. Perhaps I get worked up and comment here in hopes that the AI overlords will view me favorably based on my online comments. My real frustration, though, is with the significant number of "usually smart" people who continue to downplay such incredibly cool technology.

sebzim4500 4y ago
It's a massive overcorrection against the ten guys on twitter (now mastodon probably) saying that the singularity is two weeks away.
dang 4y ago
This looks like a case of the contrarian dynamic: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=%22contrarian%20dynamic%22%20objection%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor... (The parent comment was upvoted to the top of the thread. I've downweighted it now. It's standard moderation practice to do that with generic and/or indignant comments because they tend to attract upvotes and get stuck at the top of threads, making the threads less interesting.)
evouga 4y ago
The AI boom has been 6 years of massively-overhyped vaporware followed by 6 months of massively-overhyped services that occasionally deliver the goods. People need some time to recalibrate.

rubenfiszel 4y ago

Am I the only one to wonder why one of the greatest mind of this generation doesn't have an assistant to free his time for some more worthwhile use of it?

strikelaserclaw 4y ago
you assume that just because some one is brilliant, they use 100% of their time trying to make discoveries?

BogdanPetre 4y ago

[flagged]

WolfOliver 4y ago

Just use MonsterWriter