9 ms·

wow, the site is so slow. getting "This resource could not be found" errors on reloads. anyway, here's the text: > Today was the first day that I could defini

wow, the site is so slow. getting "This resource could not be found" errors on reloads. anyway, here's the text: > Today was the first day that I could definitively say that #GPT4 has saved me a significant amount of tedious work. As part of my responsibilities as chair of the ICM Structure Committee, I needed to gather various statistics on the speakers at the previous ICM (for instance, how many speakers there were for each section, taking into account that some speakers were jointly assigned to multiple sections). The raw data (involving about 200 speakers) was not available to me in spreadsheet form, but instead in a number of tables in web pages and PDFs. In the past I would have resigned myself to the tedious taks of first manually entering the data into a spreadsheet and then looking up various spreadsheet functions to work out how to calculate exactly what I needed; but both tasks were easily accomplished in a few minutes by GPT4, and the process was even somewhat enjoyable (with the only tedious aspect being the cut-and-paste between the raw data, GPT4, and the spreadsheet). > Am now looking forward to native integration of AI into the various software tools that I use, so that even the cut-and-paste step can be omitted. (Just being able to resolve >90% of LaTeX compilation issues automatically would be wonderful...) Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? In this case GPT-4 is used to solve a problem that shouldn't have even been one to begin with. The administrators of the ICM could've simply exported the raw data as a Google Sheet (for example) and his problem could've been trivially solved even without GPT-4.

sebzim4500 4y ago
Yeah bold prediction I know but I think basically everyone will move back to twitter by the end of the year
- bantunes 4y ago
  > Yeah bold prediction I know but I think basically everyone will move back to twitter by the end of the year You do know Mastodon is a federation of servers, right? I predict most people will move off this particular one to a faster one.
  - sebzim4500 4y ago
    Sure, but on twitter you don't have to think about which federation you are on
aphit 4y ago
Yeah, I waited a good minute of it trying to load before I gave up. Anyone that got in can share what it says?
- yellow_lead 4y ago
  Check the comment to which you replied (maybe it was added after)
- kzrdude 4y ago
  I accessed the message through a mastodon client instead. They are a lot better at federation than the various instance sites IMO, a more consistent interface when there is no option of sending you out to a non home instance page.
  - SkyMarshal 4y ago
    I didn’t even know there were mastodon clients, though I should have assumed. Know and good ones for Linux and Mac?
    - kzrdude 4y ago
      I use Tusky/Android and that's the limit of my knowledge
red_trumpet 4y ago
HackerNews' hug of death hitting a site with normally 4k active users. Not that surprising...
- endisneigh 4y ago
  it's plain text. it can be cached. it's embarrassing.
  - blowski 4y ago
    Perhaps they've optimised for the normal levels of traffic they receive, not unanticpated spikes. That's hardly an embarrassing choice.
    - endisneigh 4y ago
      I'm not talking about the administrator of this instance, I'm talking about how mastodon is fundamentally designed. Even an iPhone could probably serve the (static) text of Tao's post to a million people on a 5G connection. Computers are fast.
      - blowski 4y ago
        I run Mastodon on a $5 VM. If I hit HN homepage, my site will probably be slow as well. I don’t see how that reflects on Mastodon though.
        endisneigh 4y ago
        what am I saying is that there's a way to design a site on a $5 VM that serves static text that wouldn't be slow even if it was #1 on HN. if you disagree with that then we'll have to agree to disagree.
  - shakow 4y ago
    Feel free to share your know-how https://github.com/mastodon/mastodon https://github.com/mastodon/mastodon
webdood90 4y ago
> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. incredible what mental hoops people will jump through to disqualify AI! the reality is that a lot of things have a terrible UI with unorganized data. that's why this tool is so amazing - because it doesn't matter anymore.
- endisneigh 4y ago
  > the reality is that a lot of things have a terrible UI with unorganized data. that's why this tool is so amazing - because it doesn't matter anymore. how naive. how do you know it's right? ah, you have to manually do the calculation anyway to confirm, this is what Tao ended up saying in a reply asking as much. AI is great, but it's not a silver bullet, since its correctness can never be 100% under the current LLM framework.
  - webdood90 4y ago
    I'm convinced people like you are in for such a rude awakening. I don't understand how you can hold this position with AI considering it's only the beginning.
    - endisneigh 4y ago
      ChatGPT is impressive, but I do not think it is a silver bullet, even for data cleansing and processing tasks. I would use your words and say people who think (that ChatGPT solves the problem of data cleansing once and for all) so are the ones who will be in for a rude awakening.
    - ipaddr 4y ago
      I can't wait for 3d printers to take over manufacturing or self driving cars to self drive. Still waiting for the metaverse. Or cryto to take over banking. We are not even at those points for AI. You might see the future.. let's get a 3d printer in every home first.
      - 4y ago
hn_throwaway_99 4y ago
> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Umm, and I want a pony? The world doesn't come in perfectly structured data formats. I was pretty amazed when I pasted in some of my medical lab test results and ChatGPT was accurately able to parse everything out and summarize my data. It worked extremely well.
- endisneigh 4y ago
  > The world doesn't come in perfectly structured data formats. In Tao's case, the data was organized already, simply not given to him. In any case, Tao, per the post, had to manually calculate to confirm that GPT-4 was correct in any case (he implies as much in a comment asking if he checked for correctness).
  - WastingMyTime89 4y ago
    Checking a table for correctness and filling up a table are not the same thing. I also check for correctness when I manually count entries. It's an extremely error prone task.
    - endisneigh 4y ago
      How exactly could you check for the correctness without effectively finding out the answer? I mean this discrepancy is fundamentally the problem of P = NP or not.
      - wetmore 4y ago
        NP is the class of problems for which it's hard to find the answer but easy to verify. The fact such a class even exists should hint to you that finding the answer and checking correctness can be two different things.
        endisneigh 4y ago
        the point of my comment is that whether P = NP to begin with is still to be determined. in the case of ChatGPT, it's easy for it to respond, but how difficult is it for you to verify that it was correct, or not? how does that compare with the difficulty in doing the task to begin with?
fauxpause_ 4y ago
Solving for every last mile problem in UX is hard though. A lot of AI value is being a flexible, generic interface to software automation.
- SkyMarshal 4y ago
  We can probably even think of AI as a generalized meta-interface to everything - there’s probably no better UI/UX than natural human language that we’ve all been speaking since childhood.
  - fauxpause_ 4y ago
    I don’t know that it’s a better UX, but it is a viable UX, and that is very valuable
rvz 4y ago
> wow, the site is so slow. getting "This resource could not be found" errors on reloads. Oh dear. Federation at its finest /s If the instance is too small, it will easily fall over under heavy traffic. If it is too big, it is highly centralized and even then it cannot scale at a maximum of 300,000+ users at the same time. Eventually, they would all be re-centralizing back to Cloudflare. Once that goes down, all of the biggest Mastodon instances go down at once. As for GPT-4 in mathematics (LLMs in particular), in goes inline on what I said before. Fundamentally, these LLMs cannot reason transparently, nor can it directly bring up its own mathematical proofs with thorough explanations that allows mathematicians to work with. Only low hanging fruit of summarization of existing text. If Mr. Tao can see its limitations, surely it puts the hype of unexplainable magical black-box LLMs to rest as being great sophists and bullshit generators.
- williamstein 4y ago
  I would not discount the potential of LLM's to support research level mathematics. There's interesting research going into combining the LEAN theorem prover with LLMs, e.g., this recent paper: https://arxiv.org/abs/2202.01344 https://arxiv.org/abs/2202.01344 "... automatically solving multiple challenging problems drawn from high school olympiads." This combination of GPT and LEAN is the first thing that has finally got me really interested in the potential of LEAN. And LEAN very directly addresses the "bullshit" issue, since it formally verifies whether or not a proof is correct (it's like a compiler like Rust, but for mathematical proofs).
77pt77 4y ago
https://archive.is/wbwUa https://archive.is/wbwUa
HelloMcFly 4y ago
> In this case GPT-4 is used to solve a problem that shouldn't have even been one to begin with. But the problem DID exist, and problems like that are likely to continue to exist for the foreseeable future.
helen___keller 4y ago
> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? I’m fairly convinced that for many problems (including this), AI is not the best solution but will become the preferred solution. “Best” doesnt get in the way of “good enough” when convenience is at stake (and making a good UX is often very very inconvenient)
- teleforce 4y ago
  I think it's not the case of AI or ChatGPT is the best solution because it can be a local optima. But it's currently the "best" solution available to the average Joe. The fact in this OP case study the average Joe is probably one of the best mathematicians of our time, makes the situation much more interesting and intriguing. I'm in the middle of reading Steve Jobs' very own words, Make Something Wonderful book. Apparently throughout his whole life he frequently mentioned and discussed a lot about providing and creating easy access to the computer because ultimately people do not want to program but want to use computer instead. He did mention about how Morse code was not popular as much of the later telephone technology because people refused to learn and use the unintuitive Morse code even though it only takes 40 hours for average person to learn the entire Morse code. As a trained communication engineers I can very much relate to this because even though we learnt much of the underlying technology of communication that enable the Morse code or the telephone to work across the Atlantic ocean most of us can't be bothered to learn the Morse code and use it if we can help it because the telephone make it redundant and it's counter intuitive to use. Granted, now we know that telephone or circuit switching in general is a suboptimal in providing and solving the human communication problems and needs. As of now the Internet, packet switching and the multimedia approaches are probably the best solutions but telephone has served us a potent and still one of the best solutions for communication until very recently.
poulpy123 4y ago
> Why was the data so disjoint to begin with? Because real life is messy. > Why is Latex so hard to work with? Because people that use latex want to be able to brag they use latex (just joking for the second one.... unless ?)
- endisneigh 4y ago
  > Because real life is messy. This doesn't make sense - the data inherently is centralized. Tao implied as much.
hzay 4y ago
Imagine Terrence Tao manually entering data into spreadsheets!!
SkyMarshal 4y ago
> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? > In this case GPT-4 is used to solve a problem that shouldn't have even been one to begin with. Friction. These problems shouldn’t exist, but they do anyway, and they’re everywhere. Anything human is inevitably going to be imperfect and messy to some greater or lesser degree, introducing friction into dealing with it. Especially as we produce more and more unstructured or semi-structured data, which AI is particularly good at wrangling. If AI can help us cut through that friction significantly faster and more accurately, that’s a win.
- moffkalast 4y ago
  Let's just hope that someone doesn't task the AGI with eliminating friction and it realizes that humans are the problem.
- layer8 4y ago
  Ironically, interacting with ChatGPT still comes with a lot of human-like friction and imperfection.
  - dwringer 4y ago
    IME most of that is of the form "I'm sorry, but as a large language model I..." and seems to have been drilled into it with training to dodge tricky subjects.
Jeff_Brown 4y ago
Better source data is indeed a better solution, but the source data is often not under one's control.
dpcan 4y ago
They should have done it right the first time? That's not the world we live in. Coworkers, associates, clients, family - everyone sends us funky data. Sometimes it's a PDF of text instead of a text file, or a picture of their computer screen with a paragraph in it, or copy-pasted text with unwanted formatting, etc, etc. And the point of this article wasn't really about the tedious job of copy/pasting, it was that once the data was ready, instead of hunkering down and having to work through a problem with an unknown number of variables, GPT-4 can drop an answer in seconds flat. Maybe a few more if you have to tweak your prompt a few times. Part 2 here is that we have a problem to solve, and that's how data is sent and received, and that it all needs to end up in a structural format that's usable by our AI tools. I am thinking that writing a script that can dump textual data from literally anything I'm looking at so it's AI-ready makes a lot of sense, then we have step 1 taken care of as well for my own personal GPT-solvable situations.
PurpleRamen 4y ago
> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Because the world is not a database. His source were formats meant for Human consumption, not machines. This will never change, people are lazy, greedy, or fear "data leaks", so we will never have a machine-first-format that everyone will use. I mean, that guy is using latex, others have markdown, or org-mode, or Microsoft Word, those all are not meant for easy scripting. This is why AI will be such a dealbreaker, because it will close this gap, make human-formats to machine-formats when neccessary.
- rhn_mk1 4y ago
  I find the "human consumption" here funny, because it's the "human" part which led the author to turn to machines which can do it better.
  - pegasus 4y ago
    Well, yes, this is kinda the whole point of the argument. That's what LLMs bring to the table - machines that can better deal with data and documents meant for human consumption.
belugacat 4y ago
And here’s an exchange about how he validated the data was correct - which to me seems like the most important part: > Out of curiosity, are you sure that GPT did it correctly? If yes, is it because you were able to "spot check" it in a few places? Or have you used GPT enough that you trust it with a task like this? Or is this for some kind of internal use where a few small errors is unimportant, and you only need the broad strokes to be correct? > Yes, yes, and yes. (There were some obvious checksums, for instance by computing the total number of speakers several different ways.) Still cool, but the circumstances above aren’t always true. In fact, in a lot of meaningful work, none of them probably are. If GPT is like an assistant whose work you always have to double check in order to be sure of the validity of the results, that seems to be a pretty big caveat to me. In that case it’s probably better to just write a script whose output you know to be deterministic, even if it takes slightly longer (and the bigger your dataset to verify, the more likely it is that it’ll take less time to write a script than to validate results one by one). The scary part is if/when companies/bureaucracies/governments/etc start using GPT for all sorts of Important Tasks and skip the validation part because they assume the machine will always get it right.
- humanistbot 4y ago
  > If GPT is like an assistant whose work you always have to double check in order to be sure of the validity of the results, that seems to be a pretty big caveat to me. Sounds like every junior developer fresh out of college I've had to work with. Interns are even worse. Even senior devs mess up from time to time. That's why we do code review and other QA. Trust no one.
  - belugacat 4y ago
    True, but one obvious difference is that a senior contributor can typically validate the work of a more junior one in less time than it took them to produce, that’s why the overall equation works. And if you decide to keep the more junior person, it’s probably because you feel like they’re improving and becoming more trustworthy over time. Another key difference is that a deterministic script that has determined to be valid will be valid for all subsequent runs, regardless of input size. That is not the case for ChatGPT, which might be correct on some runs and not others, or do well on small input sizes but start messing up when thousands (or more) of data items are involved. Again, I still think ChatGPT is really cool, but thinking about all these nuances about the nature of the work that is actually being done when you use ChatGPT vs writing a script vs handing it off to an intern seems crucial to the debate.
6gvONxR4sf7o 4y ago
> AI, though amazing, isn't really the solution. Better UX and data quality is. In this case, AI was the tool used to have a better UX. People build something with use-case X in mind, and use the best tool for that job. Piping that into use-case Y requires some duct tape and plumbing. It turns out that AI is great at that sort of repurposing. It's great if someone using the best tool for their job is nearly the best tool for the flexible infinity of other use cases, which is what flexible enough duct-tape gets us.
sdfghswe 4y ago
Wait.... is it possible to tell chat gpt to grab data from a page?
ugh123 4y ago
>Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? Do you mean build better, specific, UX and normalize data for only this task? Why spend that time, even as a developer of whatever improved app/system you're thinking of, when he can just turn to AI? I'd rather lean on GPT4's ability to generalize highly specific technical work rather than ruminate on each individual app's lackings and how it could be better than using AI if we just... put more effort into it?
gvhst 4y ago
And AI is a way to enable better UX and cleaner data...
jacquesm 4y ago
And how does he know the results are correct without verifying each and every entry?
- tjr 4y ago
  I find this a real concern when it comes to the proposition of "replace every programmer with GPT". I think the reality is, there are cases where it matters, and cases where it doesn't matter (at least, not very much). I work on aerospace software systems. I suspect that most software developers would be surprised at the process and rigor involved. That does not mean that humans are flawless; of course not. But there are going to be a lot more hurdles to replacing all of the humans with non-deterministic machines. There are numerous software components on a modern airplane. How good is good enough? If the overall system works correctly 99% of the time, is that good enough? There are about 16 million flights annually in the United States alone. If 160,000 of them crash, is that good enough? I mean, like, if Microsoft Word failed to save your document 99% of the time, you might be irked. If Professor Tao's data was only 99% accurate, he probably still has a reasonably good picture of the information he needs. Regardless of if a human or a machine is writing the code, there are different levels of software criticality. As humans, we don't apply avionics software development process to iOS games because it's not worth the extra time and expense. Likewise, there will be a spectrum of where it makes good enough sense to automate tasks with AI and where it doesn't. Which is not to dismiss AI! It's amazingly useful technology. But there really are places where above average accuracy is needed, and AI that works great for handling some tasks might not suffice for other tasks. If you truly need all of the results correct, then a nondeterministic neural network is probably not the best path to be on. There are other methods; even other automated methods.
  - jacquesm 4y ago
    > I work on aerospace software systems. I suspect that most software developers would be surprised at the process and rigor involved. I did this once, a fuel estimation program for 747's, the degree of understanding of the problem required to create something that would pass review was off the scale compared to any other program that I've ever written. It is also the only time that the customer wanted to get it right rather than that they were looking at what it cost (because the savings would pay for the development many times over this was probably less of an issue anyway). I'd love to be able to always work like that.
csomar 4y ago
> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? AI is just that. A UI/UX for any/your data.

sebzim4500 4y ago

Yeah bold prediction I know but I think basically everyone will move back to twitter by the end of the year

bantunes 4y ago
> Yeah bold prediction I know but I think basically everyone will move back to twitter by the end of the year You do know Mastodon is a federation of servers, right? I predict most people will move off this particular one to a faster one.
- sebzim4500 4y ago
  Sure, but on twitter you don't have to think about which federation you are on

aphit 4y ago

Yeah, I waited a good minute of it trying to load before I gave up. Anyone that got in can share what it says?

yellow_lead 4y ago
Check the comment to which you replied (maybe it was added after)
kzrdude 4y ago
I accessed the message through a mastodon client instead. They are a lot better at federation than the various instance sites IMO, a more consistent interface when there is no option of sending you out to a non home instance page.
- SkyMarshal 4y ago
  I didn’t even know there were mastodon clients, though I should have assumed. Know and good ones for Linux and Mac?
  - kzrdude 4y ago
    I use Tusky/Android and that's the limit of my knowledge

red_trumpet 4y ago

HackerNews' hug of death hitting a site with normally 4k active users. Not that surprising...

endisneigh 4y ago
it's plain text. it can be cached. it's embarrassing.
- blowski 4y ago
  Perhaps they've optimised for the normal levels of traffic they receive, not unanticpated spikes. That's hardly an embarrassing choice.
  - endisneigh 4y ago
    I'm not talking about the administrator of this instance, I'm talking about how mastodon is fundamentally designed. Even an iPhone could probably serve the (static) text of Tao's post to a million people on a 5G connection. Computers are fast.
    - blowski 4y ago
      I run Mastodon on a $5 VM. If I hit HN homepage, my site will probably be slow as well. I don’t see how that reflects on Mastodon though.
      - endisneigh 4y ago
        what am I saying is that there's a way to design a site on a $5 VM that serves static text that wouldn't be slow even if it was #1 on HN. if you disagree with that then we'll have to agree to disagree.
- shakow 4y ago
  Feel free to share your know-how https://github.com/mastodon/mastodon https://github.com/mastodon/mastodon

webdood90 4y ago

> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. incredible what mental hoops people will jump through to disqualify AI! the reality is that a lot of things have a terrible UI with unorganized data. that's why this tool is so amazing - because it doesn't matter anymore.

endisneigh 4y ago
> the reality is that a lot of things have a terrible UI with unorganized data. that's why this tool is so amazing - because it doesn't matter anymore. how naive. how do you know it's right? ah, you have to manually do the calculation anyway to confirm, this is what Tao ended up saying in a reply asking as much. AI is great, but it's not a silver bullet, since its correctness can never be 100% under the current LLM framework.
- webdood90 4y ago
  I'm convinced people like you are in for such a rude awakening. I don't understand how you can hold this position with AI considering it's only the beginning.
  - endisneigh 4y ago
    ChatGPT is impressive, but I do not think it is a silver bullet, even for data cleansing and processing tasks. I would use your words and say people who think (that ChatGPT solves the problem of data cleansing once and for all) so are the ones who will be in for a rude awakening.
  - ipaddr 4y ago
    I can't wait for 3d printers to take over manufacturing or self driving cars to self drive. Still waiting for the metaverse. Or cryto to take over banking. We are not even at those points for AI. You might see the future.. let's get a 3d printer in every home first.
    - 4y ago

hn_throwaway_99 4y ago

> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Umm, and I want a pony? The world doesn't come in perfectly structured data formats. I was pretty amazed when I pasted in some of my medical lab test results and ChatGPT was accurately able to parse everything out and summarize my data. It worked extremely well.

endisneigh 4y ago
> The world doesn't come in perfectly structured data formats. In Tao's case, the data was organized already, simply not given to him. In any case, Tao, per the post, had to manually calculate to confirm that GPT-4 was correct in any case (he implies as much in a comment asking if he checked for correctness).
- WastingMyTime89 4y ago
  Checking a table for correctness and filling up a table are not the same thing. I also check for correctness when I manually count entries. It's an extremely error prone task.
  - endisneigh 4y ago
    How exactly could you check for the correctness without effectively finding out the answer? I mean this discrepancy is fundamentally the problem of P = NP or not.
    - wetmore 4y ago
      NP is the class of problems for which it's hard to find the answer but easy to verify. The fact such a class even exists should hint to you that finding the answer and checking correctness can be two different things.
      - endisneigh 4y ago
        the point of my comment is that whether P = NP to begin with is still to be determined. in the case of ChatGPT, it's easy for it to respond, but how difficult is it for you to verify that it was correct, or not? how does that compare with the difficulty in doing the task to begin with?

fauxpause_ 4y ago

Solving for every last mile problem in UX is hard though. A lot of AI value is being a flexible, generic interface to software automation.

SkyMarshal 4y ago
We can probably even think of AI as a generalized meta-interface to everything - there’s probably no better UI/UX than natural human language that we’ve all been speaking since childhood.
- fauxpause_ 4y ago
  I don’t know that it’s a better UX, but it is a viable UX, and that is very valuable

rvz 4y ago

> wow, the site is so slow. getting "This resource could not be found" errors on reloads. Oh dear. Federation at its finest /s If the instance is too small, it will easily fall over under heavy traffic. If it is too big, it is highly centralized and even then it cannot scale at a maximum of 300,000+ users at the same time. Eventually, they would all be re-centralizing back to Cloudflare. Once that goes down, all of the biggest Mastodon instances go down at once. As for GPT-4 in mathematics (LLMs in particular), in goes inline on what I said before. Fundamentally, these LLMs cannot reason transparently, nor can it directly bring up its own mathematical proofs with thorough explanations that allows mathematicians to work with. Only low hanging fruit of summarization of existing text. If Mr. Tao can see its limitations, surely it puts the hype of unexplainable magical black-box LLMs to rest as being great sophists and bullshit generators.

williamstein 4y ago
I would not discount the potential of LLM's to support research level mathematics. There's interesting research going into combining the LEAN theorem prover with LLMs, e.g., this recent paper: https://arxiv.org/abs/2202.01344 https://arxiv.org/abs/2202.01344 "... automatically solving multiple challenging problems drawn from high school olympiads." This combination of GPT and LEAN is the first thing that has finally got me really interested in the potential of LEAN. And LEAN very directly addresses the "bullshit" issue, since it formally verifies whether or not a proof is correct (it's like a compiler like Rust, but for mathematical proofs).

77pt77 4y ago

https://archive.is/wbwUa https://archive.is/wbwUa

HelloMcFly 4y ago

> In this case GPT-4 is used to solve a problem that shouldn't have even been one to begin with. But the problem DID exist, and problems like that are likely to continue to exist for the foreseeable future.

helen___keller 4y ago

> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? I’m fairly convinced that for many problems (including this), AI is not the best solution but will become the preferred solution. “Best” doesnt get in the way of “good enough” when convenience is at stake (and making a good UX is often very very inconvenient)

teleforce 4y ago
I think it's not the case of AI or ChatGPT is the best solution because it can be a local optima. But it's currently the "best" solution available to the average Joe. The fact in this OP case study the average Joe is probably one of the best mathematicians of our time, makes the situation much more interesting and intriguing. I'm in the middle of reading Steve Jobs' very own words, Make Something Wonderful book. Apparently throughout his whole life he frequently mentioned and discussed a lot about providing and creating easy access to the computer because ultimately people do not want to program but want to use computer instead. He did mention about how Morse code was not popular as much of the later telephone technology because people refused to learn and use the unintuitive Morse code even though it only takes 40 hours for average person to learn the entire Morse code. As a trained communication engineers I can very much relate to this because even though we learnt much of the underlying technology of communication that enable the Morse code or the telephone to work across the Atlantic ocean most of us can't be bothered to learn the Morse code and use it if we can help it because the telephone make it redundant and it's counter intuitive to use. Granted, now we know that telephone or circuit switching in general is a suboptimal in providing and solving the human communication problems and needs. As of now the Internet, packet switching and the multimedia approaches are probably the best solutions but telephone has served us a potent and still one of the best solutions for communication until very recently.

poulpy123 4y ago

> Why was the data so disjoint to begin with? Because real life is messy. > Why is Latex so hard to work with? Because people that use latex want to be able to brag they use latex (just joking for the second one.... unless ?)

endisneigh 4y ago
> Because real life is messy. This doesn't make sense - the data inherently is centralized. Tao implied as much.

hzay 4y ago

Imagine Terrence Tao manually entering data into spreadsheets!!

SkyMarshal 4y ago

> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? > In this case GPT-4 is used to solve a problem that shouldn't have even been one to begin with. Friction. These problems shouldn’t exist, but they do anyway, and they’re everywhere. Anything human is inevitably going to be imperfect and messy to some greater or lesser degree, introducing friction into dealing with it. Especially as we produce more and more unstructured or semi-structured data, which AI is particularly good at wrangling. If AI can help us cut through that friction significantly faster and more accurately, that’s a win.

moffkalast 4y ago
Let's just hope that someone doesn't task the AGI with eliminating friction and it realizes that humans are the problem.
layer8 4y ago
Ironically, interacting with ChatGPT still comes with a lot of human-like friction and imperfection.
- dwringer 4y ago
  IME most of that is of the form "I'm sorry, but as a large language model I..." and seems to have been drilled into it with training to dodge tricky subjects.

Jeff_Brown 4y ago

Better source data is indeed a better solution, but the source data is often not under one's control.

dpcan 4y ago

They should have done it right the first time? That's not the world we live in. Coworkers, associates, clients, family - everyone sends us funky data. Sometimes it's a PDF of text instead of a text file, or a picture of their computer screen with a paragraph in it, or copy-pasted text with unwanted formatting, etc, etc. And the point of this article wasn't really about the tedious job of copy/pasting, it was that once the data was ready, instead of hunkering down and having to work through a problem with an unknown number of variables, GPT-4 can drop an answer in seconds flat. Maybe a few more if you have to tweak your prompt a few times. Part 2 here is that we have a problem to solve, and that's how data is sent and received, and that it all needs to end up in a structural format that's usable by our AI tools. I am thinking that writing a script that can dump textual data from literally anything I'm looking at so it's AI-ready makes a lot of sense, then we have step 1 taken care of as well for my own personal GPT-solvable situations.

PurpleRamen 4y ago

> Ironically, Tao's post convinces me that AI, though amazing, isn't really the solution. Better UX and data quality is. Why was the data so disjoint to begin with? Because the world is not a database. His source were formats meant for Human consumption, not machines. This will never change, people are lazy, greedy, or fear "data leaks", so we will never have a machine-first-format that everyone will use. I mean, that guy is using latex, others have markdown, or org-mode, or Microsoft Word, those all are not meant for easy scripting. This is why AI will be such a dealbreaker, because it will close this gap, make human-formats to machine-formats when neccessary.

rhn_mk1 4y ago
I find the "human consumption" here funny, because it's the "human" part which led the author to turn to machines which can do it better.
- pegasus 4y ago
  Well, yes, this is kinda the whole point of the argument. That's what LLMs bring to the table - machines that can better deal with data and documents meant for human consumption.

belugacat 4y ago

And here’s an exchange about how he validated the data was correct - which to me seems like the most important part: > Out of curiosity, are you sure that GPT did it correctly? If yes, is it because you were able to "spot check" it in a few places? Or have you used GPT enough that you trust it with a task like this? Or is this for some kind of internal use where a few small errors is unimportant, and you only need the broad strokes to be correct? > Yes, yes, and yes. (There were some obvious checksums, for instance by computing the total number of speakers several different ways.) Still cool, but the circumstances above aren’t always true. In fact, in a lot of meaningful work, none of them probably are. If GPT is like an assistant whose work you always have to double check in order to be sure of the validity of the results, that seems to be a pretty big caveat to me. In that case it’s probably better to just write a script whose output you know to be deterministic, even if it takes slightly longer (and the bigger your dataset to verify, the more likely it is that it’ll take less time to write a script than to validate results one by one). The scary part is if/when companies/bureaucracies/governments/etc start using GPT for all sorts of Important Tasks and skip the validation part because they assume the machine will always get it right.

humanistbot 4y ago
> If GPT is like an assistant whose work you always have to double check in order to be sure of the validity of the results, that seems to be a pretty big caveat to me. Sounds like every junior developer fresh out of college I've had to work with. Interns are even worse. Even senior devs mess up from time to time. That's why we do code review and other QA. Trust no one.
- belugacat 4y ago
  True, but one obvious difference is that a senior contributor can typically validate the work of a more junior one in less time than it took them to produce, that’s why the overall equation works. And if you decide to keep the more junior person, it’s probably because you feel like they’re improving and becoming more trustworthy over time. Another key difference is that a deterministic script that has determined to be valid will be valid for all subsequent runs, regardless of input size. That is not the case for ChatGPT, which might be correct on some runs and not others, or do well on small input sizes but start messing up when thousands (or more) of data items are involved. Again, I still think ChatGPT is really cool, but thinking about all these nuances about the nature of the work that is actually being done when you use ChatGPT vs writing a script vs handing it off to an intern seems crucial to the debate.

6gvONxR4sf7o 4y ago

> AI, though amazing, isn't really the solution. Better UX and data quality is. In this case, AI was the tool used to have a better UX. People build something with use-case X in mind, and use the best tool for that job. Piping that into use-case Y requires some duct tape and plumbing. It turns out that AI is great at that sort of repurposing. It's great if someone using the best tool for their job is nearly the best tool for the flexible infinity of other use cases, which is what flexible enough duct-tape gets us.

sdfghswe 4y ago

Wait.... is it possible to tell chat gpt to grab data from a page?

ugh123 4y ago

>Better UX and data quality is. Why was the data so disjoint to begin with? Why is Latex so hard to work with? Do you mean build better, specific, UX and normalize data for only this task? Why spend that time, even as a developer of whatever improved app/system you're thinking of, when he can just turn to AI? I'd rather lean on GPT4's ability to generalize highly specific technical work rather than ruminate on each individual app's lackings and how it could be better than using AI if we just... put more effort into it?

gvhst 4y ago

And AI is a way to enable better UX and cleaner data...

jacquesm 4y ago

And how does he know the results are correct without verifying each and every entry?

tjr 4y ago
I find this a real concern when it comes to the proposition of "replace every programmer with GPT". I think the reality is, there are cases where it matters, and cases where it doesn't matter (at least, not very much). I work on aerospace software systems. I suspect that most software developers would be surprised at the process and rigor involved. That does not mean that humans are flawless; of course not. But there are going to be a lot more hurdles to replacing all of the humans with non-deterministic machines. There are numerous software components on a modern airplane. How good is good enough? If the overall system works correctly 99% of the time, is that good enough? There are about 16 million flights annually in the United States alone. If 160,000 of them crash, is that good enough? I mean, like, if Microsoft Word failed to save your document 99% of the time, you might be irked. If Professor Tao's data was only 99% accurate, he probably still has a reasonably good picture of the information he needs. Regardless of if a human or a machine is writing the code, there are different levels of software criticality. As humans, we don't apply avionics software development process to iOS games because it's not worth the extra time and expense. Likewise, there will be a spectrum of where it makes good enough sense to automate tasks with AI and where it doesn't. Which is not to dismiss AI! It's amazingly useful technology. But there really are places where above average accuracy is needed, and AI that works great for handling some tasks might not suffice for other tasks. If you truly need all of the results correct, then a nondeterministic neural network is probably not the best path to be on. There are other methods; even other automated methods.
- jacquesm 4y ago
  > I work on aerospace software systems. I suspect that most software developers would be surprised at the process and rigor involved. I did this once, a fuel estimation program for 747's, the degree of understanding of the problem required to create something that would pass review was off the scale compared to any other program that I've ever written. It is also the only time that the customer wanted to get it right rather than that they were looking at what it cost (because the savings would pay for the development many times over this was probably less of an issue anyway). I'd love to be able to always work like that.

csomar 4y ago