34 ms·

Guess the author didn't bother to check that those books actually are correct? The first one I checked, "Growing Up with Lucy by April Henry" doesn't exist. The

Guess the author didn't bother to check that those books actually are correct? The first one I checked, "Growing Up with Lucy by April Henry" doesn't exist. The actual book is by Steve Grand, and it's very obviously so in the video used as input. So a cool demo, but sadly useless for anything more.

flextheruler 3y ago
I think this post and others reactions and then your comment this far down really encapsulates where we’re at with this technology. Nearly 90 percent of comments on posts about LLMs are people talking about how the near future is about to boggle our minds and that general intelligence is near, but all my experiences with these LLMs show they’re capable of making the most basic of mistakes and doing so confidently and that’s just the tip of the iceberg in terms of their problems. I’m having a hard time buying into the hype that these will be able to competently replace nearly any job anytime soon. They’re useful tools but they all come with a big asterisk of human hand holding.
- TrackerFF 3y ago
  Humans are also perfectly capable of confidently doing mistakes. The big difference here is that these models can scale the work beyond human capability. Why pay 10000 mechanical turks to extract information from vids, if you can deploy N of these models, and get the work done at a fraction of the time? Instead you can keep x% of the MTurks to check the vids where the model yields some high uncertainty score, and randomly audit other vids for quality assurance. There's crazy amounts of potential in these things. Hell, the place I work at has already replaced certain human tasks with LLM-integrated solutions, with extremely good results.
bergenbergen 3y ago
Thanks for this comment. I am yet to see any “art” produced by AI that is not superficial or hollow (best case) or deeply unsettling (common case).
- fragmede 3y ago
  how much time do you spend looking at AI art though? a casual jaunt through midjourney will certainly get you some weird things, but there are some gems in there (but also a lot of weird).
- TrackerFF 3y ago
  But then you have all the things you don't see. The CGI/fx artist that spent hours upon hours handcrafting realistic background CGI to some movie scene? Could very well be replaced in the not-so-distant future. The first huge wave of ML/AI automation will involve all the things you don't notice straight away.
TrackerFF 3y ago
No one is expecting a 0% error rate. As long as it is on par (or better) and faster than humans, that's good enough to get the ball rolling. Curious to see how I fared at the task (first vid), I used just over 4 minutes writing down the books with readable titles - and got 36 of them. Seems like there are 56-57 or something like that. So I roughly got two thirds of the books in the video. But that's still 4 minute of pausing and sliding the video for the book titles alone.
sensanaty 3y ago
For most of the people hyping up AI it doesn't matter that it makes things up more often than it doesn't. They're here to sell hype so they can build the 9 millionth startup that sells you a wrapper for one of these models, not to do anything useful or advance humanity or whatever other confabulations they like to pretend to care about
danjc 3y ago
Great for creative tasks where precision isn't required.
simonw 3y ago
I called out one hallucination - "The Personal MBA by Josh Kaufman" wasn't on my shelf. I didn't bother fact-checking every other book because I thought highlighting one mistake would illustrate that the results weren't accurate - which is pretty much expected for anything related to LLMs at this point.
- dlandau 3y ago
  I don't think highlighting one mistake is enough, when these can sometimes have more mistakes than corrects. I've found use for LLMs (in large part thanks to your teaching) in cases where I can easily verify the results fully like code and process documentation, but tasks where "fact-checking everything" would be too much work are very much on the danger zone for getting accidentally scammed by AI.
- timeon 3y ago
  No result is better than misinformation.

flextheruler 3y ago

I think this post and others reactions and then your comment this far down really encapsulates where we’re at with this technology. Nearly 90 percent of comments on posts about LLMs are people talking about how the near future is about to boggle our minds and that general intelligence is near, but all my experiences with these LLMs show they’re capable of making the most basic of mistakes and doing so confidently and that’s just the tip of the iceberg in terms of their problems. I’m having a hard time buying into the hype that these will be able to competently replace nearly any job anytime soon. They’re useful tools but they all come with a big asterisk of human hand holding.

TrackerFF 3y ago
Humans are also perfectly capable of confidently doing mistakes. The big difference here is that these models can scale the work beyond human capability. Why pay 10000 mechanical turks to extract information from vids, if you can deploy N of these models, and get the work done at a fraction of the time? Instead you can keep x% of the MTurks to check the vids where the model yields some high uncertainty score, and randomly audit other vids for quality assurance. There's crazy amounts of potential in these things. Hell, the place I work at has already replaced certain human tasks with LLM-integrated solutions, with extremely good results.

bergenbergen 3y ago

Thanks for this comment. I am yet to see any “art” produced by AI that is not superficial or hollow (best case) or deeply unsettling (common case).

fragmede 3y ago
how much time do you spend looking at AI art though? a casual jaunt through midjourney will certainly get you some weird things, but there are some gems in there (but also a lot of weird).
TrackerFF 3y ago
But then you have all the things you don't see. The CGI/fx artist that spent hours upon hours handcrafting realistic background CGI to some movie scene? Could very well be replaced in the not-so-distant future. The first huge wave of ML/AI automation will involve all the things you don't notice straight away.

TrackerFF 3y ago

No one is expecting a 0% error rate. As long as it is on par (or better) and faster than humans, that's good enough to get the ball rolling. Curious to see how I fared at the task (first vid), I used just over 4 minutes writing down the books with readable titles - and got 36 of them. Seems like there are 56-57 or something like that. So I roughly got two thirds of the books in the video. But that's still 4 minute of pausing and sliding the video for the book titles alone.

sensanaty 3y ago

For most of the people hyping up AI it doesn't matter that it makes things up more often than it doesn't. They're here to sell hype so they can build the 9 millionth startup that sells you a wrapper for one of these models, not to do anything useful or advance humanity or whatever other confabulations they like to pretend to care about

danjc 3y ago

Great for creative tasks where precision isn't required.

simonw 3y ago

I called out one hallucination - "The Personal MBA by Josh Kaufman" wasn't on my shelf. I didn't bother fact-checking every other book because I thought highlighting one mistake would illustrate that the results weren't accurate - which is pretty much expected for anything related to LLMs at this point.

dlandau 3y ago
I don't think highlighting one mistake is enough, when these can sometimes have more mistakes than corrects. I've found use for LLMs (in large part thanks to your teaching) in cases where I can easily verify the results fully like code and process documentation, but tasks where "fact-checking everything" would be too much work are very much on the danger zone for getting accidentally scammed by AI.
timeon 3y ago
No result is better than misinformation.