5 ms·

I would expect nothing but hallucinations and nonsense coming out of any LLM regarding recently-released movies (aka. the ones you often find on flights).

by Alupis 2y ago

tracerbulletx 2y ago
In every post about LLMs there is someone to blindly say something like this. When in reality if you ask ChatGPT for 10 good movies from this year you will get this. Anora - Directed by Sean Baker, a compelling drama about the life of a sex worker in Coney Island. Challengers - A provocative tennis drama directed by Luca Guadagnino, starring Zendaya. Dune: Part Two - Denis Villeneuve's continuation of the epic science fiction saga. Furiosa: A Mad Max Saga - An action-packed prequel exploring the origins of Furiosa, directed by George Miller. Inside Out 2 - Pixar's sequel that dives deeper into the complexities of human emotions. Wicked - A musical fantasy adaptation directed by Jon M. Chu . The Zone of Interest - A thought-provoking film about Auschwitz, directed by Jonathan Glazer. The Idea of You - A steamy romance starring Anne Hathaway. Hit Man - A comedy thriller starring Glen Powell. The Outrun - A powerful drama about a recovering alcoholic, starring Saoirse Ronan. Let me know if you'd like more details about any of these! Which is a great list.
- tehwebguy 2y ago
  These details are already available in the in flight entertainment interface
  - tracerbulletx 2y ago
    This type of response is called moving the goal post. When someone responds to one claim, the claim is changed to something different which was not part of the original argument. This is debating in bad faith.
    - deleted 2y ago
      [deleted]
  - glenstein 2y ago
    They were responding to a commenter suggesting it would produce completely unusable results, the question was never about whether the results produced would be redundant. I know that any mention of fallacies, valid or otherwise, causes instinctive eye rolls, but in this instance I agree with them that this amounts to moving the goalposts.
- slg 2y ago
  Those descriptions are less detailed than the information you will see on basically any streaming interface and yet it still manages to not being very good. For example, no person who had actually seen Anora would describe it as "a compelling drama about the life of a sex worker in Coney Island".
  - glenstein 2y ago
    I haven't seen Anora so I'll give you that one, but you cited that as if it was just one of many examples, when in fact I think it's the only one, as all the other descriptions seem reasonable. Originally the problem was supposedly that it would hallucinate complete and utter gibberish, but now here we are quibbling over one example and insisting that maybe it's not quite as good as alternative descriptions. The gap between what was produced and what you're looking for is small enough that I think it could be covered with some slightly tweaked prompt instructions. I'm not saying you're wrong but want to note how the goalposts keep seeming to shift whenever we talk about these capabilities.
    - slg 2y ago
      I'm not Alupis. I can't and am not trying to speak on their behalf. I'm therefore not moving the goalposts they established. I'm making my own related point. That point is that the information provided above about these movies is worthless. It does not add any new value beyond what would already be available in the streaming interface. Several of the descriptions are nothing but the genre and one person involved in the making of the movie. And yet even with these descriptions being incredibly short and vague, they still manage to contain at least one misleading summary.
      - glenstein 2y ago
        I'm aware that you're a different commenter but you are addressing yourself to a comment that was in reply to them and therefore not necessarily appropriate to measure such a comment against entirely new criteria that you want to bring into the conversation. Despite your protestations to the contrary, these descriptions seem perfectly fine in that they're accurate and meaningful. And it if you want to start getting fast and loose with all kinds of new extra criteria and requirements for what it's supposed to do, they all seem squarely within the reach of the capabilities on offer, with some prompt tweaks.
- deleted 2y ago
  [deleted]
- th0ma5 2y ago
  Great! Now show me a system that can verify that list for accuracy as well. Not to be flippant, but this is the complaint. You can't approach outputs uncritically. And no I don't want it to be as unreliable as a person who also forgets how English or basic knowledge works at random intervals.
  - tracerbulletx 2y ago
    The system is I'm a movie nerd who knows all the best movies of the year and have seen them all.
  - conductr 2y ago
    Movie reviews are amongst the most subjective things, how do you define “verify” and “accuracy”?
    - th0ma5 2y ago
      If I run a business, I guess I would have the ability to bring a libel suit would be one way?
  - JadeNB 2y ago
    > Great! Now show me a system that can verify that list for accuracy as well. Not to be flippant, but this is the complaint. You can't approach outputs uncritically. In general you can't, but surely it's not that big a deal if ChatGPT offers an inaccurate summary of a movie you're about to use to kill time on a flight? I suppose it becomes important if, e.g., you're relying on it to tell you whether a movie is appropriate for children, but, if you're just asking it whether a movie is worth watching, that's a question that doesn't have an objective, factual answer anyway, so a hallucinated answer is probably about as useful as that of a not-previously-known reviewer.
    - th0ma5 2y ago
      If I invested money into a film, I would want its representation in the world to reflect what the movie is about at the very least.
bsimpson 2y ago
Here's a comparison of asking ChatGPT and Meta AI about actual in-flight movie choices. I pasted the same initial prompts in both, but Meta AI needed more clarification. When ChatGPT found multiple entries with similar titles, it gave information about all of them. https://gist.github.com/appsforartists/004bafe11a9e23a418fd5c902910111f https://gist.github.com/appsforartists/004bafe11a9e23a418fd5...
- slg 2y ago
  >[The Campaign] received mixed-to-positive reviews from critics. On the Rotten Tomatoes website, it holds a 65% approval rating from critics, based on 191 reviews, with an audience score of 60%. The first thing I fact-checked, the Rotten Tomatoes scores are actually 66% and 51% respectively[1]. Probably not enough of a difference to sway any opinions, but an excellent example of the type of inaccuracy that the previous comment was referencing. [1] - https://www.rottentomatoes.com/m/the_campaign https://www.rottentomatoes.com/m/the_campaign
bdangubic 2y ago
[dead]
johnbatch 2y ago
Looks like this is using GPT-4 and has no knowledge after January 2022. ' As of my knowledge cutoff in January 2022, the last movie I have information on is "Spider-Man: No Way Home", which was released in theaters in December 2021. It was one of the most highly anticipated films of that year, marking a major event in the Marvel Cinematic Universe (MCU) and the Spider-Man franchise. '
deleted 2y ago
[deleted]
freedomben 2y ago
I've heard that Twitter's (AKA X's) LLM (Grok) is really, really good at this sort of thing (in part because it has recent access to Twitter's data).
- lxgr 2y ago
  Web or other search access for LLMs really isn’t that new anymore, and I doubt that Grok will do a statistically significant sampling of everything on X, so I don’t really expect it to fare much better than a model with access to regular web search.
SkyPuncher 2y ago
You’d be pretty wrong, then. ChatGPT in particular will cite its sources via an internet site. My wife wanted a pair of boots for Christmas that I couldn’t find in her size. Google was a wasteland of SEO, but ChatGPT found 5 sites and was able to tell me current stock levels.
lxgr 2y ago
Llama in WhatsApp can search the web, so usually gets these queries right. Hilariously it often believes that it can’t access the web and then hallucinates reasons for how it can know things beyond its knowledge cutoff date. But in any case, it works very well for this use case.

I would expect nothing but hallucinations and nonsense coming out of any LLM regarding recently-released movies (aka. the ones you often find on flights).

by Alupis 2y ago

I would expect nothing but hallucinations and nonsense coming out of any LLM regarding recently-released movies (aka. the ones you often find on flights).

tracerbulletx 2y ago

In every post about LLMs there is someone to blindly say something like this. When in reality if you ask ChatGPT for 10 good movies from this year you will get this. Anora - Directed by Sean Baker, a compelling drama about the life of a sex worker in Coney Island. Challengers - A provocative tennis drama directed by Luca Guadagnino, starring Zendaya. Dune: Part Two - Denis Villeneuve's continuation of the epic science fiction saga. Furiosa: A Mad Max Saga - An action-packed prequel exploring the origins of Furiosa, directed by George Miller. Inside Out 2 - Pixar's sequel that dives deeper into the complexities of human emotions. Wicked - A musical fantasy adaptation directed by Jon M. Chu . The Zone of Interest - A thought-provoking film about Auschwitz, directed by Jonathan Glazer. The Idea of You - A steamy romance starring Anne Hathaway. Hit Man - A comedy thriller starring Glen Powell. The Outrun - A powerful drama about a recovering alcoholic, starring Saoirse Ronan. Let me know if you'd like more details about any of these! Which is a great list.

tehwebguy 2y ago
These details are already available in the in flight entertainment interface
- tracerbulletx 2y ago
  This type of response is called moving the goal post. When someone responds to one claim, the claim is changed to something different which was not part of the original argument. This is debating in bad faith.
  - deleted 2y ago
    [deleted]
- glenstein 2y ago
  They were responding to a commenter suggesting it would produce completely unusable results, the question was never about whether the results produced would be redundant. I know that any mention of fallacies, valid or otherwise, causes instinctive eye rolls, but in this instance I agree with them that this amounts to moving the goalposts.
slg 2y ago
Those descriptions are less detailed than the information you will see on basically any streaming interface and yet it still manages to not being very good. For example, no person who had actually seen Anora would describe it as "a compelling drama about the life of a sex worker in Coney Island".
- glenstein 2y ago
  I haven't seen Anora so I'll give you that one, but you cited that as if it was just one of many examples, when in fact I think it's the only one, as all the other descriptions seem reasonable. Originally the problem was supposedly that it would hallucinate complete and utter gibberish, but now here we are quibbling over one example and insisting that maybe it's not quite as good as alternative descriptions. The gap between what was produced and what you're looking for is small enough that I think it could be covered with some slightly tweaked prompt instructions. I'm not saying you're wrong but want to note how the goalposts keep seeming to shift whenever we talk about these capabilities.
  - slg 2y ago
    I'm not Alupis. I can't and am not trying to speak on their behalf. I'm therefore not moving the goalposts they established. I'm making my own related point. That point is that the information provided above about these movies is worthless. It does not add any new value beyond what would already be available in the streaming interface. Several of the descriptions are nothing but the genre and one person involved in the making of the movie. And yet even with these descriptions being incredibly short and vague, they still manage to contain at least one misleading summary.
    - glenstein 2y ago
      I'm aware that you're a different commenter but you are addressing yourself to a comment that was in reply to them and therefore not necessarily appropriate to measure such a comment against entirely new criteria that you want to bring into the conversation. Despite your protestations to the contrary, these descriptions seem perfectly fine in that they're accurate and meaningful. And it if you want to start getting fast and loose with all kinds of new extra criteria and requirements for what it's supposed to do, they all seem squarely within the reach of the capabilities on offer, with some prompt tweaks.
deleted 2y ago
[deleted]
th0ma5 2y ago
Great! Now show me a system that can verify that list for accuracy as well. Not to be flippant, but this is the complaint. You can't approach outputs uncritically. And no I don't want it to be as unreliable as a person who also forgets how English or basic knowledge works at random intervals.
- tracerbulletx 2y ago
  The system is I'm a movie nerd who knows all the best movies of the year and have seen them all.
- conductr 2y ago
  Movie reviews are amongst the most subjective things, how do you define “verify” and “accuracy”?
  - th0ma5 2y ago
    If I run a business, I guess I would have the ability to bring a libel suit would be one way?
- JadeNB 2y ago
  > Great! Now show me a system that can verify that list for accuracy as well. Not to be flippant, but this is the complaint. You can't approach outputs uncritically. In general you can't, but surely it's not that big a deal if ChatGPT offers an inaccurate summary of a movie you're about to use to kill time on a flight? I suppose it becomes important if, e.g., you're relying on it to tell you whether a movie is appropriate for children, but, if you're just asking it whether a movie is worth watching, that's a question that doesn't have an objective, factual answer anyway, so a hallucinated answer is probably about as useful as that of a not-previously-known reviewer.
  - th0ma5 2y ago
    If I invested money into a film, I would want its representation in the world to reflect what the movie is about at the very least.

bsimpson 2y ago

Here's a comparison of asking ChatGPT and Meta AI about actual in-flight movie choices. I pasted the same initial prompts in both, but Meta AI needed more clarification. When ChatGPT found multiple entries with similar titles, it gave information about all of them. https://gist.github.com/appsforartists/004bafe11a9e23a418fd5c902910111f https://gist.github.com/appsforartists/004bafe11a9e23a418fd5...

slg 2y ago
>[The Campaign] received mixed-to-positive reviews from critics. On the Rotten Tomatoes website, it holds a 65% approval rating from critics, based on 191 reviews, with an audience score of 60%. The first thing I fact-checked, the Rotten Tomatoes scores are actually 66% and 51% respectively[1]. Probably not enough of a difference to sway any opinions, but an excellent example of the type of inaccuracy that the previous comment was referencing. [1] - https://www.rottentomatoes.com/m/the_campaign https://www.rottentomatoes.com/m/the_campaign

bdangubic 2y ago

[dead]

johnbatch 2y ago

Looks like this is using GPT-4 and has no knowledge after January 2022. ' As of my knowledge cutoff in January 2022, the last movie I have information on is "Spider-Man: No Way Home", which was released in theaters in December 2021. It was one of the most highly anticipated films of that year, marking a major event in the Marvel Cinematic Universe (MCU) and the Spider-Man franchise. '

deleted 2y ago

[deleted]

freedomben 2y ago

I've heard that Twitter's (AKA X's) LLM (Grok) is really, really good at this sort of thing (in part because it has recent access to Twitter's data).

lxgr 2y ago
Web or other search access for LLMs really isn’t that new anymore, and I doubt that Grok will do a statistically significant sampling of everything on X, so I don’t really expect it to fare much better than a model with access to regular web search.

SkyPuncher 2y ago

You’d be pretty wrong, then. ChatGPT in particular will cite its sources via an internet site. My wife wanted a pair of boots for Christmas that I couldn’t find in her size. Google was a wasteland of SEO, but ChatGPT found 5 sites and was able to tell me current stock levels.

lxgr 2y ago

Llama in WhatsApp can search the web, so usually gets these queries right. Hilariously it often believes that it can’t access the web and then hallucinates reasons for how it can know things beyond its knowledge cutoff date. But in any case, it works very well for this use case.