46 ms·

One of the interview questions I ask all UX Researchers is to open up rottentomatoes, sort recent films by rating and then look at the movies that have the lowe

by Kranar 5y ago

One of the interview questions I ask all UX Researchers is to open up rottentomatoes, sort recent films by rating and then look at the movies that have the lowest critic scores. Almost always you can see the audience score for those movies is much higher than the critic scores. I then ask why this is the case. The overwhelming majority of answers involve critics being out of touch with the audience, critics being perhaps snobby, or watching movies in an elitist fashion whereas the audience just wants to sit back and relax, not think too much, just be entertained. A very small minority of UX Researchers come to the correct conclusion, which is that most professional critics have to review every single movie, whereas the audience typically only goes to see movies that they are likely to enjoy, so that one should not be surprised that the audience score will often be significantly higher than the critic score. Those researchers are typically the ones I hire.

raincom 5y ago
Actually, both statements boil down to the same with little modifications: (a) being out of touch with the audience is same as (b) the audience typically only going to see movies that they are likely to enjoy. (a) is broader than (b), can subsume (b).
parkingrift 5y ago
Written another way... You interview people, ask them their opinions, and hire those that agree with your opinion. The most excellent way I've seen to hire a group of people biased toward a singular way of thinking.
yhoneycomb 5y ago
“I hire people who think exactly like I do”
- tomrod 5y ago
  When you have the right mental model, this is an appropriate approach. Not every mental model is correct, either locally to a domain or globally in all cases. 1 < 2, in almost all cases except for small Z_{0,1} or similar type cases where mod functions modify the space to something exotic or comparator functions are defined far out of the typical. If you think there is a case for 2 < 1, and you aren't appealing to something exotic, you have the wrong approach.
  - _v7gu 5y ago
    Any mental model that assigns zero weight to the probability of being wrong is a wrong mental model. That said, biases arising from endogeneity might have negative effects too. You can't conclude a parameter should have a different/zero sign just because of endogeneity, you have to go fix your model, and re-estimate the parameters.
    - tomrod 5y ago
      > Any mental model that assigns zero weight to the probability of being wrong is a wrong mental model. This is true when there is uncertainty. It also doesn't connect to "I only hire people who think like I do", which is the context of my response, so I don't follow your point. Agreed on your other point that you need to instrument endogeneity. Another requirement is plausibility. Hookworm presence in Greenland should be uncorrelated to solar sunspots.
babypuncher 5y ago
I find it distressing how many people miss the obvious selection bias at play here.
vmception 5y ago
and whats the explanation for the opposite? when critics scores are much higher than audience scores? I assume "out of touch" but almost any explanation I think of and ridicule could be seen as a strawman argument, so I'll wait for your response.
- joebob42 5y ago
  I could imagine the answer being: People went to see these movies not because they wanted to per se but because critics said they were amazing. And because they went mostly due to critical reviews, anyone who didn't like it was even more likely than normal to be like "wait I feel tricked this was bad" This might be a bit of a stretch, but it seems almost plausible.
  - dnissley 5y ago
    Two more possibilities: - A film may have been marketed as one particular type of movie via trailers (e.g. as a horror film) but was actually a different type of movie, often a subversion of the genre it was marketed as (e.g. ended up as social commentary). So audiences feel deceived and report that in their ratings. - Similarly, there are movies that are "applause lights" where critics are obliged to give positive ratings due to the social significance of a film (e.g. made by a woman / featuring a woman in a lead role). But the movie might just not be that good, and audiences report this in their ratings.
    - vmception 5y ago
      > social significance of a film (e.g. made by a woman / featuring a woman in a lead role) for a less divisive example: movies about the film or theatre industry get high marks and fit your example while the audience is left scratching their heads
      - ripe 5y ago
        Excellent point. Critics have often remarked that movies about Hollywood tend to get Oscars just because of the subject matter, which Academy members like. Also, a less cynical example: some movies are about something new: you might see people who live in a developing country, e.g., Somali fishermen in Captain Philips, or in unusual occupations (e.g., WWF wrestler in The Wrestler). Most audiences just want to relax and not learn something new, but critics love these kinds of movies.
omarhaneef 5y ago
Never thought of that but it makes a lot of sense! Edit: worth noting that this also creates an implicit prediction about the extent to which the advertising is an accurate representation of the movie. This could go in the opposite direction and audience scores would be lower than the critical assessment.
ashtonbaker 5y ago
That's a clever way of looking at it. Did you come up with it yourself? If not, I wonder what you think it says about a person that they were exposed to this idea before they interviewed with you, versus after.
- klodolph 5y ago
  IMO it seems obvious to me, but that might be because I took a statistics class in college. (Not “obvious in retrospect”—I’ve had a number of personal discussions with people where we talk about the difference between critic & user scores on Metacritic.)
  - wccrawford 5y ago
    You also don't have any emotional or financial investment in the situation, though. ;)
    - klodolph 5y ago
      I don’t understand what you’re trying to say.
- Kranar 5y ago
  No I did not come up with it myself. It's a common pitfall in statistics known as selection bias [1]. My problem is that if I ask a researcher about selection bias, they have no problem defining it for me and even giving me a textbook example of it. But when I present to a researcher a real life scenario that is highly susceptible to selection bias, they will almost all fail to identify it. It's a problem where many people are able to compartmentalize knowledge but fail to actually make use of that knowledge in practice. As a run a quant firm I see this among so many researchers across many disciplines. They fully understand a concept when you ask them directly, but they fail to put those concepts to use outside of a very direct and artificial setting. [1] https://en.wikipedia.org/wiki/Selection_bias https://en.wikipedia.org/wiki/Selection_bias
  - 5y ago
logicalmonster 5y ago
> come to the correct conclusion, What makes you so sure that this is the correct interpretation? Both things could be true at the same time: there could be selection bias at the same time as critics are perhaps out of touch and snobby. You could backtest this theory: see if perhaps critics scores were at one time a bit closer to general audiences or not. I think you'd find that at one time good critics were a bit closer to general audience scores.
- handmodel 5y ago
  Looking at the mechanisms alone I think its pretty unlikely that film critics have not shifted over time. The standard film critic job in the 80s/90s was a local newspaper. That meant your audience was men/women of various ages and political beliefs who happened to live in the same area. If you were a successful critic you had to write something that appealed to all of them. This was actually a vital service that people used to determine if a movie was good. Now the typical movie critic is online and has an audience which probably has a singular viewpoint. This isn't bad - but makes it unlikely the critics with "everyman" sensibilities would do well. People who consume an above average amount of film reviews will have their views represented more. Chuck Klosterman (who was once a local newspaper movie reviewer and now writes to a certain type of audience online) has written about this.
- tshaddox 5y ago
  It's also possible for critics to have systematically different preferences than movie-goers that don't deserve to be described pejoratively as "out of touch" or "snobby." In fact, that's precisely what I would expect and hope to see from people who are professional critics. I wouldn't want expect a list of best restaurants from a food critic to be identical to the list of top restaurant chains by revenue (1: McDonalds, 2: Starbucks, etc.).
  - logicalmonster 5y ago
    > I wouldn't want expect a list of best restaurants from a food critic to be identical to the list of top restaurant chains by revenue That's a very good point that the role of the critic is to provide their own expertise and opinions on the overall subject. But the critic still has to be within the ballpark of popular tastes if the expectation is a mass audience that really trusts their recommendations, right? Many Americans might listen to a food critic that said that he thought some pasta dish at some Italian restaurant was the best, even if pasta is not currently everybody's favorite food. But if he he recommended boiled snails or braised calf brains, many people might ignore his judgement because it's a bit too far out of range of what they're comfortable with. At a certain distance from popular opinion, a mass audience might no longer trust that critic. Just a thought.
deleted 5y ago
[deleted]
fullshark 5y ago
Gotcha type interview questions with a single "trick" that you either get/have seen before or don't provide terrible signal imo.
- thaway2839 5y ago
  The comment literally states that it's one of the questions they ask, which is the exact opposite of a gotcha question. And the specific question can lead to a conversation, which may "reveal" the answer (obviously, I don't know whether this is the correct answer), but at the very least it reveals whether the interviewee thinks about selection bias, among other concerns.
  - tshaddox 5y ago
    > The comment literally states that it's one of the questions they ask, which is the exact opposite of a gotcha question. Huh? "Gotcha question" doesn't imply it's a single question that's not part of a series of questions. A "gotcha question" is simply a question designed such that it's very easy to respond with something that sounds bad or is incorrect.
    - bee_rider 5y ago
      Is it really easy not to come up with the desired explanation here? It seems like the most obvious possible answer, that people tend to watch the types of movies that they like. Actually it seems to provide almost no information unless their hiring pool is mostly candidates who utterly lack common sense.
      - setr 5y ago
        I recall reading that someone found fizzbuzz (with nothing fancy) to be a highly effective filter for programming interviews Hiring pools tend to be filled with nonsense candidates
mikequinlan 5y ago
The article claims that the difference between audiences and critics has been increasing over the last two decades. If the difference was only because audiences self-selected while critics had to review everything, why would it be increasing over time?
- KingOfCoders 5y ago
  No clue, one moonshot people live more in a bubble and narrow their film selection?
- V-2 5y ago
  If anything, I'd expect that the audience would nown have more of a hit-and-miss experience than 20 years ago - with the abundance of streaming services at your fingertips, it's much easier to binge watch movies just because, whereas, say, having to rent a VHS tape had to bea more conscious and committal decision.
bufferoverflow 5y ago
If your narrative was true, we would always see critics scores lower than the audience (assuming statistically significant number of critics reviewing a given movie). But we see so many counterexamples with radically different ratings: Spy kids, Ad Astra, Noah, King Kong (2005), Babe, The Blair Witch Project, Chicken Run, and thousands of others.
- UnpossibleJim 5y ago
  Not to mention a number of films who stopped the audience scores, decrying "review bombing". Some of these films were fine, while some were objectively bad (-cough- Ghostbusters reboot -cough-). I have a feeling (though it is unsubstantiated rumor) that there are payoffs in the critics circles for certain movies.
- wccrawford 5y ago
  It's possible that those movies failed to advertise to the correct audience. If they pulled in people who thought they were getting 1 experience, but ended up with a different experience, the reviews are going to be lower than if they hit the right audience. Critics, as noted, see everything and so they wouldn't be affected by bad advertising. I've definitely been wrong, in both directions, about movies based on their advertising.
nopenopenopeno 5y ago
I used to be a film critic for the largest newspaper in a pretty big US city. Movie critics absolutely do not watch every movie. Not even close.
- rurban 5y ago
  I used to be a better professional film critic, mostly festival movies. movie critics do watch the best movies. eg on a festival with 100-250 movies they watch from 30 to 50 typically. from these 30-50 there are about 20 good movies, the best movies of the year. from these regular moviegoers watch 1-2. normal people do not watch the best but the worst with the biggest ad budget. of course critics don't watch every trash movie which starts every week. from 5-8 filmstarts per week, they watch 1-3. these weeks more, around 3, because now we have the best weeks of the year, with last year's Cannes movies arriving. and then again late fall, with the best Oscar movies being pushed. but eg this year's Oscars had only 2 good movies overall. for comparison: my letterboxd https://letterboxd.com/rurban/films/diary/ https://letterboxd.com/rurban/films/diary/
- Kranar 5y ago
  Fair point, it's categorically true that no critic watches every single movie released and I definitely overstated that. A brief search on Statista indicates that about 700 movies are released a year, the average American sees 2.3 movies a year (in theaters), and professional critics review over 200 movies a year, some even review 300. I maintain that given this disparity (2.3 movies a year versus 200 a year) my point still stands, critics will tend to watch movies regardless of any predisposition towards them whereas an audience member will be much more discriminating about which 2.3 movies they will pay to see.
dragontamer 5y ago
> Those researchers are typically the ones I hire. But there's no evidence that demonstrates either possibility (or if the reality is a 3rd, 4th or 5th possibility).
pie42000 5y ago
"A very small minority of UX Researchers come to the correct conclusion, which is that most professional critics have to review every single movie" Could you provide your sources? This seems like it could be very incorrect, or only partially correct. Keep in mind rottentomatoes is owned by movie studios so they have a vested interest in changing the formula for disregarding audience ratings. I think they already disregard ratings below a certain threshold and other weird shit.
davidktr 5y ago
Then how do you explain that there are systematic differences for certain film genres? The article has a graphic on it. The differences are by no means uniform. Crowds love Action and Thriller, critiques History and Documentary. D'oh. That is not to say there is no selection bias. But the population of critiques and film-going crowds are almost certainly different in education, lifestyle etc. That has certainly an effect on the observed discrepancies.
pie42000 5y ago
"The score, made up of audience ratings on a scale of 100, is calculated by taking the percentage of people who rated it at least 60 out of 100 (or 6 out 10) and multiplying it by 100%." The critics score is calculated with a different formula. Do you discuss this with candidates? Maybe research your interview questions a bit more before using them.
- Kranar 5y ago
  I absolutely do go over that, I also go over Metacritic and point out that this phenomenon applies to movies, video games, music, books, and almost all review systems that segregate professional reviews with customer reviews.
bluehorseray 5y ago
Calling that the "correct conclusion" is absurd. Whether or not a critic "wanted" to see a movie shouldn't really factor into the review at all. At the end of the day, the typical movie-goer is probably judging a film based off of an entirely different set of criteria (excitement / satisfaction) than a typical movie critic (overall artistic merit). Which isn't an exciting conclusion, but probably more correct. I would personally rate many movies I've seen lower if you told me to judge it from the perspective of a movie critic as opposed to a general audience member (The Purge for example).
lhorie 5y ago
> audience typically only goes to see movies that they are likely to enjoy That doesn't really explain the opposite phenomenon: movies that have higher critic rating than audience rating (e.g. movies like Mulan and Turning Red come to mind). IMHO, if you were to apply that rubric across all movies, the conclusion you should get is that critics and general audiences are simply different demographics and that it is normal for discrepancies to arise from sampling different demographics, not that general audiences are more biased towards what they like to watch. For example, parents wouldn't necessarily pick kids movies as their first choice of entertainment if age appropriate-ness wasn't a consideration (I think a lot of parents can relate to being sick of replaying Frozen, for example)
onlyrealcuzzo 5y ago
Also - a non-trivial percentage of ratings on Rotten Tomatoes & iMDB (and all review sites) come from accounts that have reviewed more movies than there is time in a human life to watch. The amount fake user reviews can skew a rating is not to be taken lightly - and IIUC - Rotten Tomatoes & iMDB might be filtering out some/a lot of spam - but definitely not even close to all of it. The same is true for GoodReads - the amount of teenage booktubers that have reviewed 10k+ books is awfully suspicious...
pjmlp 5y ago
As someone that enjoys traveling and testing food places, the same applies to food critics. Most of the time when I fall again into the trap of following their advices about some place, I come out disappointed, usually getting some creative dishes that still leave me wishing to go eat elsewhere. I have discovered so many nice places just by randomly crashing into them, where one could watch from outside people were enjoying being there, regardless of how many stars the food critic gave it.
bb88 5y ago
I think another acceptable explanation is that movies are marketed not to everyone but to certain people. So say the latest horror flick that comes around is marketed to teens and 20-somethings. The over-30 crowd gets turned off by violence and gore. But that particular demographic loves it, and the more gore the more they like it. There's a certain group of people that loved Human Centipede, e.g. But it got terrible reviews. Roger Ebert refused to award it stars, but gave it a few positive notes regardless. And in it he even said: I have long attempted to take a generic approach. In other words, is a film true to its genre and does it deliver what its audiences presumably expect? “The Human Centipede” scores high on this scale. It is depraved and disgusting enough to satisfy the most demanding midnight movie fan. And it's not simply an exploitation film. https://www.rogerebert.com/reviews/the-human-centipede-2010 https://www.rogerebert.com/reviews/the-human-centipede-2010
- mrob 5y ago
  On the other hand, Ebert followed the critical consensus and gave Freddy Got Fingered (my personal pick for funniest movie of all time) a zero star rating. This is a gross-out comedy that succeeds spectacularly at what it set out to do, but very few critics were willing to judge it by the standards of the genre. The audience scores are very polarized (currently 56% positive on RT), but critic's reviews are only at 11%.
  - bb88 5y ago
    How does that challenge the argument that movies are marketed to certain groups of people? Gross out movies don't typically get marketed to soccer moms.
drugstorecowboy 5y ago
Wouldn't movie critics tend to be people to like movies in general? What about critics that stick to their preferred genre? They might start out pre-disposed to liking movies which could wipe out the effect of your assertion completely. On one hand, I think there is a good chance you are at least partially correct, but declaring your offhand intuition about movie criticism industry to be the "correct" answer is at best hubris.
deleted 5y ago
[deleted]
_moof 5y ago
You really don't think film critics being different from the general population has anything do with it? They're trained to see things in movies that most people don't. I get that you want people to say "selection bias" but that's not even close to the only significant factor at play here. You have two populations, which is also a classic statistical phenomenon. You even get close to this in your discussion of selection bias—they watch a lot more movies, and are thus more experienced—but you don't seem to want to allow this, which seems odd.
mdoms 5y ago
I disagree completely with your conclusion and I offer the same level of evidence you did.
deleted 5y ago
[deleted]
- deleted 5y ago
  [deleted]
adfhdfhdryheryh 5y ago
No, the correct conclusion is "I don't know". There is no real evidence one way or the other. Their unfounded speculation about human nature is just as plausible as yours. In the interview you can generate a hypothesis, not a conclusion. This reminds me of the "why are manhole covers round" question, where people are judged by their ability to come up with clever wrong answers.
next_xibalba 5y ago
> most professional critics have to review every single movie This isn't true. Quite far from it, in fact. Manohla Dargis at the NY Times, for example, averages only about 2 reviews per week. There are many more releases than that. Peter Travers at Rolling Stone does even fewer. Same with Justin Chang at the LA Times. Etc, etc. IMO, the explanation is going to be multifactorial. Your "correct conclusion" probably explains some of the gap as critics are still obligated to review the prestige releases (even if they aren't necessarily interested in seeing them). But surely also does the divergence between the tastes of professional movie watchers and Joe Moviegoer explain the gap. What's interesting is that in over-indexing on one bias, you've probably introduced another into your own hiring process. It does seem as though picking people who are careful, independent thinkers could still yield good hires. In the end, if you get a good candidate, who cares? I'd rather be right for the wrong reasons than just being wrong.
- robocat 5y ago
  A comment on the article was: """ I reviewed films for the whole of the nineties for the BBC and local press in Manchester rarely missing any new releases. With this luxury and dream job though comes a responsibility and after a while it became obvious that I was suffering from Movie fatigue. It was easy to dismiss much of Hollywood’s fare as formulaic and worthless because we had seen too much and seen better. Some of the movies I panned were perfectly good films and many were extremely successful and popular with audiences, but for me they were dull, and despite fabulous production values left me cold. Mark Kermode (Film Critic BBC) was one of my contemporaries and he said: “You have to see everything so that when something comes along that is good you will really notice it.” Of course, this exacerbated the fatigue and often meant that we were too quick to judge, and after a while, very little appealed because we had set ourselves far too high a standard to be realistic about the film’s quality. If you look at the reviews in The Guardian, you will see more bad reviews that good. You might argue that a bad review is much better copy than a good review so there might be an editorial decision in there. To a certain extent Movie critics can be a poor parameter for quality and an injudicious contribution to its success or failure. After ten years of reviewing films, I knew it was time to stop. I wasn’t enjoying it and I was probably doing some disservice to the film industry. Less is so much more. Harry Stafford – The Manchester Film School """ And, perhaps Manohla Dargis and Peter Travers watch far more movies than they write reviews for.
_dain_ 5y ago
I wouldn't consider either of these to be the "correct" solution. You don't need to posit anything at all about how film criticism works. Occam's Razor suggests (ROT13): Vg'f whfg erterffvba gb gur zrna pbzovarq jvgu erfgevpgvba bs enatr. Vs lbh unir gjb pbeeryngrq inevnoyrf K naq L, naq lbh cvpx gur zvavzhz A fnzcyrf sebz K, cebonoyl gur pbeerfcbaqvat A fnzcyrf sebz L jvyy unir uvture crepragvyr enax. Lbh pna pbasvez guvf ol fvzhyngvba. Gura pbzovar jvgu erfgevpgvba bs enatr: vg'f vzcbffvoyr gb tb orybj 0% fpber, fb ybj pevgvp fpberf "cvyr hc" ng mreb. Gura jura lbh ybbx ng gur pbeerfcbaqvat nhqvrapr fpber, nal aba-mreb inyhrf ng nyy jvyy envfr gur zrna. Ol flzzrgel, jr fubhyq rkcrpg gur zbivrf jvgu uvturfg pevgvp fpberf gb unir ybjre nhqvrapr fpberf, naq ivpr irefn va obgu pnfrf sbe ybj/uvtu nhqvrapr fpber.
- robocat 5y ago
  Off topic question: Is there anyone here who reads raw ROT13?
  - sdiupIGPWEfh 5y ago
    A couple years ago, I could manage reading and writing ROT13 somewhat decently. I can still do it after some warm ups first. I can also do arbitrary Caesar cyphers in my head, but not fast enough to impress anyone. Memorizing the alphabet backwards helps, as does being able to visualize the numeric value of each letter. Embarrassingly, I'm actually rather slow at the simple mental arithmetic part.

by Kranar 5y ago

raincom 5y ago

Actually, both statements boil down to the same with little modifications: (a) being out of touch with the audience is same as (b) the audience typically only going to see movies that they are likely to enjoy. (a) is broader than (b), can subsume (b).

parkingrift 5y ago

Written another way... You interview people, ask them their opinions, and hire those that agree with your opinion. The most excellent way I've seen to hire a group of people biased toward a singular way of thinking.

yhoneycomb 5y ago

“I hire people who think exactly like I do”

tomrod 5y ago
When you have the right mental model, this is an appropriate approach. Not every mental model is correct, either locally to a domain or globally in all cases. 1 < 2, in almost all cases except for small Z_{0,1} or similar type cases where mod functions modify the space to something exotic or comparator functions are defined far out of the typical. If you think there is a case for 2 < 1, and you aren't appealing to something exotic, you have the wrong approach.
- _v7gu 5y ago
  Any mental model that assigns zero weight to the probability of being wrong is a wrong mental model. That said, biases arising from endogeneity might have negative effects too. You can't conclude a parameter should have a different/zero sign just because of endogeneity, you have to go fix your model, and re-estimate the parameters.
  - tomrod 5y ago
    > Any mental model that assigns zero weight to the probability of being wrong is a wrong mental model. This is true when there is uncertainty. It also doesn't connect to "I only hire people who think like I do", which is the context of my response, so I don't follow your point. Agreed on your other point that you need to instrument endogeneity. Another requirement is plausibility. Hookworm presence in Greenland should be uncorrelated to solar sunspots.

babypuncher 5y ago

I find it distressing how many people miss the obvious selection bias at play here.

vmception 5y ago

and whats the explanation for the opposite? when critics scores are much higher than audience scores? I assume "out of touch" but almost any explanation I think of and ridicule could be seen as a strawman argument, so I'll wait for your response.

joebob42 5y ago
I could imagine the answer being: People went to see these movies not because they wanted to per se but because critics said they were amazing. And because they went mostly due to critical reviews, anyone who didn't like it was even more likely than normal to be like "wait I feel tricked this was bad" This might be a bit of a stretch, but it seems almost plausible.
- dnissley 5y ago
  Two more possibilities: - A film may have been marketed as one particular type of movie via trailers (e.g. as a horror film) but was actually a different type of movie, often a subversion of the genre it was marketed as (e.g. ended up as social commentary). So audiences feel deceived and report that in their ratings. - Similarly, there are movies that are "applause lights" where critics are obliged to give positive ratings due to the social significance of a film (e.g. made by a woman / featuring a woman in a lead role). But the movie might just not be that good, and audiences report this in their ratings.
  - vmception 5y ago
    > social significance of a film (e.g. made by a woman / featuring a woman in a lead role) for a less divisive example: movies about the film or theatre industry get high marks and fit your example while the audience is left scratching their heads
    - ripe 5y ago
      Excellent point. Critics have often remarked that movies about Hollywood tend to get Oscars just because of the subject matter, which Academy members like. Also, a less cynical example: some movies are about something new: you might see people who live in a developing country, e.g., Somali fishermen in Captain Philips, or in unusual occupations (e.g., WWF wrestler in The Wrestler). Most audiences just want to relax and not learn something new, but critics love these kinds of movies.

omarhaneef 5y ago

Never thought of that but it makes a lot of sense! Edit: worth noting that this also creates an implicit prediction about the extent to which the advertising is an accurate representation of the movie. This could go in the opposite direction and audience scores would be lower than the critical assessment.

ashtonbaker 5y ago

That's a clever way of looking at it. Did you come up with it yourself? If not, I wonder what you think it says about a person that they were exposed to this idea before they interviewed with you, versus after.

klodolph 5y ago
IMO it seems obvious to me, but that might be because I took a statistics class in college. (Not “obvious in retrospect”—I’ve had a number of personal discussions with people where we talk about the difference between critic & user scores on Metacritic.)
- wccrawford 5y ago
  You also don't have any emotional or financial investment in the situation, though. ;)
  - klodolph 5y ago
    I don’t understand what you’re trying to say.
Kranar 5y ago
No I did not come up with it myself. It's a common pitfall in statistics known as selection bias [1]. My problem is that if I ask a researcher about selection bias, they have no problem defining it for me and even giving me a textbook example of it. But when I present to a researcher a real life scenario that is highly susceptible to selection bias, they will almost all fail to identify it. It's a problem where many people are able to compartmentalize knowledge but fail to actually make use of that knowledge in practice. As a run a quant firm I see this among so many researchers across many disciplines. They fully understand a concept when you ask them directly, but they fail to put those concepts to use outside of a very direct and artificial setting. [1] https://en.wikipedia.org/wiki/Selection_bias https://en.wikipedia.org/wiki/Selection_bias
- 5y ago

logicalmonster 5y ago

> come to the correct conclusion, What makes you so sure that this is the correct interpretation? Both things could be true at the same time: there could be selection bias at the same time as critics are perhaps out of touch and snobby. You could backtest this theory: see if perhaps critics scores were at one time a bit closer to general audiences or not. I think you'd find that at one time good critics were a bit closer to general audience scores.

handmodel 5y ago
Looking at the mechanisms alone I think its pretty unlikely that film critics have not shifted over time. The standard film critic job in the 80s/90s was a local newspaper. That meant your audience was men/women of various ages and political beliefs who happened to live in the same area. If you were a successful critic you had to write something that appealed to all of them. This was actually a vital service that people used to determine if a movie was good. Now the typical movie critic is online and has an audience which probably has a singular viewpoint. This isn't bad - but makes it unlikely the critics with "everyman" sensibilities would do well. People who consume an above average amount of film reviews will have their views represented more. Chuck Klosterman (who was once a local newspaper movie reviewer and now writes to a certain type of audience online) has written about this.
tshaddox 5y ago
It's also possible for critics to have systematically different preferences than movie-goers that don't deserve to be described pejoratively as "out of touch" or "snobby." In fact, that's precisely what I would expect and hope to see from people who are professional critics. I wouldn't want expect a list of best restaurants from a food critic to be identical to the list of top restaurant chains by revenue (1: McDonalds, 2: Starbucks, etc.).
- logicalmonster 5y ago
  > I wouldn't want expect a list of best restaurants from a food critic to be identical to the list of top restaurant chains by revenue That's a very good point that the role of the critic is to provide their own expertise and opinions on the overall subject. But the critic still has to be within the ballpark of popular tastes if the expectation is a mass audience that really trusts their recommendations, right? Many Americans might listen to a food critic that said that he thought some pasta dish at some Italian restaurant was the best, even if pasta is not currently everybody's favorite food. But if he he recommended boiled snails or braised calf brains, many people might ignore his judgement because it's a bit too far out of range of what they're comfortable with. At a certain distance from popular opinion, a mass audience might no longer trust that critic. Just a thought.

deleted 5y ago

[deleted]

fullshark 5y ago

Gotcha type interview questions with a single "trick" that you either get/have seen before or don't provide terrible signal imo.

thaway2839 5y ago
The comment literally states that it's one of the questions they ask, which is the exact opposite of a gotcha question. And the specific question can lead to a conversation, which may "reveal" the answer (obviously, I don't know whether this is the correct answer), but at the very least it reveals whether the interviewee thinks about selection bias, among other concerns.
- tshaddox 5y ago
  > The comment literally states that it's one of the questions they ask, which is the exact opposite of a gotcha question. Huh? "Gotcha question" doesn't imply it's a single question that's not part of a series of questions. A "gotcha question" is simply a question designed such that it's very easy to respond with something that sounds bad or is incorrect.
  - bee_rider 5y ago
    Is it really easy not to come up with the desired explanation here? It seems like the most obvious possible answer, that people tend to watch the types of movies that they like. Actually it seems to provide almost no information unless their hiring pool is mostly candidates who utterly lack common sense.
    - setr 5y ago
      I recall reading that someone found fizzbuzz (with nothing fancy) to be a highly effective filter for programming interviews Hiring pools tend to be filled with nonsense candidates

mikequinlan 5y ago

The article claims that the difference between audiences and critics has been increasing over the last two decades. If the difference was only because audiences self-selected while critics had to review everything, why would it be increasing over time?

KingOfCoders 5y ago
No clue, one moonshot people live more in a bubble and narrow their film selection?
V-2 5y ago
If anything, I'd expect that the audience would nown have more of a hit-and-miss experience than 20 years ago - with the abundance of streaming services at your fingertips, it's much easier to binge watch movies just because, whereas, say, having to rent a VHS tape had to bea more conscious and committal decision.

bufferoverflow 5y ago

If your narrative was true, we would always see critics scores lower than the audience (assuming statistically significant number of critics reviewing a given movie). But we see so many counterexamples with radically different ratings: Spy kids, Ad Astra, Noah, King Kong (2005), Babe, The Blair Witch Project, Chicken Run, and thousands of others.

UnpossibleJim 5y ago
Not to mention a number of films who stopped the audience scores, decrying "review bombing". Some of these films were fine, while some were objectively bad (-cough- Ghostbusters reboot -cough-). I have a feeling (though it is unsubstantiated rumor) that there are payoffs in the critics circles for certain movies.
wccrawford 5y ago
It's possible that those movies failed to advertise to the correct audience. If they pulled in people who thought they were getting 1 experience, but ended up with a different experience, the reviews are going to be lower than if they hit the right audience. Critics, as noted, see everything and so they wouldn't be affected by bad advertising. I've definitely been wrong, in both directions, about movies based on their advertising.

nopenopenopeno 5y ago

I used to be a film critic for the largest newspaper in a pretty big US city. Movie critics absolutely do not watch every movie. Not even close.

rurban 5y ago
I used to be a better professional film critic, mostly festival movies. movie critics do watch the best movies. eg on a festival with 100-250 movies they watch from 30 to 50 typically. from these 30-50 there are about 20 good movies, the best movies of the year. from these regular moviegoers watch 1-2. normal people do not watch the best but the worst with the biggest ad budget. of course critics don't watch every trash movie which starts every week. from 5-8 filmstarts per week, they watch 1-3. these weeks more, around 3, because now we have the best weeks of the year, with last year's Cannes movies arriving. and then again late fall, with the best Oscar movies being pushed. but eg this year's Oscars had only 2 good movies overall. for comparison: my letterboxd https://letterboxd.com/rurban/films/diary/ https://letterboxd.com/rurban/films/diary/
Kranar 5y ago
Fair point, it's categorically true that no critic watches every single movie released and I definitely overstated that. A brief search on Statista indicates that about 700 movies are released a year, the average American sees 2.3 movies a year (in theaters), and professional critics review over 200 movies a year, some even review 300. I maintain that given this disparity (2.3 movies a year versus 200 a year) my point still stands, critics will tend to watch movies regardless of any predisposition towards them whereas an audience member will be much more discriminating about which 2.3 movies they will pay to see.

dragontamer 5y ago

> Those researchers are typically the ones I hire. But there's no evidence that demonstrates either possibility (or if the reality is a 3rd, 4th or 5th possibility).

pie42000 5y ago

"A very small minority of UX Researchers come to the correct conclusion, which is that most professional critics have to review every single movie" Could you provide your sources? This seems like it could be very incorrect, or only partially correct. Keep in mind rottentomatoes is owned by movie studios so they have a vested interest in changing the formula for disregarding audience ratings. I think they already disregard ratings below a certain threshold and other weird shit.

davidktr 5y ago

Then how do you explain that there are systematic differences for certain film genres? The article has a graphic on it. The differences are by no means uniform. Crowds love Action and Thriller, critiques History and Documentary. D'oh. That is not to say there is no selection bias. But the population of critiques and film-going crowds are almost certainly different in education, lifestyle etc. That has certainly an effect on the observed discrepancies.

pie42000 5y ago

"The score, made up of audience ratings on a scale of 100, is calculated by taking the percentage of people who rated it at least 60 out of 100 (or 6 out 10) and multiplying it by 100%." The critics score is calculated with a different formula. Do you discuss this with candidates? Maybe research your interview questions a bit more before using them.

Kranar 5y ago
I absolutely do go over that, I also go over Metacritic and point out that this phenomenon applies to movies, video games, music, books, and almost all review systems that segregate professional reviews with customer reviews.

bluehorseray 5y ago

Calling that the "correct conclusion" is absurd. Whether or not a critic "wanted" to see a movie shouldn't really factor into the review at all. At the end of the day, the typical movie-goer is probably judging a film based off of an entirely different set of criteria (excitement / satisfaction) than a typical movie critic (overall artistic merit). Which isn't an exciting conclusion, but probably more correct. I would personally rate many movies I've seen lower if you told me to judge it from the perspective of a movie critic as opposed to a general audience member (The Purge for example).

lhorie 5y ago

> audience typically only goes to see movies that they are likely to enjoy That doesn't really explain the opposite phenomenon: movies that have higher critic rating than audience rating (e.g. movies like Mulan and Turning Red come to mind). IMHO, if you were to apply that rubric across all movies, the conclusion you should get is that critics and general audiences are simply different demographics and that it is normal for discrepancies to arise from sampling different demographics, not that general audiences are more biased towards what they like to watch. For example, parents wouldn't necessarily pick kids movies as their first choice of entertainment if age appropriate-ness wasn't a consideration (I think a lot of parents can relate to being sick of replaying Frozen, for example)

onlyrealcuzzo 5y ago

Also - a non-trivial percentage of ratings on Rotten Tomatoes & iMDB (and all review sites) come from accounts that have reviewed more movies than there is time in a human life to watch. The amount fake user reviews can skew a rating is not to be taken lightly - and IIUC - Rotten Tomatoes & iMDB might be filtering out some/a lot of spam - but definitely not even close to all of it. The same is true for GoodReads - the amount of teenage booktubers that have reviewed 10k+ books is awfully suspicious...

pjmlp 5y ago

As someone that enjoys traveling and testing food places, the same applies to food critics. Most of the time when I fall again into the trap of following their advices about some place, I come out disappointed, usually getting some creative dishes that still leave me wishing to go eat elsewhere. I have discovered so many nice places just by randomly crashing into them, where one could watch from outside people were enjoying being there, regardless of how many stars the food critic gave it.

bb88 5y ago

I think another acceptable explanation is that movies are marketed not to everyone but to certain people. So say the latest horror flick that comes around is marketed to teens and 20-somethings. The over-30 crowd gets turned off by violence and gore. But that particular demographic loves it, and the more gore the more they like it. There's a certain group of people that loved Human Centipede, e.g. But it got terrible reviews. Roger Ebert refused to award it stars, but gave it a few positive notes regardless. And in it he even said: I have long attempted to take a generic approach. In other words, is a film true to its genre and does it deliver what its audiences presumably expect? “The Human Centipede” scores high on this scale. It is depraved and disgusting enough to satisfy the most demanding midnight movie fan. And it's not simply an exploitation film. https://www.rogerebert.com/reviews/the-human-centipede-2010 https://www.rogerebert.com/reviews/the-human-centipede-2010

mrob 5y ago
On the other hand, Ebert followed the critical consensus and gave Freddy Got Fingered (my personal pick for funniest movie of all time) a zero star rating. This is a gross-out comedy that succeeds spectacularly at what it set out to do, but very few critics were willing to judge it by the standards of the genre. The audience scores are very polarized (currently 56% positive on RT), but critic's reviews are only at 11%.
- bb88 5y ago
  How does that challenge the argument that movies are marketed to certain groups of people? Gross out movies don't typically get marketed to soccer moms.

drugstorecowboy 5y ago

Wouldn't movie critics tend to be people to like movies in general? What about critics that stick to their preferred genre? They might start out pre-disposed to liking movies which could wipe out the effect of your assertion completely. On one hand, I think there is a good chance you are at least partially correct, but declaring your offhand intuition about movie criticism industry to be the "correct" answer is at best hubris.

deleted 5y ago

[deleted]

_moof 5y ago

You really don't think film critics being different from the general population has anything do with it? They're trained to see things in movies that most people don't. I get that you want people to say "selection bias" but that's not even close to the only significant factor at play here. You have two populations, which is also a classic statistical phenomenon. You even get close to this in your discussion of selection bias—they watch a lot more movies, and are thus more experienced—but you don't seem to want to allow this, which seems odd.

mdoms 5y ago

I disagree completely with your conclusion and I offer the same level of evidence you did.

deleted 5y ago

[deleted]

deleted 5y ago
[deleted]

adfhdfhdryheryh 5y ago

No, the correct conclusion is "I don't know". There is no real evidence one way or the other. Their unfounded speculation about human nature is just as plausible as yours. In the interview you can generate a hypothesis, not a conclusion. This reminds me of the "why are manhole covers round" question, where people are judged by their ability to come up with clever wrong answers.

next_xibalba 5y ago

> most professional critics have to review every single movie This isn't true. Quite far from it, in fact. Manohla Dargis at the NY Times, for example, averages only about 2 reviews per week. There are many more releases than that. Peter Travers at Rolling Stone does even fewer. Same with Justin Chang at the LA Times. Etc, etc. IMO, the explanation is going to be multifactorial. Your "correct conclusion" probably explains some of the gap as critics are still obligated to review the prestige releases (even if they aren't necessarily interested in seeing them). But surely also does the divergence between the tastes of professional movie watchers and Joe Moviegoer explain the gap. What's interesting is that in over-indexing on one bias, you've probably introduced another into your own hiring process. It does seem as though picking people who are careful, independent thinkers could still yield good hires. In the end, if you get a good candidate, who cares? I'd rather be right for the wrong reasons than just being wrong.

robocat 5y ago
A comment on the article was: """ I reviewed films for the whole of the nineties for the BBC and local press in Manchester rarely missing any new releases. With this luxury and dream job though comes a responsibility and after a while it became obvious that I was suffering from Movie fatigue. It was easy to dismiss much of Hollywood’s fare as formulaic and worthless because we had seen too much and seen better. Some of the movies I panned were perfectly good films and many were extremely successful and popular with audiences, but for me they were dull, and despite fabulous production values left me cold. Mark Kermode (Film Critic BBC) was one of my contemporaries and he said: “You have to see everything so that when something comes along that is good you will really notice it.” Of course, this exacerbated the fatigue and often meant that we were too quick to judge, and after a while, very little appealed because we had set ourselves far too high a standard to be realistic about the film’s quality. If you look at the reviews in The Guardian, you will see more bad reviews that good. You might argue that a bad review is much better copy than a good review so there might be an editorial decision in there. To a certain extent Movie critics can be a poor parameter for quality and an injudicious contribution to its success or failure. After ten years of reviewing films, I knew it was time to stop. I wasn’t enjoying it and I was probably doing some disservice to the film industry. Less is so much more. Harry Stafford – The Manchester Film School """ And, perhaps Manohla Dargis and Peter Travers watch far more movies than they write reviews for.

_dain_ 5y ago

I wouldn't consider either of these to be the "correct" solution. You don't need to posit anything at all about how film criticism works. Occam's Razor suggests (ROT13): Vg'f whfg erterffvba gb gur zrna pbzovarq jvgu erfgevpgvba bs enatr. Vs lbh unir gjb pbeeryngrq inevnoyrf K naq L, naq lbh cvpx gur zvavzhz A fnzcyrf sebz K, cebonoyl gur pbeerfcbaqvat A fnzcyrf sebz L jvyy unir uvture crepragvyr enax. Lbh pna pbasvez guvf ol fvzhyngvba. Gura pbzovar jvgu erfgevpgvba bs enatr: vg'f vzcbffvoyr gb tb orybj 0% fpber, fb ybj pevgvp fpberf "cvyr hc" ng mreb. Gura jura lbh ybbx ng gur pbeerfcbaqvat nhqvrapr fpber, nal aba-mreb inyhrf ng nyy jvyy envfr gur zrna. Ol flzzrgel, jr fubhyq rkcrpg gur zbivrf jvgu uvturfg pevgvp fpberf gb unir ybjre nhqvrapr fpberf, naq ivpr irefn va obgu pnfrf sbe ybj/uvtu nhqvrapr fpber.

robocat 5y ago
Off topic question: Is there anyone here who reads raw ROT13?
- sdiupIGPWEfh 5y ago
  A couple years ago, I could manage reading and writing ROT13 somewhat decently. I can still do it after some warm ups first. I can also do arbitrary Caesar cyphers in my head, but not fast enough to impress anyone. Memorizing the alphabet backwards helps, as does being able to visualize the numeric value of each letter. Embarrassingly, I'm actually rather slow at the simple mental arithmetic part.