13 ms·

Scraping Roger Ebert’s reviews and finding his favorite movies on Amazon Prime

197 points by catwind7 6y ago

s1artibartfast 6y ago
Useful information but the results look like garbage in Chrome and IE. The multi-line titles all run together and would greatly benefit from a table outline.
- elchief 6y ago
  there's a gdoc spreadsheet link at the end
- adamzegelin 6y ago
  Safari too. I just discovered that modern user agents don't include table borders by default in their stylesheets -- at least, choosing Develop -> Disable Styles didn't make the table borders appear. I'm having flashbacks to the chiseled borders of Netscape.
sonofgod 6y ago
If you need a more stateful version of requests: import requests session = requests.session() # now use session like you would requests session.get("http://httpbin.org/cookies/set/name/value") print(session.get("http://httpbin.org/cookies").content)
- catwind7 6y ago
  oh I need to try that - I had this feeling that there was a more stateful version but for ..... some reason ... reaching for a new dependency felt easier at the time haha. Thanks
duxup 6y ago
I miss Roger Ebert. I honestly watch fewer films since his death. Maybe this list will change that. Also "Spirited Away" is on the list but doesn't seem to be included with prime.
- catwind7 6y ago
  updated the list. Thanks for the catch! sorry about that. I'm using an `in` keyword which just checks substrings ... that I meant to change. Woops
- vincentmarle 6y ago
  Spirited Away is now on HBO Max if you're looking for it
Alex3917 6y ago
FWIW the link for "The Bicycle Thief" (1949) links to "The Bicycle" (2015). Great list though. It would be interesting to see the thing done for the AFI and BFI top 100. (Although I suspect that most movies on the AFI are probably already on Ebert's list.)
intellijdd 6y ago
I was just seeing someone else's tutorial on scraping Amazon prices. They also ran into an issue where they needed to scrape twice instead of once. Not sure that it's the same issue you're facing but I thought I might drop my two cents.
- catwind7 6y ago
  I actually did notice that issue, even with using a stateful client like mechanize. Sometimes I had to scrape > 5 times in order to get through the "anti robot" page. Other times, I get no issue at all. It's weird - maybe they're doing some pattern matching on request metadata on their end?
exhilaration 6y ago
If the author reads this, the link in the Google sheet to the Battle of Algiers goes to Algiers, a totally different movie.
- catwind7 6y ago
  i'm fixing this list ... just realized a few bugs in the code. sigh
catwind7 6y ago
Author here. The list (both on my post + the google sheet) should be correct now - I underestimated how many different releases of the SAME movie title there are ... Thanks all for catching the mistakes.
- jchazin 6y ago
  FYI - the "Review URL" links at the bottom don't seem to be working properly. e.g. for "Moonstruck", the link leads to: "www.linisnil.com/articles/scraping-roger-ebert-reviews-and-amazon/www.rogerebert.com/reviews/great-movie-moonstruck-1987" instead of: "www.rogerebert.com/reviews/great-movie-moonstruck-1987"
  - catwind7 6y ago
    hey thanks for catching that - turns out leaving out the protocol in markdown causes makes it a relative url. fixed.
trimbo 6y ago
"The Man Who Shot Liberty Valance" IMO, John Ford's best movie, hands down. Unfortunately, it is not actually available on Prime without "CBS All Access" [Edit: ah, I see that this is not just "Included with prime", but all movies]
- dmix 6y ago
  Prime just recently introduced channels, which dramatically increased the amount of content available, but each is $3.99. I personally love this because I’d rather have the option to subscribe to Smithsonian content or MGMs back catalog using the same streaming service I already use, rather than paying to use 10 different or getting stuck using the small list of (mostly old, TV movie, or B movie) content on just prime or Netflix. So this is a feature, not a bug. Mostly because I’ve accepted how backwards the movie industry is going to be with copyright. But still having the option is the less evil. That said, Maybe the UI can make this more obvious?
  - joezydeco 6y ago
    Prime has had channels for almost five years now.
  - trimbo 6y ago
    I typically use Google to search which streaming service has a movie, and for The Man Who Shot Liberty Valance, Prime says "Subscribed" underneath it.
  - wolco 6y ago
    First time surprised me. I can see it being useful when you have the channel. Or as a separate tab when you want to possible add a channel. It kinda taints the other content. You ask internal.. Is it really available without subscribing? to channels you have access to. Then you finally click and they have season 1 and 3 and 5 of some show. Terrible UI. Feels like you get so much less than you do. With netflix it feels like you could just fall into a series immediately.
jamesrcole 6y ago
On a tangent: I dream of a web where whenever there are sets of items (eg eberts-great-movies, and movies-on-amazon) you can easily apply set operations (like intersection) on them (so if ‘n’ stands for ‘intersection’, eg eberts-great-movies n movies-on-amazon). So, in effect if you’re on a site that deals with a set of items, like the amazon prime movies, you can tell the browser to intersect this set with a different set at another URL. I understand that doing this would require the right ‘infrastructure’ to be in place.
- castratikron 6y ago
  I don't know why there's no good way to know what's streaming on a certain service. Seems like the only way is through third party sites that are probably made up of people manually adding to the list. If everyone is going to make their own streaming service then there needs to be a standard interface to make them collectively easier to use. A "guide", if you will.
  - pjc50 6y ago
    Unfortunately their profit incentive is the opposite: to train you to watch what they choose to recommend, rather than take a step back and look at what isn't there. See the hollowing of the Netflix catalog.
  - Exmoor 6y ago
    There was briefly an MPAA backed service for this whose name is escaping me, which I used quite a bit. Unfortunately it fairly quickly devolved into only giving links to services where you had to pay per-movie to watch.
- theshrike79 6y ago
  Ironically the pirate movie scene has tools like this[1][2], downloading new stuff based on certain rules is completely automated. All the paid options hold on to their APIs way too tight to let anything like this to be viable. I'd love to have a service that would cross-reference my queue of movies and TV shows and tell me which service has the most matches so that I'd know which one to pay for each month. [1] https://radarr.video https://radarr.video [2] https://couchpota.to https://couchpota.to
dvt 6y ago
Roger Ebert had an unexpected impact on me in my 20s. In my late teens, I started religiously reading his reviews, and this continued until his death. I've never really been a "movie buff," but Ebert's witty prose and pointed technical critique reminded me of Scalia (whom I also loved reading). Thank you for putting this list together!
- autarch 6y ago
  Besides being a well respected film critic and incredibly knowledgeable about film, he was also just a great writer. I read his reviews every week simply for the pleasure of reading them, even if I wasn't interested in the films he reviewed.
- Mediterraneo10 6y ago
  The tragedy of Roger Ebert is that, although he had a vast knowledge of great cinema, he established a niche where he was expected to mainly write about ephemeral popular films. His time was mostly spent on Hollywood blockbusters and popcorn, and he was appearing in media where he could not go into any great depth due to space limitations, or because that would be a turnoff for his mass audience. So, only in a few Ebert productions like his "Great Movies" books can one get a sense of the films that really mattered to the man and to art. Compare this to a critic like Richard Brody, who in his career has been fortunate to focus entirely on art cinema (though of course Brody’s net worth is probably an order of magnitude less than Ebert’s was).
  - ctchocula 6y ago
    I've only read a few Ebert reviews, enjoyed them and happened to be looking for a book to read, so I appreciate the recommendation! edit: The introduction alone makes the book worth reading: > Of all the arts, movies are the most powerful aid to empathy, and good ones make us into better people. Not many of them are very good.
- WalterBright 6y ago
  I enjoyed Ebert when he teamed with Siskel in the early days before they became great friends, and they'd argue. Their debates were better than the movies :-)
boomboomsubban 6y ago
Only three movies in the past fourty years, non past 87, is surprising. Is Prime that full of older content or did Ebert's recommendations just stop coming from major studios?
- zucker42 6y ago
  Using https://www.rogerebert.com/great-movies https://www.rogerebert.com/great-movies you can filter by date
  - boomboomsubban 6y ago
    Thanks, but that doesn't explain why Prime doesn't have any released after 87.
    - blaser-waffle 6y ago
      Presumably because contracts with studios won't let them stream those. Why won't they get streamed? 1980 is 40 years ago, and movies pushing 40+ years old aren't in the day-to-day zietgiest and aren't getting a lot of searches. Newer movies still catch eyeballs occasionally. I'd bet '87 is where the long-tail drops off noticeably.
      - boomboomsubban 6y ago
        It's been a while since I've looked at Prime's catalog, but is it really exclusives plus fourty year old content? I understand why those distribution rights are more valuable, but that is also why Amazon would desire them more.
    - ThePadawan 6y ago
      I'm having a hard time coming up with the data to confirm this theory but 1986/1987 seems to me to be the peak of both VHS sales and the bottom of VCR prices. I assume from that many studios had to come up with their first licensing schemes for movies "for home use" (contrary to movie theater and broadcast use), which could potentially still apply to (and restrict) streaming. Once again, this is my personal hypothesis (and I'd be happy to see some data to support or contradict it).
gittes 6y ago
Don't mean to undermine this guy's screen-scraping adventure... but if you want to use something that will tell you all the streaming services that have the list of movies, you can use: https://letterboxd.com/dvideostor/list/roger-eberts-great-movies/ https://letterboxd.com/dvideostor/list/roger-eberts-great-mo... You can look at each movie to see what streaming service it's on one at a time for free. If you have a pro paid account, you can even do: https://letterboxd.com/dvideostor/list/roger-eberts-great-movies/on/amazon-prime-us/ https://letterboxd.com/dvideostor/list/roger-eberts-great-mo... Which shows that there are 39 movies in Amazon Prime US from Ebert's "Great Movies," not 21 like this guy's spreadsheet says. To be fair, the exercise was to scrape the reference sources... so it might just need some refinement. Need to double check though if both lists are correct, only confirmed number totals. Full disclosure: That letterboxd list is not mine, I just found it
- js2 6y ago
  FWIW, I screen scraped rogerebert.com and copied all of his ratings and an excerpt of every review to letterboxd: https://letterboxd.com/re2/ https://letterboxd.com/re2/ Just the great movies: https://letterboxd.com/re2/tag/great-movie/films/by/release-earliest/ https://letterboxd.com/re2/tag/great-movie/films/by/release-... You can then filter those by streaming service, but you need a pro account. Looks like 38 movies: https://letterboxd.com/re2/tag/great-movie/films/on/amazon-prime-us/by/release-earliest/ https://letterboxd.com/re2/tag/great-movie/films/on/amazon-p... https://ibb.co/KFSj9jg https://ibb.co/KFSj9jg Apparently I missed the Buster Keaton movies: https://www.rogerebert.com/reviews/great-movie-the-films-of-buster-keaton https://www.rogerebert.com/reviews/great-movie-the-films-of-... https://letterboxd.com/director/buster-keaton/ https://letterboxd.com/director/buster-keaton/ But that means 39 isn't quite right either since Ebert is saying all of Buster Keaton's films are great. Anyway, the scraping was easy. The harder problem was parsing the html reviews (even with BeautifulSoup, the html is a mess), and then matching the reviews on Ebert's site to the correct movie, which I did via queries to tmdb and a lot of heuristics. There's nearly 8000 reviews and many have wrong years, bad titles, etc on rogerebert.com. It was a fun spare time project for a couple weeks.
  - gittes 6y ago
bogomipz 6y ago
Although it's not listed in this post because it's a rental. I feel like it deserves a mention nonetheless. "Beyond the Valley of the Dolls" is a 1960s cult classic. It was directed by Russ Meyer and written by Roger Ebert. There's a link to it on Roger's site as well as available to stream from Amazon for cheap: https://www.rogerebert.com/reviews/beyond-the-valley-of-the-dolls-1980 https://www.rogerebert.com/reviews/beyond-the-valley-of-the-...
DarknessFalls 6y ago
I think this is referring to his list of "Movies You Must See Before You Die". On that note, I should mention "Gates of Heaven", which is part of that list. A compelling documentary on animal cemeteries, by Errol Morris. https://www.rogerebert.com/reviews/great-movie-gates-of-heaven-1978 https://www.rogerebert.com/reviews/great-movie-gates-of-heav...
Mediterraneo10 6y ago
It is the fact that we have a relatively agreed-upon canon of great films (with immense re-watch value over the years) that keeps me torrenting instead of streaming. Sure, you could search from one streaming site to another for a given film, but the film may be either completely unavailable, or it may be offered one moment and the mysteriously removed the next. Meanwhile, some torrent communities are run by ardent cinephiles and they have all of these films, and once you have downloaded it, you can go back to it whenever you want.
- kjakm 6y ago
  You can also solve the “what streaming service” issue by buying the movie on DVD or Blu Ray allowing you to go back to it whenever you want. Additionally if you’re ok with DRM iTunes has most stuff. Useful tip for finding which streaming service a film is on: google “film name streaming” and google lists the answers in a box at the top of the results.
  - mywacaday 6y ago
    googling for film name streaming doesn't work, most of the results are specific to the USA instance of Netflix/Prime, not all content is available globally. Most of the websites that list content are spammy looking.
    - kjakm 6y ago
      I put my country name on the end of my query. It is a bit hit or miss though.
  - Mediterraneo10 6y ago
    The problem with recommending that people buy the DVD/Blu-Ray is that the agreed-upon canon of great films stretches into the many hundreds of titles (Ebert’s list at 364 titles is only an abridged canon.) Therefore, for someone seeking a cultural education, buying the physical releases would run into the thousands and thousands of dollars. It would be a challenge even for North Americans or Western Europeans to amass a suitable collection, let alone people in lower-income countries. Consequently, a person can only realistically learn about great cinema by streaming from multiple subscription sites, or torrenting, and people disappointed by the former ought to remember that the latter exists.
jyriand 6y ago
I guess this list of movies applies to people who live in US. Available movies differ substantially between countries. With Netflix it's easy, it doesn't even show the movie you can't watch, with Amazon you have to click through every movie you are interested in to see the screen that says that this movie is not available in your region. There are even series where only one or two episodes are unavailable.
NicoJuicy 6y ago
I cancelled my Amazon prime subscription, because of the ridiculous things they did in France. And following the dark pattern path to unsubscribe. I'm happy that I did it. Fyi, I'm from Belgium
- jrib 6y ago
  I didn't hear about this. Can you give a quick summary of what actions Amazon took in France that you are referring to?
Curnee 6y ago
Self-promotion: A number of years ago I made a little website which links you to a random review written by Roger Ebert. It isn't the cleanest of implementations, but it did what I needed it to do when I built it. http://randomebert.com/ http://randomebert.com/
greggman3 6y ago
I'm surprised not to see more people adding their favorite movies. Just sticking to things semi-related. I may have to go watch "My Man Godfrey" again but at least for me "The Thin Man" and its 5 sequels stuck with me where as though I watched "My Man Godfrey" years ago but I don't remember any details of it. (same lead actor and he pretty much always plays the same type of person) "The Man Who Shot Liberty Valance" is a good movie but if you want some other westerns starting James Stewart I can recommend "Destry Rides Again" which is not super serious but I found it throughly entertaining. Also "Broken Arrow" (1950), (not the 90s John Woo movie which is a fun popcorn movie). Seeing "The Good, the Bad, and the Ugly" on the list I recently saw "The Outlaw Josey Wales" which I enjoyed (also a Western with Clint Eastwood) Of the movies on that list I've seen the one I'd recommend the most is "Women in the Dunes". I don't want to over hype it I've seen most of the movies on the list and they are all great stories but "Woman in the Dunes" might give you something new to think about where as the others are mostly just great entertainment.
- patrec 6y ago
  In addition to “Woman in the Dunes”, I'd also nominate “The Gospel According to Saint Matthew”. And, in fact, “It's a Wonderful Life” for those who haven't seen it (it's a much more grown up and dark film than you may have been led to believe).
- gumby 6y ago
  Other good Jimmy Stewart westerns are "The Man from Laramie" and "Winchester 73"
cm2187 6y ago
A weekend project idea for geeks like me who like films, but have the feeling to have already watched everything. I find that imdb ratings have a high (but not 1.0) correlation with me liking the movie (provided I like the genre). You can still download from imdb flat files that contain all movies, ratings, as well as cast/directors/producers/writers. Stick that in a database, with a basic UI to hide movies you have already watched. And you can make a good personal recommandation engine for movies you didn’t suspect exist. The power of this approach is that imdb is pretty much an exhaustive catalog.
- arethuza 6y ago
  I tend to find that the movies I really like have a strongly bimodal distribution of comments on IMDB - some love and some hate. e.g. "Blood Machines" - which I can see why a lot of people dislike it but I think it is awesome. Edit: Probably not a great idea to search for it if you have a problem with nipples.
  - dfxm12 6y ago
    Do the movies you really dislike have the same distribution of comments? :) I do think that that there is something to media that creates a "love it or hate it" response. It's usually trying something new (Blood Machines certainly was). p.s.: I saw Blood Machines when it premiered on Shudder & really enjoyed it.
    - arethuza 6y ago
      Good question - I checked IMDB for Force Majeure and it does seem to have that "love it or loathe it" distribution - I tend to agree with the comment about it being "mind-numbingly pretentious drivel" (which I guess some people would apply to Blood Machines).
- hhmc 6y ago
  > I find that imdb ratings have a high (but not 1.0) correlation with me liking the movie You could track your own personal rating of the movies, then regress these ratings against a variety of external rating sources (IMDB, ebert, etc) to see which is the most predictive.
chillee 6y ago
Self-promotion (of sorts), but I (with some friends) have been watching lots of movies and keeping globally ordered rankings on github here: https://github.com/chillee/movierankings https://github.com/chillee/movierankings I find globally ordered rankings of movies to be an interesting exercise of consistency.
sfaruque 6y ago
There used to be a site call ClerkDogs.com that probably had the best movie recommendation system I've used. You started off naming a few movies you liked, and it would provide a list of movies you'd also probably like, and it was very accurate. From what I remember, the database was cataloged and maintained by actual humans, and not some algorithm following behavior patterns.
- fastball 6y ago
  Yep, jinni.com did the exact same thing, and I loved it. Unfortunately, it seems like B2C just didn't work financially so they switched to an entirely B2B model to help providers with their recommendation engines and no longer have their data accessible to end-users.
  - cpach 6y ago
    It’s a shame that the economics of recommendation engines doesn’t seem to work very well in the B2C space. Good rec. engines can be very useful.
- hanklazard 6y ago
  https://www.criticker.com/ https://www.criticker.com/ This does almost exactly what you describe. I really like it. Not human curated but matches your preferences with other humans with similar preferences to give you recs.
lostgame 6y ago
I’d like to see this for more services. Amazon Prime has IMHO the worst UI/UX I have used in a streaming service.
- TechBro8615 6y ago
  Agreed. What’s most annoying is how they deal with geo blocked content. They don’t tell you it’s blocked until you start watching it. So you browse the catalog for five minutes, finally find a movie to watch, press play, and then get an error message.
  - glenneroo 6y ago
    Or how they sometimes won't allow you to watch the original English version because you're not in an English speaking country. Even if your language is set to English (of which around 1/4 is still not translated). All movie previews are in native language which necessitates a visit to YT or IMDB. Even non-prime content is affected, just recently I paid 2.99 to watch some cheesy 90s movie with my friend who had always wanted to see this "classic". The syncing was abysmal and resolution 480p i.e. 720x480 i.e. DVD. After we were done, I noticed there were a handful of HD torrents in various languages. Reminded me immediately of Gabe Newell's thoughts on pirating games: "Piracy is almost always a service problem and not a pricing problem".
xtiansimon 6y ago
Are these free on Amazon or is this just an advertisement for Amazon Prime (or does everyone have a subscription but me?)
- catwind7 6y ago
  Not an advertisement for amazon prime :) These are free with prime, not free for non-subscribers I'm actually going to be cancelling my prime membership, which is another reason I wanted to see what I could watch for "free".
- wolco 6y ago
  Everyone had a subscription at one point not so sure anymore.
alexilliamson 6y ago
Side note: one excellent film that Roger Ebert didn't review is Life Itself, the documentary about his life that came out right after he died. It's full of joy and heartbreaking at times, but it really solidified his place of most universally relatable movie critic in my mind.
- catwind7 6y ago
  thank you for this. We will be checking this out!
cosmiccatnap 6y ago
Im not a fan of his reviews, they are sporatic and someone phoned in. Now Lenard Malton...
gabrielsroka 6y ago
See also "Where to Find Roger Ebert’s Great Movies Streaming" [0] which has US listings for Netflix, Amazon Prime, Hulu, Disney Plus, Criterion Channel, Kanopy, HBO, Starz, and Showtime as of March 2020. [0] https://www.rogerebert.com/features/where-to-find-roger-eberts-great-movies-streaming https://www.rogerebert.com/features/where-to-find-roger-eber...
samteeeee 6y ago
Folks interested in this topic might like my new side-project: get an email when your favorite director releases a new movie - https://directoralerts.website https://directoralerts.website
staycoolboy 6y ago
Thank you for doing this. I wish "top critics" was a search option on Prime (but their Roku interface is just abhorrent). There was a time when I hoped Netflix would do something similar: click here to see your favorite critics film list, but since Netflix has lost the streaming rights to soooo many films compared when they first started streaming ca. 2007, this is not doable. (Even DVD rights in most cases: films that are in my "Movies You Rated" queue from 2003 are no longer in Netflix's library.)
- ddrt 6y ago
  They have a really poor UX imo. The categories are seemingly arbitrary, and they have niche and seasonal ones that don’t make sense. During the December holidays they didn’t have a holiday themed category until near holiday end. Further, they make “innovation” like auto play when nobody asked for this, and without any way to stop it (unless you go to desktop and disable it deep in your settings). When searching you will receive similar results for titles they used to have... well why not state “are you searching for x? It’s no longer available, here are some recommendations” so the user isn’t wasting time wondering if the search feature is just “bad”. In the world of constant, unannounced, and live experimenting on huge user bases a little messaging goes a long way (with the comparison being nothing)
cottager 6y ago
When you said you were faking a proper agent with `requests`, do you mean you were setting the headers to look like a browser, as in here?: https://stackoverflow.com/questions/27652543/how-to-use-python-requests-to-fake-a-browser-visit https://stackoverflow.com/questions/27652543/how-to-use-pyth... That was going to be my suggestion for how to get around the anti-robot responses.
- catwind7 6y ago
  yeah so I have a user-agent generator that uses a random (valid) agent for each request. that, combined with cookies enabled, still didn't seem enough to get around the anti-robot barrier 100% of the time. 20% of the time, I get stopped. My guess is that they must do some kind of pattern matching on their end that kicks for some sets of requests but not others? But maybe it's not that sophisticated and I'm just missing something else in my headers
- x3blah 6y ago
  No agent at all is required. I got past the anti-robot response using no user-agent header and a simple delay.
youngamerican 6y ago
Reading Ebert's Great Movies site was hugely formative for me. I also love that he's low-key one of the best writers about addiction and recovery. Side note: A colleague who worked at the Sun-Times when Ebert also did recently told me about how whenever management threatened cuts, he'd come into the newsroom to throw his weight around against it. Even when he was getting to be in ill health. Much respect.
ngcc_hk 6y ago
Minor instruction to make it work for me: pip3 install BeautifulSoup pip3 install mechanize mkdir data Guess you can check data directory but not sure about the pip3, python does not like R I am not sure and can it pip3 install when not exist ....
x3blah 6y ago
Instead of using Python, here is a solution that only requires sh, curl, sed, sort, uniq and grep. This solution uses a generous 87s delay to retrieve the Amazon pages. There are 328 films listed as "great movies" on rogerebert.com. As such, the script, named "1.sh", needs 8h to complete, e.g., the time while you are at work or sleeping. No cookies, no state, no problems. Usage: sh -c 1.sh > 1.html Open 1.html in a browser and it shows whether each "great movie" is available as Prime Video or whether it is only available in some other format, such as Blu-ray, DVD, Multi-format, Hardcover. A link to the item on Amazon is provided. #!/bin/sh curl -HUser-Agent: -H'Accept: application/json' --compressed 'https://www.rogerebert.com/great-movies/page/[1-16]?utf8=%E2%9C%93&filters%5Btitle%5D=&sort%5Border%5D=newest&filters%5Byears%5D%5B%5D=1914&filters%5Byears%5D%5B%5D=2020&filters%5Bstar_rating%5D%5B%5D=0.0&filters%5Bstar_rating%5D%5B%5D=4.0&filters%5Bno_stars%5D=1'|grep -o "/reviews/great-movie-[^\\]*"|sed 's/.reviews.great-movie-//'|sort|uniq|while read x;do y=$(echo $x|sed 's/-/+/g');echo $x;curl -s --compressed -HUser-Agent: https://www.amazon.com/s/?k=$y 2>/dev/null|grep -m1 -C4 a-link-normal.a-text-bold;sleep 87;done|sed '/^[^< ]/s/.*/@&,/;1s|.*|<base href=https://www.amazon.com />&|;s/ *//;/^$/d;/^[@<]/!s|$|</a>|;1s/@//;s/@/<br>/'

197 points by catwind7 6y ago

Useful information but the results look like garbage in Chrome and IE. The multi-line titles all run together and would greatly benefit from a table outline.

elchief 6y ago
there's a gdoc spreadsheet link at the end
adamzegelin 6y ago
Safari too. I just discovered that modern user agents don't include table borders by default in their stylesheets -- at least, choosing Develop -> Disable Styles didn't make the table borders appear. I'm having flashbacks to the chiseled borders of Netscape.

sonofgod 6y ago

If you need a more stateful version of requests: import requests session = requests.session() # now use session like you would requests session.get("http://httpbin.org/cookies/set/name/value") print(session.get("http://httpbin.org/cookies").content)

catwind7 6y ago
oh I need to try that - I had this feeling that there was a more stateful version but for ..... some reason ... reaching for a new dependency felt easier at the time haha. Thanks

duxup 6y ago

I miss Roger Ebert. I honestly watch fewer films since his death. Maybe this list will change that. Also "Spirited Away" is on the list but doesn't seem to be included with prime.

catwind7 6y ago
updated the list. Thanks for the catch! sorry about that. I'm using an `in` keyword which just checks substrings ... that I meant to change. Woops
vincentmarle 6y ago
Spirited Away is now on HBO Max if you're looking for it

Alex3917 6y ago

FWIW the link for "The Bicycle Thief" (1949) links to "The Bicycle" (2015). Great list though. It would be interesting to see the thing done for the AFI and BFI top 100. (Although I suspect that most movies on the AFI are probably already on Ebert's list.)

intellijdd 6y ago

I was just seeing someone else's tutorial on scraping Amazon prices. They also ran into an issue where they needed to scrape twice instead of once. Not sure that it's the same issue you're facing but I thought I might drop my two cents.

catwind7 6y ago
I actually did notice that issue, even with using a stateful client like mechanize. Sometimes I had to scrape > 5 times in order to get through the "anti robot" page. Other times, I get no issue at all. It's weird - maybe they're doing some pattern matching on request metadata on their end?

exhilaration 6y ago

If the author reads this, the link in the Google sheet to the Battle of Algiers goes to Algiers, a totally different movie.

catwind7 6y ago
i'm fixing this list ... just realized a few bugs in the code. sigh

catwind7 6y ago

Author here. The list (both on my post + the google sheet) should be correct now - I underestimated how many different releases of the SAME movie title there are ... Thanks all for catching the mistakes.

jchazin 6y ago
FYI - the "Review URL" links at the bottom don't seem to be working properly. e.g. for "Moonstruck", the link leads to: "www.linisnil.com/articles/scraping-roger-ebert-reviews-and-amazon/www.rogerebert.com/reviews/great-movie-moonstruck-1987" instead of: "www.rogerebert.com/reviews/great-movie-moonstruck-1987"
- catwind7 6y ago
  hey thanks for catching that - turns out leaving out the protocol in markdown causes makes it a relative url. fixed.

trimbo 6y ago

"The Man Who Shot Liberty Valance" IMO, John Ford's best movie, hands down. Unfortunately, it is not actually available on Prime without "CBS All Access" [Edit: ah, I see that this is not just "Included with prime", but all movies]

dmix 6y ago
Prime just recently introduced channels, which dramatically increased the amount of content available, but each is $3.99. I personally love this because I’d rather have the option to subscribe to Smithsonian content or MGMs back catalog using the same streaming service I already use, rather than paying to use 10 different or getting stuck using the small list of (mostly old, TV movie, or B movie) content on just prime or Netflix. So this is a feature, not a bug. Mostly because I’ve accepted how backwards the movie industry is going to be with copyright. But still having the option is the less evil. That said, Maybe the UI can make this more obvious?
- joezydeco 6y ago
  Prime has had channels for almost five years now.
- trimbo 6y ago
  I typically use Google to search which streaming service has a movie, and for The Man Who Shot Liberty Valance, Prime says "Subscribed" underneath it.
- wolco 6y ago
  First time surprised me. I can see it being useful when you have the channel. Or as a separate tab when you want to possible add a channel. It kinda taints the other content. You ask internal.. Is it really available without subscribing? to channels you have access to. Then you finally click and they have season 1 and 3 and 5 of some show. Terrible UI. Feels like you get so much less than you do. With netflix it feels like you could just fall into a series immediately.

jamesrcole 6y ago

On a tangent: I dream of a web where whenever there are sets of items (eg eberts-great-movies, and movies-on-amazon) you can easily apply set operations (like intersection) on them (so if ‘n’ stands for ‘intersection’, eg eberts-great-movies n movies-on-amazon). So, in effect if you’re on a site that deals with a set of items, like the amazon prime movies, you can tell the browser to intersect this set with a different set at another URL. I understand that doing this would require the right ‘infrastructure’ to be in place.

castratikron 6y ago
I don't know why there's no good way to know what's streaming on a certain service. Seems like the only way is through third party sites that are probably made up of people manually adding to the list. If everyone is going to make their own streaming service then there needs to be a standard interface to make them collectively easier to use. A "guide", if you will.
- pjc50 6y ago
  Unfortunately their profit incentive is the opposite: to train you to watch what they choose to recommend, rather than take a step back and look at what isn't there. See the hollowing of the Netflix catalog.
- Exmoor 6y ago
  There was briefly an MPAA backed service for this whose name is escaping me, which I used quite a bit. Unfortunately it fairly quickly devolved into only giving links to services where you had to pay per-movie to watch.
theshrike79 6y ago
Ironically the pirate movie scene has tools like this[1][2], downloading new stuff based on certain rules is completely automated. All the paid options hold on to their APIs way too tight to let anything like this to be viable. I'd love to have a service that would cross-reference my queue of movies and TV shows and tell me which service has the most matches so that I'd know which one to pay for each month. [1] https://radarr.video https://radarr.video [2] https://couchpota.to https://couchpota.to

dvt 6y ago

Roger Ebert had an unexpected impact on me in my 20s. In my late teens, I started religiously reading his reviews, and this continued until his death. I've never really been a "movie buff," but Ebert's witty prose and pointed technical critique reminded me of Scalia (whom I also loved reading). Thank you for putting this list together!

autarch 6y ago
Besides being a well respected film critic and incredibly knowledgeable about film, he was also just a great writer. I read his reviews every week simply for the pleasure of reading them, even if I wasn't interested in the films he reviewed.
Mediterraneo10 6y ago
The tragedy of Roger Ebert is that, although he had a vast knowledge of great cinema, he established a niche where he was expected to mainly write about ephemeral popular films. His time was mostly spent on Hollywood blockbusters and popcorn, and he was appearing in media where he could not go into any great depth due to space limitations, or because that would be a turnoff for his mass audience. So, only in a few Ebert productions like his "Great Movies" books can one get a sense of the films that really mattered to the man and to art. Compare this to a critic like Richard Brody, who in his career has been fortunate to focus entirely on art cinema (though of course Brody’s net worth is probably an order of magnitude less than Ebert’s was).
- ctchocula 6y ago
  I've only read a few Ebert reviews, enjoyed them and happened to be looking for a book to read, so I appreciate the recommendation! edit: The introduction alone makes the book worth reading: > Of all the arts, movies are the most powerful aid to empathy, and good ones make us into better people. Not many of them are very good.
WalterBright 6y ago
I enjoyed Ebert when he teamed with Siskel in the early days before they became great friends, and they'd argue. Their debates were better than the movies :-)

boomboomsubban 6y ago

Only three movies in the past fourty years, non past 87, is surprising. Is Prime that full of older content or did Ebert's recommendations just stop coming from major studios?

zucker42 6y ago
Using https://www.rogerebert.com/great-movies https://www.rogerebert.com/great-movies you can filter by date
- boomboomsubban 6y ago
  Thanks, but that doesn't explain why Prime doesn't have any released after 87.
  - blaser-waffle 6y ago
    Presumably because contracts with studios won't let them stream those. Why won't they get streamed? 1980 is 40 years ago, and movies pushing 40+ years old aren't in the day-to-day zietgiest and aren't getting a lot of searches. Newer movies still catch eyeballs occasionally. I'd bet '87 is where the long-tail drops off noticeably.
    - boomboomsubban 6y ago
      It's been a while since I've looked at Prime's catalog, but is it really exclusives plus fourty year old content? I understand why those distribution rights are more valuable, but that is also why Amazon would desire them more.
  - ThePadawan 6y ago
    I'm having a hard time coming up with the data to confirm this theory but 1986/1987 seems to me to be the peak of both VHS sales and the bottom of VCR prices. I assume from that many studios had to come up with their first licensing schemes for movies "for home use" (contrary to movie theater and broadcast use), which could potentially still apply to (and restrict) streaming. Once again, this is my personal hypothesis (and I'd be happy to see some data to support or contradict it).

gittes 6y ago

Don't mean to undermine this guy's screen-scraping adventure... but if you want to use something that will tell you all the streaming services that have the list of movies, you can use: https://letterboxd.com/dvideostor/list/roger-eberts-great-movies/ https://letterboxd.com/dvideostor/list/roger-eberts-great-mo... You can look at each movie to see what streaming service it's on one at a time for free. If you have a pro paid account, you can even do: https://letterboxd.com/dvideostor/list/roger-eberts-great-movies/on/amazon-prime-us/ https://letterboxd.com/dvideostor/list/roger-eberts-great-mo... Which shows that there are 39 movies in Amazon Prime US from Ebert's "Great Movies," not 21 like this guy's spreadsheet says. To be fair, the exercise was to scrape the reference sources... so it might just need some refinement. Need to double check though if both lists are correct, only confirmed number totals. Full disclosure: That letterboxd list is not mine, I just found it

js2 6y ago
FWIW, I screen scraped rogerebert.com and copied all of his ratings and an excerpt of every review to letterboxd: https://letterboxd.com/re2/ https://letterboxd.com/re2/ Just the great movies: https://letterboxd.com/re2/tag/great-movie/films/by/release-earliest/ https://letterboxd.com/re2/tag/great-movie/films/by/release-... You can then filter those by streaming service, but you need a pro account. Looks like 38 movies: https://letterboxd.com/re2/tag/great-movie/films/on/amazon-prime-us/by/release-earliest/ https://letterboxd.com/re2/tag/great-movie/films/on/amazon-p... https://ibb.co/KFSj9jg https://ibb.co/KFSj9jg Apparently I missed the Buster Keaton movies: https://www.rogerebert.com/reviews/great-movie-the-films-of-buster-keaton https://www.rogerebert.com/reviews/great-movie-the-films-of-... https://letterboxd.com/director/buster-keaton/ https://letterboxd.com/director/buster-keaton/ But that means 39 isn't quite right either since Ebert is saying all of Buster Keaton's films are great. Anyway, the scraping was easy. The harder problem was parsing the html reviews (even with BeautifulSoup, the html is a mess), and then matching the reviews on Ebert's site to the correct movie, which I did via queries to tmdb and a lot of heuristics. There's nearly 8000 reviews and many have wrong years, bad titles, etc on rogerebert.com. It was a fun spare time project for a couple weeks.
- gittes 6y ago

bogomipz 6y ago

Although it's not listed in this post because it's a rental. I feel like it deserves a mention nonetheless. "Beyond the Valley of the Dolls" is a 1960s cult classic. It was directed by Russ Meyer and written by Roger Ebert. There's a link to it on Roger's site as well as available to stream from Amazon for cheap: https://www.rogerebert.com/reviews/beyond-the-valley-of-the-dolls-1980 https://www.rogerebert.com/reviews/beyond-the-valley-of-the-...

DarknessFalls 6y ago

I think this is referring to his list of "Movies You Must See Before You Die". On that note, I should mention "Gates of Heaven", which is part of that list. A compelling documentary on animal cemeteries, by Errol Morris. https://www.rogerebert.com/reviews/great-movie-gates-of-heaven-1978 https://www.rogerebert.com/reviews/great-movie-gates-of-heav...

Mediterraneo10 6y ago

It is the fact that we have a relatively agreed-upon canon of great films (with immense re-watch value over the years) that keeps me torrenting instead of streaming. Sure, you could search from one streaming site to another for a given film, but the film may be either completely unavailable, or it may be offered one moment and the mysteriously removed the next. Meanwhile, some torrent communities are run by ardent cinephiles and they have all of these films, and once you have downloaded it, you can go back to it whenever you want.

kjakm 6y ago
You can also solve the “what streaming service” issue by buying the movie on DVD or Blu Ray allowing you to go back to it whenever you want. Additionally if you’re ok with DRM iTunes has most stuff. Useful tip for finding which streaming service a film is on: google “film name streaming” and google lists the answers in a box at the top of the results.
- mywacaday 6y ago
  googling for film name streaming doesn't work, most of the results are specific to the USA instance of Netflix/Prime, not all content is available globally. Most of the websites that list content are spammy looking.
  - kjakm 6y ago
    I put my country name on the end of my query. It is a bit hit or miss though.
- Mediterraneo10 6y ago
  The problem with recommending that people buy the DVD/Blu-Ray is that the agreed-upon canon of great films stretches into the many hundreds of titles (Ebert’s list at 364 titles is only an abridged canon.) Therefore, for someone seeking a cultural education, buying the physical releases would run into the thousands and thousands of dollars. It would be a challenge even for North Americans or Western Europeans to amass a suitable collection, let alone people in lower-income countries. Consequently, a person can only realistically learn about great cinema by streaming from multiple subscription sites, or torrenting, and people disappointed by the former ought to remember that the latter exists.

jyriand 6y ago

I guess this list of movies applies to people who live in US. Available movies differ substantially between countries. With Netflix it's easy, it doesn't even show the movie you can't watch, with Amazon you have to click through every movie you are interested in to see the screen that says that this movie is not available in your region. There are even series where only one or two episodes are unavailable.

NicoJuicy 6y ago

I cancelled my Amazon prime subscription, because of the ridiculous things they did in France. And following the dark pattern path to unsubscribe. I'm happy that I did it. Fyi, I'm from Belgium

jrib 6y ago
I didn't hear about this. Can you give a quick summary of what actions Amazon took in France that you are referring to?

Curnee 6y ago

Self-promotion: A number of years ago I made a little website which links you to a random review written by Roger Ebert. It isn't the cleanest of implementations, but it did what I needed it to do when I built it. http://randomebert.com/ http://randomebert.com/

greggman3 6y ago

I'm surprised not to see more people adding their favorite movies. Just sticking to things semi-related. I may have to go watch "My Man Godfrey" again but at least for me "The Thin Man" and its 5 sequels stuck with me where as though I watched "My Man Godfrey" years ago but I don't remember any details of it. (same lead actor and he pretty much always plays the same type of person) "The Man Who Shot Liberty Valance" is a good movie but if you want some other westerns starting James Stewart I can recommend "Destry Rides Again" which is not super serious but I found it throughly entertaining. Also "Broken Arrow" (1950), (not the 90s John Woo movie which is a fun popcorn movie). Seeing "The Good, the Bad, and the Ugly" on the list I recently saw "The Outlaw Josey Wales" which I enjoyed (also a Western with Clint Eastwood) Of the movies on that list I've seen the one I'd recommend the most is "Women in the Dunes". I don't want to over hype it I've seen most of the movies on the list and they are all great stories but "Woman in the Dunes" might give you something new to think about where as the others are mostly just great entertainment.

patrec 6y ago
In addition to “Woman in the Dunes”, I'd also nominate “The Gospel According to Saint Matthew”. And, in fact, “It's a Wonderful Life” for those who haven't seen it (it's a much more grown up and dark film than you may have been led to believe).
gumby 6y ago
Other good Jimmy Stewart westerns are "The Man from Laramie" and "Winchester 73"

cm2187 6y ago

A weekend project idea for geeks like me who like films, but have the feeling to have already watched everything. I find that imdb ratings have a high (but not 1.0) correlation with me liking the movie (provided I like the genre). You can still download from imdb flat files that contain all movies, ratings, as well as cast/directors/producers/writers. Stick that in a database, with a basic UI to hide movies you have already watched. And you can make a good personal recommandation engine for movies you didn’t suspect exist. The power of this approach is that imdb is pretty much an exhaustive catalog.

arethuza 6y ago
I tend to find that the movies I really like have a strongly bimodal distribution of comments on IMDB - some love and some hate. e.g. "Blood Machines" - which I can see why a lot of people dislike it but I think it is awesome. Edit: Probably not a great idea to search for it if you have a problem with nipples.
- dfxm12 6y ago
  Do the movies you really dislike have the same distribution of comments? :) I do think that that there is something to media that creates a "love it or hate it" response. It's usually trying something new (Blood Machines certainly was). p.s.: I saw Blood Machines when it premiered on Shudder & really enjoyed it.
  - arethuza 6y ago
    Good question - I checked IMDB for Force Majeure and it does seem to have that "love it or loathe it" distribution - I tend to agree with the comment about it being "mind-numbingly pretentious drivel" (which I guess some people would apply to Blood Machines).
hhmc 6y ago
> I find that imdb ratings have a high (but not 1.0) correlation with me liking the movie You could track your own personal rating of the movies, then regress these ratings against a variety of external rating sources (IMDB, ebert, etc) to see which is the most predictive.

chillee 6y ago

Self-promotion (of sorts), but I (with some friends) have been watching lots of movies and keeping globally ordered rankings on github here: https://github.com/chillee/movierankings https://github.com/chillee/movierankings I find globally ordered rankings of movies to be an interesting exercise of consistency.

sfaruque 6y ago

There used to be a site call ClerkDogs.com that probably had the best movie recommendation system I've used. You started off naming a few movies you liked, and it would provide a list of movies you'd also probably like, and it was very accurate. From what I remember, the database was cataloged and maintained by actual humans, and not some algorithm following behavior patterns.

fastball 6y ago
Yep, jinni.com did the exact same thing, and I loved it. Unfortunately, it seems like B2C just didn't work financially so they switched to an entirely B2B model to help providers with their recommendation engines and no longer have their data accessible to end-users.
- cpach 6y ago
  It’s a shame that the economics of recommendation engines doesn’t seem to work very well in the B2C space. Good rec. engines can be very useful.
hanklazard 6y ago
https://www.criticker.com/ https://www.criticker.com/ This does almost exactly what you describe. I really like it. Not human curated but matches your preferences with other humans with similar preferences to give you recs.

lostgame 6y ago

I’d like to see this for more services. Amazon Prime has IMHO the worst UI/UX I have used in a streaming service.

TechBro8615 6y ago
Agreed. What’s most annoying is how they deal with geo blocked content. They don’t tell you it’s blocked until you start watching it. So you browse the catalog for five minutes, finally find a movie to watch, press play, and then get an error message.
- glenneroo 6y ago
  Or how they sometimes won't allow you to watch the original English version because you're not in an English speaking country. Even if your language is set to English (of which around 1/4 is still not translated). All movie previews are in native language which necessitates a visit to YT or IMDB. Even non-prime content is affected, just recently I paid 2.99 to watch some cheesy 90s movie with my friend who had always wanted to see this "classic". The syncing was abysmal and resolution 480p i.e. 720x480 i.e. DVD. After we were done, I noticed there were a handful of HD torrents in various languages. Reminded me immediately of Gabe Newell's thoughts on pirating games: "Piracy is almost always a service problem and not a pricing problem".

xtiansimon 6y ago

Are these free on Amazon or is this just an advertisement for Amazon Prime (or does everyone have a subscription but me?)

catwind7 6y ago
Not an advertisement for amazon prime :) These are free with prime, not free for non-subscribers I'm actually going to be cancelling my prime membership, which is another reason I wanted to see what I could watch for "free".
wolco 6y ago
Everyone had a subscription at one point not so sure anymore.

alexilliamson 6y ago

Side note: one excellent film that Roger Ebert didn't review is Life Itself, the documentary about his life that came out right after he died. It's full of joy and heartbreaking at times, but it really solidified his place of most universally relatable movie critic in my mind.

catwind7 6y ago
thank you for this. We will be checking this out!

cosmiccatnap 6y ago

Im not a fan of his reviews, they are sporatic and someone phoned in. Now Lenard Malton...

gabrielsroka 6y ago

See also "Where to Find Roger Ebert’s Great Movies Streaming" [0] which has US listings for Netflix, Amazon Prime, Hulu, Disney Plus, Criterion Channel, Kanopy, HBO, Starz, and Showtime as of March 2020. [0] https://www.rogerebert.com/features/where-to-find-roger-eberts-great-movies-streaming https://www.rogerebert.com/features/where-to-find-roger-eber...

samteeeee 6y ago

Folks interested in this topic might like my new side-project: get an email when your favorite director releases a new movie - https://directoralerts.website https://directoralerts.website

staycoolboy 6y ago

Thank you for doing this. I wish "top critics" was a search option on Prime (but their Roku interface is just abhorrent). There was a time when I hoped Netflix would do something similar: click here to see your favorite critics film list, but since Netflix has lost the streaming rights to soooo many films compared when they first started streaming ca. 2007, this is not doable. (Even DVD rights in most cases: films that are in my "Movies You Rated" queue from 2003 are no longer in Netflix's library.)

ddrt 6y ago
They have a really poor UX imo. The categories are seemingly arbitrary, and they have niche and seasonal ones that don’t make sense. During the December holidays they didn’t have a holiday themed category until near holiday end. Further, they make “innovation” like auto play when nobody asked for this, and without any way to stop it (unless you go to desktop and disable it deep in your settings). When searching you will receive similar results for titles they used to have... well why not state “are you searching for x? It’s no longer available, here are some recommendations” so the user isn’t wasting time wondering if the search feature is just “bad”. In the world of constant, unannounced, and live experimenting on huge user bases a little messaging goes a long way (with the comparison being nothing)

cottager 6y ago

When you said you were faking a proper agent with `requests`, do you mean you were setting the headers to look like a browser, as in here?: https://stackoverflow.com/questions/27652543/how-to-use-python-requests-to-fake-a-browser-visit https://stackoverflow.com/questions/27652543/how-to-use-pyth... That was going to be my suggestion for how to get around the anti-robot responses.

catwind7 6y ago
yeah so I have a user-agent generator that uses a random (valid) agent for each request. that, combined with cookies enabled, still didn't seem enough to get around the anti-robot barrier 100% of the time. 20% of the time, I get stopped. My guess is that they must do some kind of pattern matching on their end that kicks for some sets of requests but not others? But maybe it's not that sophisticated and I'm just missing something else in my headers
x3blah 6y ago
No agent at all is required. I got past the anti-robot response using no user-agent header and a simple delay.

youngamerican 6y ago

Reading Ebert's Great Movies site was hugely formative for me. I also love that he's low-key one of the best writers about addiction and recovery. Side note: A colleague who worked at the Sun-Times when Ebert also did recently told me about how whenever management threatened cuts, he'd come into the newsroom to throw his weight around against it. Even when he was getting to be in ill health. Much respect.

ngcc_hk 6y ago

Minor instruction to make it work for me: pip3 install BeautifulSoup pip3 install mechanize mkdir data Guess you can check data directory but not sure about the pip3, python does not like R I am not sure and can it pip3 install when not exist ....

x3blah 6y ago

Instead of using Python, here is a solution that only requires sh, curl, sed, sort, uniq and grep. This solution uses a generous 87s delay to retrieve the Amazon pages. There are 328 films listed as "great movies" on rogerebert.com. As such, the script, named "1.sh", needs 8h to complete, e.g., the time while you are at work or sleeping. No cookies, no state, no problems. Usage: sh -c 1.sh > 1.html Open 1.html in a browser and it shows whether each "great movie" is available as Prime Video or whether it is only available in some other format, such as Blu-ray, DVD, Multi-format, Hardcover. A link to the item on Amazon is provided. #!/bin/sh curl -HUser-Agent: -H'Accept: application/json' --compressed 'https://www.rogerebert.com/great-movies/page/[1-16]?utf8=%E2%9C%93&filters%5Btitle%5D=&sort%5Border%5D=newest&filters%5Byears%5D%5B%5D=1914&filters%5Byears%5D%5B%5D=2020&filters%5Bstar_rating%5D%5B%5D=0.0&filters%5Bstar_rating%5D%5B%5D=4.0&filters%5Bno_stars%5D=1'|grep -o "/reviews/great-movie-[^\\]*"|sed 's/.reviews.great-movie-//'|sort|uniq|while read x;do y=$(echo $x|sed 's/-/+/g');echo $x;curl -s --compressed -HUser-Agent: https://www.amazon.com/s/?k=$y 2>/dev/null|grep -m1 -C4 a-link-normal.a-text-bold;sleep 87;done|sed '/^[^< ]/s/.*/@&,/;1s|.*|<base href=https://www.amazon.com />&|;s/ *//;/^$/d;/^[@<]/!s|$|</a>|;1s/@//;s/@/<br>/'