5 ms·

Don't mean to undermine this guy's screen-scraping adventure... but if you want to use something that will tell you all the streaming services that have the lis

by gittes 6y ago

Don't mean to undermine this guy's screen-scraping adventure... but if you want to use something that will tell you all the streaming services that have the list of movies, you can use: https://letterboxd.com/dvideostor/list/roger-eberts-great-movies/ https://letterboxd.com/dvideostor/list/roger-eberts-great-mo... You can look at each movie to see what streaming service it's on one at a time for free. If you have a pro paid account, you can even do: https://letterboxd.com/dvideostor/list/roger-eberts-great-movies/on/amazon-prime-us/ https://letterboxd.com/dvideostor/list/roger-eberts-great-mo... Which shows that there are 39 movies in Amazon Prime US from Ebert's "Great Movies," not 21 like this guy's spreadsheet says. To be fair, the exercise was to scrape the reference sources... so it might just need some refinement. Need to double check though if both lists are correct, only confirmed number totals. Full disclosure: That letterboxd list is not mine, I just found it

js2 6y ago
FWIW, I screen scraped rogerebert.com and copied all of his ratings and an excerpt of every review to letterboxd: https://letterboxd.com/re2/ https://letterboxd.com/re2/ Just the great movies: https://letterboxd.com/re2/tag/great-movie/films/by/release-earliest/ https://letterboxd.com/re2/tag/great-movie/films/by/release-... You can then filter those by streaming service, but you need a pro account. Looks like 38 movies: https://letterboxd.com/re2/tag/great-movie/films/on/amazon-prime-us/by/release-earliest/ https://letterboxd.com/re2/tag/great-movie/films/on/amazon-p... https://ibb.co/KFSj9jg https://ibb.co/KFSj9jg Apparently I missed the Buster Keaton movies: https://www.rogerebert.com/reviews/great-movie-the-films-of-buster-keaton https://www.rogerebert.com/reviews/great-movie-the-films-of-... https://letterboxd.com/director/buster-keaton/ https://letterboxd.com/director/buster-keaton/ But that means 39 isn't quite right either since Ebert is saying all of Buster Keaton's films are great. Anyway, the scraping was easy. The harder problem was parsing the html reviews (even with BeautifulSoup, the html is a mess), and then matching the reviews on Ebert's site to the correct movie, which I did via queries to tmdb and a lot of heuristics. There's nearly 8000 reviews and many have wrong years, bad titles, etc on rogerebert.com. It was a fun spare time project for a couple weeks.
- gittes 6y ago
  Nice! Yeah, I wish letterboxd was free somehow without ads and they made their beta api public. Yeah, I bet there's not a great standard for normalization/corrections of tiles, making a distinction of like when a movie was made and when a movie was released and translations and imports. Good work.
  - js2 6y ago
    I ended up using the director and cast that are listed for most reviews on Ebert's site for matching the right movie. Even that required some tricks due to spelling errors or differences in how names were listed. I then flagged any matches that weren't unique or where the title wasn't similar enough for me to manually review. I think I only had to double-check about a hundred or so. I didn't use the letterboxd api. Instead, I generated csv files for the letterboxd importer. I then did a csv export from their site I could reconcile to look for import errors. Trivia: Ebert reviewed a few adult films which I couldn't import to letterboxd because the site officially doesn't allow those. BTW, it's only $19/year for an account. I have my own account I pay for which follows the re2 account. That way I can easily see any of the re2 reviews for movies in my own watchlist.
    - gittes 6y ago
      Yep, I caved and got an account last month too
- js2 6y ago
  I've added Buster Keaton's silent films from 1920-1929 since those are the ones Ebert is referencing in his review. That brings the total number of Great Movies available on Prime up to 43.
x3blah 6y ago
"Which shows that there are 39 movies in Amazon Prime US from Ebert's "Great Movies," not 21 like this guy's spreadsheet says." I could be wrong, I am not a Prime Video user, but the result I got was that there are 217 movies in Prime Video from Ebert's great movies. links -dump 1.html|grep -c Prime.Video Instructions on how to generate 1.html are here: https://news.ycombinator.com/item?id=23508182 https://news.ycombinator.com/item?id=23508182

by gittes 6y ago

js2 6y ago

FWIW, I screen scraped rogerebert.com and copied all of his ratings and an excerpt of every review to letterboxd: https://letterboxd.com/re2/ https://letterboxd.com/re2/ Just the great movies: https://letterboxd.com/re2/tag/great-movie/films/by/release-earliest/ https://letterboxd.com/re2/tag/great-movie/films/by/release-... You can then filter those by streaming service, but you need a pro account. Looks like 38 movies: https://letterboxd.com/re2/tag/great-movie/films/on/amazon-prime-us/by/release-earliest/ https://letterboxd.com/re2/tag/great-movie/films/on/amazon-p... https://ibb.co/KFSj9jg https://ibb.co/KFSj9jg Apparently I missed the Buster Keaton movies: https://www.rogerebert.com/reviews/great-movie-the-films-of-buster-keaton https://www.rogerebert.com/reviews/great-movie-the-films-of-... https://letterboxd.com/director/buster-keaton/ https://letterboxd.com/director/buster-keaton/ But that means 39 isn't quite right either since Ebert is saying all of Buster Keaton's films are great. Anyway, the scraping was easy. The harder problem was parsing the html reviews (even with BeautifulSoup, the html is a mess), and then matching the reviews on Ebert's site to the correct movie, which I did via queries to tmdb and a lot of heuristics. There's nearly 8000 reviews and many have wrong years, bad titles, etc on rogerebert.com. It was a fun spare time project for a couple weeks.

gittes 6y ago
Nice! Yeah, I wish letterboxd was free somehow without ads and they made their beta api public. Yeah, I bet there's not a great standard for normalization/corrections of tiles, making a distinction of like when a movie was made and when a movie was released and translations and imports. Good work.
- js2 6y ago
  I ended up using the director and cast that are listed for most reviews on Ebert's site for matching the right movie. Even that required some tricks due to spelling errors or differences in how names were listed. I then flagged any matches that weren't unique or where the title wasn't similar enough for me to manually review. I think I only had to double-check about a hundred or so. I didn't use the letterboxd api. Instead, I generated csv files for the letterboxd importer. I then did a csv export from their site I could reconcile to look for import errors. Trivia: Ebert reviewed a few adult films which I couldn't import to letterboxd because the site officially doesn't allow those. BTW, it's only $19/year for an account. I have my own account I pay for which follows the re2 account. That way I can easily see any of the re2 reviews for movies in my own watchlist.
  - gittes 6y ago
    Yep, I caved and got an account last month too
js2 6y ago
I've added Buster Keaton's silent films from 1920-1929 since those are the ones Ebert is referencing in his review. That brings the total number of Great Movies available on Prime up to 43.

x3blah 6y ago

"Which shows that there are 39 movies in Amazon Prime US from Ebert's "Great Movies," not 21 like this guy's spreadsheet says." I could be wrong, I am not a Prime Video user, but the result I got was that there are 217 movies in Prime Video from Ebert's great movies. links -dump 1.html|grep -c Prime.Video Instructions on how to generate 1.html are here: https://news.ycombinator.com/item?id=23508182 https://news.ycombinator.com/item?id=23508182