3 ms·
I used BS to scrape rogerebert.com and post his reviews to letterboxd: https://letterboxd.com/re2/ https://letterboxd.com/re2/ I copied over only the first tw
by js2 4y ago
I used BS to scrape rogerebert.com and post his reviews to letterboxd:
https://letterboxd.com/re2/ https://letterboxd.com/re2/
I copied over only the first two paragraphs of each of his reviews with a link back to the original.
The HTML is a total mess having obviously moved around the web a couple times. So it required a bunch of cleanup. But that wasn't even the hard part. The hard part was getting the correct TMDB ID for the movies because his reviews also have either no useful metadata or metadata that's wrong, like incorrect movie years, misspelled actor names, etc.
I never was able to get API access to letterboxd, but they have a CSV import feature which worked-out well enough.
- ravishi 4y agoI had my share of gigs where we just decided to scrape the old site with BS and extract structured data from there to render a new site. It was sometimes cheaper than dealing with their ancient ad-hoc cms monstrosities.
- js2 4y agoAfter I managed to wrangle the review text from the HTML it still needed this sort of cleanup: def clean_text(text): text = re.sub(r"[\x7f-\x9f]", "", text) # remove control chars text = re.sub(r"[\xa0\r\t]+", " ", text) # replace with spaces text = re.sub(r"\n+", "\n", text) # squash runs of newlines text = re.sub(r"\s+", " ", text) # squash runs of spaces # Remove newlines unless they appear to be at the end of a sentence # or if the sentence is shorter than 80 characters. text = re.sub(r"([^.?!\"\)])\n", r"\1 ", text) text = re.sub(r"\n([^\n]{,80})\n", r"\1 ", text) return text.strip()