3 ms·

Parsing arbitrary titles for season/episode numbers was a mess of regexes ... 95% of threads are simple "S1E1" or "Season 1 Episode 4" patterns, but a few are h

by hoten 5y ago

Parsing arbitrary titles for season/episode numbers was a mess of regexes ... 95% of threads are simple "S1E1" or "Season 1 Episode 4" patterns, but a few are hokey: ``` EastEnders discussion 2020-09-07 - episode 6125 (or series 2 episode 1!) ``` That was parsed as Season 1 Episode 6125 :) My regexes grew gnarly from occasional tweaks, so I ripped it all out and changed up my process a bit. At some point mid-collection, I decided to default to season 1 if not mentioned (as many first-season subreddits fails to mention season number), but failed to re-run the parsing on everything. Just reparsed everything and deployed a new version. That resulted in ~1000 changes, but a cursory glance shows that it's all good changes, so should be much better now!

selcuka 5y ago
In any case it might be a good idea to replace `null` with something better (e.g. "Season ? Ep 1") or omit the season in such cases altogether.
lmm 5y ago
> That was parsed as Season 1 Episode 6125 :) And it sounds like that was legitimate?
- hoten 5y ago
  Actually yeah, I suppose so. I hadn't realized it was a soap opera and assumed it was a non-standard season/episode format. It's moot in this case, because the show doesn't seem to have any structured episode discussions in their subreddit. This one instance is the only thing my process found. https://www.redditdiscuss.com/eastenders/ https://www.redditdiscuss.com/eastenders/ Apparently they have 4 episodes/week. And there's been no success in starting regular discussion threads. And if they did, they'd probably prefer one thread per week (covering many episodes). It's quite unintuitive that communities formed around soap operas wouldn't have robust discussion threads. I guess because it'd be a full time job at the rate they release episodes... https://www.reddit.com/r/eastenders/comments/e3lyjh/episode_discussion_threads/ https://www.reddit.com/r/eastenders/comments/e3lyjh/episode_... https://www.reddit.com/r/eastenders/comments/nulg8n/episode_discussion_threads/ https://www.reddit.com/r/eastenders/comments/nulg8n/episode_...
duckmysick 5y ago
How did you select the subreddits? Most seem to match. I found that The Last Airbender: The Legend Of Korra has /r/Korra with 1.7k subscribers, but there's also /r/legendofkorra with 200k subscribers and more rewatch threads. > My regexes grew gnarly from occasional tweaks, so I ripped it all out and changed up my process a bit. Are you still using regex? There was a thread recently about collecting book recommendations from HN comments [1]. The app creator used deep learning and fine-tuned a BERT model with their own data. The results were impressive and I think it could work with unstructured data like yours too. I don't know what's your stack, but anything with Named Entity Recognition (NER) would be a good start. I really like the lists you compiled. Bookmarked. 1 - https://news.ycombinator.com/item?id=28595967 https://news.ycombinator.com/item?id=28595967
- hoten 5y ago
  Most of the subreddit/show pairs came from here: http://tv-subreddits.wikidot.com/ http://tv-subreddits.wikidot.com/ . Sourcing this data is the tricky part: I can't find any other sources. Idea I just had: maybe searching /r/television posts for "/r/..." links would be a good lead? > Are you still using regex? Yup, just in a much better way. I won't bother explaining how it was before, but now: I have a collection of regexes that try to match a "season" and "episode" at the same time. If that fails, I fall back to matching for them separately. If "episode" is not found, I ignore the thread as a possibly related to a specific episode. If I have further issues with the parsing, I'll consider a whole new approach like you've recommended. But I don't have any experience with that stuff so my first thought is that it'd be overkill.

by hoten 5y ago

selcuka 5y ago

In any case it might be a good idea to replace `null` with something better (e.g. "Season ? Ep 1") or omit the season in such cases altogether.

lmm 5y ago

> That was parsed as Season 1 Episode 6125 :) And it sounds like that was legitimate?

hoten 5y ago
Actually yeah, I suppose so. I hadn't realized it was a soap opera and assumed it was a non-standard season/episode format. It's moot in this case, because the show doesn't seem to have any structured episode discussions in their subreddit. This one instance is the only thing my process found. https://www.redditdiscuss.com/eastenders/ https://www.redditdiscuss.com/eastenders/ Apparently they have 4 episodes/week. And there's been no success in starting regular discussion threads. And if they did, they'd probably prefer one thread per week (covering many episodes). It's quite unintuitive that communities formed around soap operas wouldn't have robust discussion threads. I guess because it'd be a full time job at the rate they release episodes... https://www.reddit.com/r/eastenders/comments/e3lyjh/episode_discussion_threads/ https://www.reddit.com/r/eastenders/comments/e3lyjh/episode_... https://www.reddit.com/r/eastenders/comments/nulg8n/episode_discussion_threads/ https://www.reddit.com/r/eastenders/comments/nulg8n/episode_...

duckmysick 5y ago

How did you select the subreddits? Most seem to match. I found that The Last Airbender: The Legend Of Korra has /r/Korra with 1.7k subscribers, but there's also /r/legendofkorra with 200k subscribers and more rewatch threads. > My regexes grew gnarly from occasional tweaks, so I ripped it all out and changed up my process a bit. Are you still using regex? There was a thread recently about collecting book recommendations from HN comments [1]. The app creator used deep learning and fine-tuned a BERT model with their own data. The results were impressive and I think it could work with unstructured data like yours too. I don't know what's your stack, but anything with Named Entity Recognition (NER) would be a good start. I really like the lists you compiled. Bookmarked. 1 - https://news.ycombinator.com/item?id=28595967 https://news.ycombinator.com/item?id=28595967

hoten 5y ago
Most of the subreddit/show pairs came from here: http://tv-subreddits.wikidot.com/ http://tv-subreddits.wikidot.com/ . Sourcing this data is the tricky part: I can't find any other sources. Idea I just had: maybe searching /r/television posts for "/r/..." links would be a good lead? > Are you still using regex? Yup, just in a much better way. I won't bother explaining how it was before, but now: I have a collection of regexes that try to match a "season" and "episode" at the same time. If that fails, I fall back to matching for them separately. If "episode" is not found, I ignore the thread as a possibly related to a specific episode. If I have further issues with the parsing, I'll consider a whole new approach like you've recommended. But I don't have any experience with that stuff so my first thought is that it'd be overkill.