9 ms·
Sorry about that! I didn't think it would be a problem, but I can see the programming error I made which is causing it. Normally, when we pull pages from RSS,
by e1ven 16y ago
Sorry about that!
I didn't think it would be a problem, but I can see the programming error I made which is causing it.
Normally, when we pull pages from RSS, we have a UUID attached to each, to avoid pulling it again. Since it's scraping HN, it doesn't get those, and isn't behaving nicely.
The reason it pulls the destination page is to get the final link- Too many sites are tinyurl/bit.ly/etc, so I load them to see who I'm dealing with at the end of the day.
I'll generate some sort of Token for those to avoid the problem later tonight.
Thanks, and again, Sorry for the trouble!
- e1ven 16y agoI've added a new check to the code- It'll check the base URL against existing stories, before doing the URL normalization. This should fix the problem for every URL that isn't a tinyurl. Thanks for the heads-up. Can you verify if it's nicer for your server now?