3 ms·
Why don't try to actually read the HTML within the <head> tags instead, and parse the title from there? Should be clean data and survive for much longer. As fo
by capableweb 5y ago
Why don't try to actually read the HTML within the <head> tags instead, and parse the title from there? Should be clean data and survive for much longer.
As for why someone would embed JavaScript directly on page instead of via a script, usually you want that piece of code to run as soon as possible when the user opens the code. Including scripts that gets loaded over network can get loaded much later compared to when you inline it.
- progval 5y agoAs URLs are provided by random users, a malicious server could send an infinite file to DoS the bot, so there has to be a limit.
- capableweb 5y agoCouldn't you disconnect the steps? First, start loading contents of URL and if it's below N bytes, try to parse the HTML and then query it via whatever library you can use in Python for querying HTML documents.