Couldn't you disconnect the steps? First, start loading contents of URL and if it's below N bytes, try to parse the HTML and then query it via whatever library

Couldn't you disconnect the steps? First, start loading contents of URL and if it's below N bytes, try to parse the HTML and then query it via whatever library you can use in Python for querying HTML documents.