Would love help simplifying this code (it looks really ugly right now) I am considering redesigning the api using lxml instead of BeautifulSoup for performance

Would love help simplifying this code (it looks really ugly right now) I am considering redesigning the api using lxml instead of BeautifulSoup for performance reasons.