I think that when you're not expected to publish any papers to rationalize what you're doing, you're free to use any possible ugly hack to improve your results,

I think that when you're not expected to publish any papers to rationalize what you're doing, you're free to use any possible ugly hack to improve your results, (using a "kitchen sink" approach where you just combine the results of lots of unrelated techniques, extracting words from the URL, using the URL to actually fetch some related textual content on the website, etc). This gives private companies a competitive advantage over research institutions - their only purpose is to "make things work", not to introduce new techniques and have interesting insight about them.