The problem domain is sprawling and varied, so you ideally want a bit of everything. You definitely need to do a bunch of custom low level programming, as a lo

The problem domain is sprawling and varied, so you ideally want a bit of everything. You definitely need to do a bunch of custom low level programming, as a lot of standard tools can't cope with the data volumes involved. Like I'm not even storing known URLs in a DBMS anymore, the table would be too slow to update. High level language with the ability to make downcalls to a low level language is probably a sane compromise.