Ask HN: textbooks, references for crawl+ingestion architecture

Hi, I'm writing a system that ultimately wants to grab lots of data and organize it into a database. I'm currently using a crawl to file, ingest file to database scheme, and I'm just wondering what is some good literature to read so I don't re-invent all the wheels (not just in the task at hand, but also in logging to evaluate performance, etc) Thanks.