Would you mind sharing your spider technology stack. How do you find new urls efficiently? What spider and what storage do you use? Do you append everything in

Would you mind sharing your spider technology stack. How do you find new urls efficiently? What spider and what storage do you use? Do you append everything in one WARC file and split it if it gets too big?