3 ms·
Here's the original problem statement: ----- We'd like you to write a simple web crawler in a programming language you're familiar with. Given a URL, it shoul
by yashmurty 7y ago
Here's the original problem statement:
-----
We'd like you to write a simple web crawler in a programming language you're familiar with. Given a URL, it should output a simple textual sitemap, showing the links between pages. The crawler should be limited to one subdomain - so when you start with https://google.com/* https://google.com/*, it would crawl all pages within google.com, but not follow external links, for example to facebook.com or mail.google.com.
We would like to see your own implementation of a web crawler. Please do not use frameworks like scrapy or go-colly which handle all the crawling behind the scenes. You are welcome to use libraries to handle things like HTML parsing.
-----
I hope this adds more context to my post. :)