3 ms·
Here is the original problem statement. ----- We'd like you to write a simple web crawler in a programming language you're familiar with. Given a URL, it shou
by yashmurty 7y ago
Here is the original problem statement.
-----
We'd like you to write a simple web crawler in a programming language you're familiar with. Given a URL, it should output a simple textual sitemap, showing the links between pages. The crawler should be limited to one subdomain - so when you start with https://google.com/* https://google.com/*, it would crawl all pages within google.com, but not follow external links, for example to facebook.com or mail.google.com.
We would like to see your own implementation of a web crawler. Please do not use frameworks like scrapy or go-colly which handle all the crawling behind the scenes. You are welcome to use libraries to handle things like HTML parsing.
-----