4 ms·

Web Crawler in Golang. [Rejected]

2 points by yashmurty 7y ago

yashmurty 7y ago
My submission was recently rejected. The problem statement was: Create a simple web crawler which creates sitemaps in a textfile. I am wondering where I can improve it, and how not to be rejected the next time.
- gus_massa 7y ago
  Where did you submit it to? Who rejected it?
  - yashmurty 7y ago
    A tech startup, whose name I will not be sharing here. :)
    - gus_massa 7y ago
      OK, I think it's fine to avoid naming them. But the context of the original question was not clear enough.
      - yashmurty 7y ago
        Here's the original problem statement: ----- We'd like you to write a simple web crawler in a programming language you're familiar with. Given a URL, it should output a simple textual sitemap, showing the links between pages. The crawler should be limited to one subdomain - so when you start with https://google.com/* https://google.com/*, it would crawl all pages within google.com, but not follow external links, for example to facebook.com or mail.google.com. We would like to see your own implementation of a web crawler. Please do not use frameworks like scrapy or go-colly which handle all the crawling behind the scenes. You are welcome to use libraries to handle things like HTML parsing. ----- I hope this adds more context to my post. :)
  - yashmurty 7y ago
    They were looking for some golang devs.
- yashmurty 7y ago
  Here is the original problem statement. ----- We'd like you to write a simple web crawler in a programming language you're familiar with. Given a URL, it should output a simple textual sitemap, showing the links between pages. The crawler should be limited to one subdomain - so when you start with https://google.com/* https://google.com/*, it would crawl all pages within google.com, but not follow external links, for example to facebook.com or mail.google.com. We would like to see your own implementation of a web crawler. Please do not use frameworks like scrapy or go-colly which handle all the crawling behind the scenes. You are welcome to use libraries to handle things like HTML parsing. -----

2 points by yashmurty 7y ago

My submission was recently rejected. The problem statement was: Create a simple web crawler which creates sitemaps in a textfile. I am wondering where I can improve it, and how not to be rejected the next time.

gus_massa 7y ago
Where did you submit it to? Who rejected it?
- yashmurty 7y ago
  A tech startup, whose name I will not be sharing here. :)
  - gus_massa 7y ago
    OK, I think it's fine to avoid naming them. But the context of the original question was not clear enough.
    - yashmurty 7y ago
      Here's the original problem statement: ----- We'd like you to write a simple web crawler in a programming language you're familiar with. Given a URL, it should output a simple textual sitemap, showing the links between pages. The crawler should be limited to one subdomain - so when you start with https://google.com/* https://google.com/*, it would crawl all pages within google.com, but not follow external links, for example to facebook.com or mail.google.com. We would like to see your own implementation of a web crawler. Please do not use frameworks like scrapy or go-colly which handle all the crawling behind the scenes. You are welcome to use libraries to handle things like HTML parsing. ----- I hope this adds more context to my post. :)
- yashmurty 7y ago
  They were looking for some golang devs.
yashmurty 7y ago
Here is the original problem statement. ----- We'd like you to write a simple web crawler in a programming language you're familiar with. Given a URL, it should output a simple textual sitemap, showing the links between pages. The crawler should be limited to one subdomain - so when you start with https://google.com/* https://google.com/*, it would crawl all pages within google.com, but not follow external links, for example to facebook.com or mail.google.com. We would like to see your own implementation of a web crawler. Please do not use frameworks like scrapy or go-colly which handle all the crawling behind the scenes. You are welcome to use libraries to handle things like HTML parsing. -----