5 ms·
This is awesome. I am very new to scraping, so bear with me if this is very obvious. Would it be possible to follow a list of URLs from a home page (Ex: List o
by justboxing 12y ago
This is awesome. I am very new to scraping, so bear with me if this is very obvious.
Would it be possible to follow a list of URLs from a home page (Ex: List of Marathon Runners), and then follow the link in their name that goes to their stats page, and download / save the scraped data as JSON to a text file on the local machine's C:\Runners\Data\ folder for example?
Also, does anyone know of a reliable and tested C# / .Net / ASP.Net web page scrapper?
- cwbrandsma 12y agoOn the second question, Typically a web scraper just interacts with the output of a web server, is shouldn't matter if it is asp.net or any other system.
- misterbwong 12y agoMostly this. In ASP.NET/C# you're probably looking at using the built in HttpClient lib [0] and an html parser lib like HTMLAgilityPack [1]. I've used this combo in the past and am happy with it. [0] http://msdn.microsoft.com/en-us/library/system.net.http.httpclient(v=vs.118).aspx http://msdn.microsoft.com/en-us/library/system.net.http.http... [1] http://htmlagilitypack.codeplex.com/ http://htmlagilitypack.codeplex.com/
- justboxing 12y agoThanks!!
- ruipgil 12y agoOn the first question, yes, it's possible and easily done with the Router.
- justboxing 12y agoThank you!
- guiomie 12y agoSince this runs un nodejs, you could use edge.js to use scraperjs in your .net project.