8 ms·
Colly: Fast and Elegant Scraping Framework for Golang
- gjem97 9y agoThe HTML parsing part appears to be "golang.org/x/net/html". Does anyone have experience parsing "real world" html with this? How does it do?
- jerf 9y agoIt's an HTML5-compliant parser. That means that, modulo bugs, it should produce the same results as any other modern HTML parser, which should also all be based on HTML5. For context, since I think a lot of people are still unaware of this, the HTML5 standard precisely specifies how HTML should be parsed: https://www.w3.org/TR/html5/syntax.html#parsing https://www.w3.org/TR/html5/syntax.html#parsing This is based on a survey of how the various browsers were handling it in reality, so it's not just one of those theoretical things that everybody ignores, it's an algorithm extracted from the brutal pragmatism of many separate code bases over many years. In theory now, all HTML parsing libraries should now be able to take the same input and produce the same DOM nodes. In practice I've not used a variety of such libraries, nor have I fed them very much pathological input, so I can't vouch for if this 100% true in practice, but in theory, there should no longer be any significant differences between HTML parsers in various languages, as they come on board with HTML5 compliance.
- anonacct37 9y agoI've used it to parse alot of broken HTML for a very large company. I have not used it to attempt to parse all the HTML on the web. My impression is that it's pretty good.
- maxpert 9y agoOnHtml callbacks? Not a big fan of callbacks when you have channels.
- asciimoo 9y agoInteresting idea, how do you imagine a channel based API for this?
- JoeAcchino 9y agoWould this work? c := colly.NewCollector() // this functions create a goroutine and returns a channel ch := c.HTML("a") e := <- ch link := e.Attr("href") // ... I'm a bit rusty (ah!) with go, so bear with me if the above contains errors.
- hellcow 9y agoI would ignore the GP's advice. Channels are prone to big errors -- panics and blocking -- which aren't detectable at compile time. They make sense to use internally but shouldn't be exposed in a public API. As one example, notice how the standard library's net/http package doesn't require you to use channels, but it uses them internally.
- afandian 9y agoOne thing I'm interested in is scraping those annoying sites that require JavaScript execution. More and more webpages are requiring JS even to display anything beyond a blank page. These sites self-select themselves for exclusion in scraping like this. I've looked into Headless Chrome, but I'd be interested to see a 'scraping framework' level abstraction for those sites.
- deleted 9y ago[deleted]
- spiderfarmer 9y agoYou could look into PhantomJS. Also effectively a headless browser, but it's scriptable. http://phantomjs.org http://phantomjs.org
- afandian 9y agoI started with that. Last time I looked, the maintainer had stepped down because of Chrome Headless. And it's not a "scraping framework" any more than Chrome Headless is. edit: Headless Chrome has very rich scripting integration with e.g. https://github.com/webfolderio/cdp4j https://github.com/webfolderio/cdp4j
- tym0 9y agoYou might have missed puppeteer[1], if you don't mind writing Javascript it seems to provide a simple interface for scraping. [1] https://github.com/GoogleChrome/puppeteer https://github.com/GoogleChrome/puppeteer
- asciimoo 9y agoScraping JS-only sites is also possible without a headless browser, but requires a bit more debugging of the internal structure of these sites. Most of JS-only websites have API endpoints with JSON responses, which can make scraping more reliable than parsing custom (and sometimes invalid) HTML. The drawback of headless browser based scraping is that it requires significant amount of cpu time and memory compared to "static" scraping frameworks.
- 9y ago
- tree_of_item 9y agoIs there any reason to use this instead of Puppeteer? It feels like Puppeteer is going to dominate this space unless another browser vendor makes their own framework.
- Xeoncross 9y agoRemember to setup DNS caching on the box or use something like https://github.com/viki-org/dnscache https://github.com/viki-org/dnscache. Also, there doesn't seem to be any checking reading the response body. You want to limit the read length.
- stesch 9y agoWhy are scrapers and scraping so popular? What is a real use case for it?
- stephengillie 9y agoNot seeing ads. Secondarily, there is a lot of data on the internet stored only in HTML pages. For data with multiple sources, HTML is still usually a common format. HTML just has more punctuation and errata to filter out than JSON, XML, or CSV.
- stesch 9y agoSo you want the data but you don't want to ask for it?
- stephengillie 9y agoIf I come across data I want/need, why should I have to highlight, copy, and read from my clipboard? It's much more elegant to have automation do this.
- deleted 9y ago[deleted]
- troels 9y agoMost businesses I've worked for/with had one reason or another to scrape data from various places. Often it's watching competitors, but can also be part of an automation of some process that is already happening manually.
- weberc2 9y agoAccessing information that isn't available via an API. For example, much information on HN is only available via scraping. Many banks can't afford to rebuild their backend (probably largely due to the liability/compliance costs) to support an API/client model, but they still want an application--such applications are pretty much required to scrape said banks HTML.
- integrii 9y agoYou will get to initial source code with this, but if you want JavaScript to work, you just use PhantomJS. If you want PhantomJS to be usable, you is casperJS... Until you find a site with FuzeJS or some other JavaScript or html intense site. Those won't render in PhantomJS. For that stuff, as of a few months ago, you can use Chrome headless. I wrote a couple go packages to make that easy. It basically runs headless chrome with a JavaScript REPL console you can use to interact with the session. https://GitHub.com/integrii/headlessChrome https://GitHub.com/integrii/headlessChrome I was also able to smash my while scraper bot into a docker container after working around a couple bugs.
- stcredzero 9y agoIt basically runs headless chrome with a JavaScript REPL console you can use to interact with the session. That looks cool! Would I be able to run Node scripts?
- simlevesque 9y ago> Would I be able to run Node scripts? No. Since you can't run Node scripts on Chrome, the same is true for Chrome headless.
- drej 9y agoBorrowing an argument from an article talking the speed of Python: focus on what's your bottleneck. If you're worried about the performance of your tool, make sure it's not actually waiting for something else (IO, network, scheduling, ...). Here, unless you're parsing a large amount of already downloaded files (a website dump, [re]parsing of a long-standing archive etc.), you're not going to get a huge benefit from using a fast parser, because network is going to be the challenging factor. Keep that in mind.
- Walkman 9y agoI totally agree! I like Go but this is not a field I would ever use it. Parsing the site-s will never will be the bottleneck, but following the HTML changes or making it work for multiple sites is... When it takes maybe even seconds to download a page, a couple of millisecond performance advantage of Go doesn't matter at all.
- megamindbrian 9y agoI wrote one that just takes a list of XPath queries and returns the set. It also accepts an object with properties set to XPath and returns the JSON object populated with the results of the XPath. Obviously you would need to end the XPath in text() or @attribute.