6 ms·
Colly – Scraping Framework for Golang
- blowski 9y agoThe obvious question - why would I use this over Scrapy?
- tptacek 9y agoBecause you're using Golang and not Python.
- sheraz 9y agoHere here. The right tool for the right job. And I can't think of a "righter" tool for this kind of job. Edit - not picking on you, but given the quality and ecosystem of libraries and ancillary tools for scrapy, I don't even consider alternatives at this point. Good on anyone who does it to learn but for actual workloads I won't consider anything else.
- jackmott 9y agoF# type provider could be.
- deleted 9y ago[deleted]
- mlevental 9y ago>Lightning Fast and Elegant Scraping Framework for Gophers the bottle neck in scraping is never the parsing/DOM representation/traversal.
- deoxxa 9y agoTell that to the project I migrated from scrapy to go six months back. Granted, scrapy might be doing other "fun" things to eat into performance, but it was really night and day. Immediately went from CPU bottleneck to network.
- mlevental 9y agoi don't understand? your insuperable bottleneck was always network then
- tampo9 9y agoGood performance matters if you have decent networking infrastructure or your server has limited resources. Bandwidth and IP limits are the most common bottle necks, but these can be solved using multiple proxies and ssh tunnels. Colly has built in support for switching proxies [1]. [1] http://go-colly.org/docs/best_practices/distributed/ http://go-colly.org/docs/best_practices/distributed/
- mlevental 9y agothe limitation is usually rate throttling on the page >but these can be solved using multiple proxies and ssh tunnels. Colly has built in support for switching proxies interesting >your server has limited resources. possibly.
- deleted 9y ago[deleted]
- ospider 9y agoIt has always been scheduling and anti-blocking
- dguaraglia 9y agoI'm always surprised by how many web scraping frameworks/libraries I see sprout here on HN on a regular basis. Is web scraping something people are doing, or is web scraping the new high-concurrency version of the "to do list" utility everyone used to write as an exercise? This is an honest question, I'm not trying to take a dig at anyone in particular.
- mlevental 9y agoi worked at a startup for 6 months (fintech) whose entire value prop was built on the back of scraping.
- dguaraglia 9y agoOh, I know there are very niche cases of companies making money out of it (one of the two jobs in my life that required web scraping was a freelance job for a company doing sector-specific news aggregation) I just find the stream of new libraries for doing this kind of fascinating.
- CyberShadow 9y agoJust yesterday I wrote a program to scrape some Amazon search and product pages [0]. Why? Because Amazon's search is outright broken. The number of results changes when you change sorting mode, and sometimes, sorting by a different criteria will just serve you a "no products found" error page. I'll generally write a personal product comparison program when it becomes clear that I can't be certain that I can find the best product by hand. Often, even specialized websites that should have parametrized product search/filtering don't have their data properly indexed, so you have to scrape and parse it yourself. Another reason is to cross-reference with data from other sources. E.g. what laptop can I buy that has the best single-threaded CPU performance (within some other restrictions) [1]? [0]: https://github.com/CyberShadow/choose-product/blob/master/amazon/common.d https://github.com/CyberShadow/choose-product/blob/master/am... [1]: https://github.com/CyberShadow/choose-product/blob/master/lenovo/choose.d https://github.com/CyberShadow/choose-product/blob/master/le...
- 9y ago
- deleted 9y ago[deleted]
- deleted 9y ago[deleted]
- Xeoncross 9y agoPlease breakup your main `colly.go` file into separate parts. If possible you shouldn't have a 30 line imports definition covering everything from cookies and regex to html and sync access. Make sure to use DNS caching on the box else add it in Go. Colly only supports a single machine via map of visited URL's. Would be great if you replace with a queue like redis or beanastalkd. visitedURLs map[uint64]bool
- guessmyname 9y ago> […] It's very helpful and healthy to have everything in a single file if you consider that manageable. SQLite has a different opinion — https://www.sqlite.org/amalgamation.html https://www.sqlite.org/amalgamation.html
- detaro 9y agoI don't think that's a good comparison to justify large code files: a) it's automatically generated, so you can do dev work on the "easier" split version b) Go comes with package management that helps solve the deployment issue c) Question: Does the concept of a "compilation unit" affect the Go compiler the same way?
- Xeoncross 9y ago> SQLite would like to disagree with you... ...Over 100 separate source files Um, how does this disagree?
- fiatjaf 9y agoPlease don't follow this suggestion. It's very helpful and healthy to have everything in a single file if you consider that manageable, so no problem at all.
- JepZ 9y agoWell, in Go all files in one directory are part of one package (best practice) and files within the same package do not have to import each other to have access to other functions. Therefore, breaking the package into several files is common practice. A sane approach to this is for example to create a separate file for each type (Collector, HTMlElement, Request, Response, ...) and its attached functions/methods.
- fiatjaf 9y agoFor DOM parsing I cannot imagine that there could anything better than https://github.com/PuerkitoBio/goquery https://github.com/PuerkitoBio/goquery.
- jjuel 9y agoWhich is funny because if you look at the code this is using goquery. Which then makes you wonder why would I use this when I can just use goquery?
- WaltPurvis 9y agoBecause goquery is only a DOM parser/manipulator and not a full-fledged (or even half-fledged) "scraping framework"?
- stablemap 9y agoSome discussion from two months ago: https://news.ycombinator.com/item?id=15408784 https://news.ycombinator.com/item?id=15408784
- dang 9y agoThanks! Missed that earlier.
- kondro 9y agoHow does this go with running the JS on the SPAs that make up a large portion of the web today?
- wiradikusuma 9y agoDoes it support JS/SPA? I don't see it mentioned. Unrelated: the name means "to masturbate" in Indonesian slang