4 ms·
>Lightning Fast and Elegant Scraping Framework for Gophers the bottle neck in scraping is never the parsing/DOM representation/traversal.
by mlevental 9y ago
>Lightning Fast and Elegant Scraping Framework for Gophers
the bottle neck in scraping is never the parsing/DOM representation/traversal.
- deoxxa 9y agoTell that to the project I migrated from scrapy to go six months back. Granted, scrapy might be doing other "fun" things to eat into performance, but it was really night and day. Immediately went from CPU bottleneck to network.
- mlevental 9y agoi don't understand? your insuperable bottleneck was always network then
- tampo9 9y agoGood performance matters if you have decent networking infrastructure or your server has limited resources. Bandwidth and IP limits are the most common bottle necks, but these can be solved using multiple proxies and ssh tunnels. Colly has built in support for switching proxies [1]. [1] http://go-colly.org/docs/best_practices/distributed/ http://go-colly.org/docs/best_practices/distributed/
- mlevental 9y agothe limitation is usually rate throttling on the page >but these can be solved using multiple proxies and ssh tunnels. Colly has built in support for switching proxies interesting >your server has limited resources. possibly.
- deleted 9y ago[deleted]
- ospider 9y agoIt has always been scheduling and anti-blocking