4 ms·
A Simple Web Scraper in Go
- natch 11y ago>In my day job at Sendwithus, we've been having trouble writing performant concurrent systems in Python. I'm curious, at a company that does email campaigns, what use do you have for a web crawler? (I realize that you've called this a scraper, but the code does have the underpinnings of a crawler, and that is reflected in the naming used in the code). Which version of Python are you using? Did you try asyncio?
- gschier 11y agoThis exercise was more about learning Go, and not so much writing something directly work-related. We don't have a specific use case for a web crawler. We do, however, have a need for parsing and manipulating HTML (for dynamic templating), making HTTP requests, and writing concurrent code, so this exercise actually taught me a lot of relevant things. I started off with a crawler in mind (hence the function named "crawl" (good catch)), but I stopped a bit short so that I could write a comprehensible tutorial (I find writing tutorials of things I've just learned to really help my understanding). I haven't looked at asyncio yet, but it does look interesting. We've mostly been using gevent (py 2.7) and have found it to be a bit too magical (ie monkey-patching core libraries) and hard to track down specific bottlenecks.