3 ms·
I'd love to see what kind of scraping tech Facebook use. I've been getting increasingly frustrated with the python scrapers and ecosystem and have been checking
by primax 4y ago
I'd love to see what kind of scraping tech Facebook use. I've been getting increasingly frustrated with the python scrapers and ecosystem and have been checking out Colly. But I'm sure whatever they have built must be nuts.
I wonder if you could train scrapers with AI with a reasonable level of success.
- cldellow 4y agoAs someone who maintains a scraper [1] in the python ecosystem, can I ask what you find frustrating? My scraper is targeted at smaller crawls, so your feedback might be totally irrelevant to me... but I always want to keep an ear out. [1]: https://github.com/cldellow/datasette-scraper/ https://github.com/cldellow/datasette-scraper/
- primax 4y agoI haven't used your scraper, I worked with scrapy. I found debugging as well as command and control to be challenging. Colly was a big step up. For reference I am primarily a Golang programmer so I find weakly typed tools frustrating.