3 ms·
I wrote my own spider in RoR (to manage a database of historical information since I run it weekly). The spidering is done by controlling Firefox using Selenium
by jgrahamc 17y ago
I wrote my own spider in RoR (to manage a database of historical information since I run it weekly). The spidering is done by controlling Firefox using Selenium from the RoR application.
- strick 17y agoI'd love to have a copy of this spider if you ever decide to open source it. I've built WATIR-based robots to drive a browser to do useful stuff before, but have not yet tinkered with Selenium.
- aristus 17y agoI'm working on a "headless safari" script for performance testing. It's based on Paul Hammond's webkit2png. There's also Perl's WWW-Mechanize. It seems kind of odd that these kinds of programs tend to be either one-trick hacks, or hugely complex "suites" like Selenium.