3 ms·
Hey Ben, thanks for the write up on my framework. Firstly, the lack of documentation for more advanced scraping is something I plan on getting around to. You c
by chrisohara 16y ago
Hey Ben, thanks for the write up on my framework. Firstly, the lack of documentation for more advanced scraping is something I plan on getting around to. You can incorporate proxies, scraping pages behind logins, etc. If anyone needs help in the mean time, send me a message on github.
The main thing I'd like to point out is that I built it primarily as a command line tool. "By implementing an input method there’s no way to specify a search term from the command line" - so leave/comment the input method out! The default input method is to read lines from STDIN, just like the default output method is to write to STDOUT
Try commenting out the input line on that google example and running it with a list of words in a file:
node.io google_keywords < input.txt
Or you could feed the results in to another node.io job:
cat input.txt | node.io google_keywords | node.io someotherjob