binux

6 ms·

1.
by binux 12y ago
https://gist.github.com/binux/67b276c51e988f8e2c31
2.
by binux 12y ago
I'm working on a benchmarking suite https://gist.github.com/binux/67b276c51e988f8e2c31 and meet some problem... pyspider comes from a vertical search engine project. we have two issues: - 100+ websites, they may c
3.
by binux 12y ago
Images not downloaded default. Both the fetcher and the phantomjs proxy is totally async.
4.
by binux 12y ago
I want to make it a http proxy in the beginning. But I found it hard to do so. Then I post every to it, but haven't change the name. But it works like a proxy, that any request with `fetch_type == 'js'` would be fetched throu
5.
by binux 12y ago
The fetcher fit you already...
6.
by binux 12y ago
the architecture of pyspider: http://blog.binux.me/assets/image/pyspider-arch.png And yes for centralized queue which is in scheduler. It's designed to satisfy about 10-100 million urls for each project. sch
7.
by binux 12y ago
sorry :(
8.
by binux 12y ago
To make it more flexible and easy to reuse? I have implemented most features I need now.
9.
by binux 12y ago
http://demo.pyspider.org/debug/js_test_sciencedirect is a sample for this. There is a phantomjs fetcher that can render the page as WebKit did. Furthermore, you can have some JavaScript running before/after page l
10.
by binux 12y ago
I have "organize the code using a single top-level package".
11.
by binux 12y ago
Yes, the scheduler, fetcher, processor is stand alone here, they are running in different process. But they are sharing some common libs. I haven't made a decision how to put them into a single package, and running together. Any advice
12.
by binux 12y ago
agree
13.
by binux 12y ago
Currently, yes and no. pyspider is running original python code, something like portia is a code generator (Apologize if I'm wrong, I have not use it). So it can been made as another WebUI module. But for flexible, I have no idea how t
14.
Show HN: A Python Spider System with Web UI(github.com)
159 points by binux 12y ago|40 comments