4 ms·
it monitors third-party HTTP requests, not links: "webXray is a tool for detecting third-party HTTP requests on large numbers of web pages and matching them to
by tlibert 11y ago
it monitors third-party HTTP requests, not links: "webXray is a tool for detecting third-party HTTP requests on large numbers of web pages and matching them to the companies which receive user data...The core of webXray is a python program which ingests addresses of webpages, passes them to the headless web browser PhantomJS, and parses requests in order to determine those which go to domains which are exogenous to the primary (or first-party) domain of the site. This data is then stored in MySQL for later analysis."
- kazinator 11y agoSo in other words PhantomJS is used as an API to have some pages crawled, and the links emanating from those pages are captured.
- tlibert 11y agoI use phantomjs because it can run headless and is low resource. the request data is processed in python though.
- tlibert 11y agosorry, it doesn't monitor links, it captures network-level requests, that's what I analyze.
- deleted 11y ago[deleted]
- detaro 11y agoI think the requests PhantomJS makes while loading and executing the page are recorded and their URLs stored.
- kazinator 11y agoSo it wouldn't work if PhantomJS informed about this with, say, a callback function?
- rsanders 11y agoNot links. Requests. PhantomJS is a scriptable browser, as a second of googling will tell you.
- tlibert 11y agoyes! from the api docs: onResourceRequested Introduced: PhantomJS 1.2 This callback is invoked when the page requests a resource. The first argument to the callback is the requestData metadata object. The second argument is the networkRequest object itself. The requestData metadata object contains these properties: id : the number of the requested resource method : http method url : the URL of the requested resource time : Date object containing the date of the request headers : list of http headers The networkRequest object contains these functions: abort() : aborts the current network request. Aborting the current network request will invoke onResourceError callback. changeUrl(newUrl) : changes the current URL of the network request. By calling networkRequest.changeUrl(newUrl), we can change the request url to the new url. This is an excellent and only way to provide alternative implementation of a remote resource. (see Example-2) setHeader(key, value)