2 ms·
Suppose you have a batch job which does web scraping. The job accepts a parameter which tells it what User-Agent value to use. And you have a web page which la
by skissane 5y ago
Suppose you have a batch job which does web scraping. The job accepts a parameter which tells it what User-Agent value to use.
And you have a web page which launches the job. And that web page passes the User-Agent header from the incoming HTTP request as that job parameter.
You call the web page using curl to scrape a site. The batch job sends a curl User-Agent to the site, and the site blocks the request.
Next you try calling the web page using Google Chrome. This time the crawler batch job sends a Google Chrome User-Agent to the site, and the site allows the crawling to happen.
Did you commit fraud in the above scenario? And at which step was the fraud committed?