3 ms·
I automate a log of all the HTTP requests the computer makes, which naturally includes all the websites I visit.^1 I am not always using a browser to make HTTP
by 1vuio0pswjnm7 4y ago
I automate a log of all the HTTP requests the computer makes, which naturally includes all the websites I visit.^1 I am not always using a browser to make HTTP requests, and for recreational web use I use a text-only one exclusively, so a "browser history" is not adequate.
In the loopback-bound forward proxy that handles all HTTP traffic from all applications, I add a line to include the request URL in an HTTP response header, called "url:" in this example. As such, it will appear in the log. For example, something like
http-response add-header url "https://%[capture.req.hdr(1)]%[capture.req.uri]"
This allows me to write simple scripts to copy URLs into a simple HTML page. I then read the simple HTML with a text-only browser (links).
For example, something like
cat > 1.sh
#!/bn/sh
WIDTH=120;
echo "<!-- $(date '+%Y-%m-%d %T') --><ol><pre>"
x=$(echo x|tr x '\034');
tr -d '\034' \
|sed -e "s/.*/& &/;s/ .\{${WIDTH}\}/&$x/;s/$x.*//" \
|sed -e "/./{s/.* /<li><a href=&>/;s|$|</a></li>|;}" \
-e '#certain urls to exclude' \
-e '/cdx?url=/d' \
-e '/dns-query?dns=/d' \
-e '/ds5q9oxwqwsfj.cloudfront.net/d' \
-e '/index.commoncrawl.org/d'
^D
grep url: 1.log|cut -d' ' -f3-|1.sh > 1.htm
links 1.htm
What about POST data. That is captured into another HTTP response header, called "post-data:" in this example
http-request add-header post-data %[req.body] if { method POST }
To look at the POST data I might do something like
grep post-data: 1.log|cut -d' ' -f3-|less
1. I also use a system for searching the www or specific www sites from the command line. The search results URLs for each query are stored in simple HTML format similar to the above. One query per file. What's non-obvious is that each file can contain search results from different sources, sort of like the "meta-search engine" idea but more flexible. The simple HTML format contains the information necessary to continue searches, at any time, thus allowing a more diverse and greater number of search results to be retrieved. (Sadly, www search engines have been effectively limiting the number of search result URLs we can retrieve with Javascript and cookies disabled.) The command line program reads the information to continue a search from the simple HTML comments.