4 ms·
Reading WSJ without Javascript Examples curl -A "" https://www.wsj.com/amp/articles/meta-scrutinizing-sheryl-sandbergs-use-of-facebook-resources-over-sev
by 1vuio0pswjnm7 4y ago
Reading WSJ without Javascript
Examples
curl -A "" https://www.wsj.com/amp/articles/meta-scrutinizing-sheryl-sandbergs-use-of-facebook-resources-over-several-years-11654882829|hxextract p /dev/stdin > 1.htm
firefox ./1.htm
curl -A "" https://www.wsj.com/amp/articles/meta-scrutinizing-sheryl-sandbergs-use-of-facebook-resources-over-several-years-11654882829|grep -o "<p.*</p>" > 1.htm
firefox ./1.htm
x=$(echo x|tr x '\34');
curl -A "" https://www.wsj.com/amp/articles/meta-scrutinizing-sheryl-sandbergs-use-of-facebook-resources-over-several-years-11654882829 \
|tr -d '\12' \
|sed "s/<p[^a]/$x&/g" \
|tr '\34' '\12' \
|sed -n 's/<\/p>.*/<\/p>/p' > 1.htm
firefox ./1.htm
links -no-connect https://www.wsj.com/amp/articles/meta-scrutinizing-sheryl-sandbergs-use-of-facebook-resources-over-several-years-11654882829
- ratsmack 4y agoThe hxextract needs a '-' at the end to read from stdin, otherwise curl ends with error 23 indicating pipe closed prematurely. You also need a space after the '-A' or it sees the '-s' as the user agent name. The double quotes adjacent to the '-A' does nothing in that context.
- 1vuio0pswjnm7 4y agoForgive me, I do not use curl myself, except in HN examples. As a matter of practice, I never send a User-Agent header. The localhost forward proxy I use does not send it, nor do the HTTP client utilities I write. If some application sends the UA header, the proxy removes it. How to remove the User-Agent in curl, without editing the source. The idea to use -A "" comes from the curl help text: If you give an empty argument to -A, --user-agent (""), it will remove the header completely from the request. If you prefer a blank header, you can set it to a single space (" "). Another way is to use -H: curl -H User-Agent: https://example.com One might think that would send a UA header with no value, however it actually removes the UA header. One of the reasons I dislike curl is that there are too many options and the program is constantly changing. I would have to check older versions but I seem to recall in some past version(s) leaving off a space between the option and the option value had some (useful) effect in curl. Maybe not. At any rate, I apologise for the mistake. Thank you for the correction.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]