5 ms·
It's traditionally very easy to download all the posts of Blogger blogs with a single HTTP request. If the format is nameofblog.domain.tld, then to get a JSON
by 1vuio0pswjnm7 3y ago
It's traditionally very easy to download all the posts of Blogger blogs with a single HTTP request.
If the format is nameofblog.domain.tld, then to get a JSON feed of all the posts, something like
curl -sA "" https://www.blogger.com/feeds/nameofblog/posts/default
This has worked for many years.
- simonw 3y agoDoes that get all posts? I thought it was just the first "page".
- 1vuio0pswjnm7 3y agohttps://developers.google.com/blogger/docs/1.0/developers_guide_protocol https://developers.google.com/blogger/docs/1.0/developers_gu...
- simonw 3y agoDo you have an example URL that returns all of the posts for a large blog? I only managed to get a subset, but I might not have found the right example. (I wasn't sure where to get nameofblog from)
- 1vuio0pswjnm7 3y agoWhy not share the large blog are trying to retrieve. Note that Google's own blogs no longer use Blogger.
- simonw 3y agoHah, that explains why I couldn't find an example blog that this would work on! I was trying it against https://blogger.googleblog.com/ https://blogger.googleblog.com/ I don't have any particular blog in mind, I just want to see how big an XML file that old platform is willing to serve me. I'm interested in streaming large responses generally, see https://simonwillison.net/2021/Jun/25/streaming-large-api-responses/ https://simonwillison.net/2021/Jun/25/streaming-large-api-re...
- simonw 3y agoI don't understand. Do you need to be authenticated for the is to work? Can you provide an example URL that I can run curl against to get the full content of a blog?
- 1vuio0pswjnm7 3y agoSection 6.1: "Retrieving all blog posts"
- 1vuio0pswjnm7 3y agoCorrection: s/JSON/XML/;s/nameofblog/blogID/ Been some years since I last used this. blogID is extracted from HTML at nameofblog.blogspot.com as in the example script below. According to personal preference, I have a custom filter to process the XML into formatted plaintext, no images. To see some sample blogNames and blogIDs, search site:www.blogger.com targetBlogID Something like tr @ '\15' << eof|openssl s_client -tls1_3 -connect 142.250.185.132:443 -ign_eof > 1.htm GET /search?q=site%3Awww.blogger.com%20targetBlogID&num=100&filter=0&start=1&scoring=d HTTP/1.0@ Host: www.google.com@ Connection: close@ @ eof firefox ./1.htm Example script test $1||exec echo usage: $0 nameofblog x=$(curl -sA "" https://$1.blogspot.com|grep -o targetBlogID=[0-9]*|sed -n '1s/.*=//p;s/ //'); test ${#x} = 19||exit;curl -sA "" https://www.blogger.com/feeds/$x/posts/default;