5 ms·
When I was recently writing a Python script to ping the urls in places.sqlite to check for link rot, I had to change the urllib's useragent to mimick a modern b
by gkya 9y ago
When I was recently writing a Python script to ping the urls in places.sqlite to check for link rot, I had to change the urllib's useragent to mimick a modern browser, because otherwise lots of web sites redirect me around or return errors (even 404).
- UncleEntity 9y agoYep, one bored day I was playing with the oneom.tk API from python and it took a long while to figure out why I could get a page in the browser but the python API fetch wasn't working. Until I set User-Agent to "Mozilla/5.0" that is. Why anyone thinks it's a good idea not to send API data to a python User-Agent is kind of beyond me but who knows...can't really complain since they provide data for free and I really was just playing around without actually needing to do anything productive. Actually found a bug in urllib3 (that I probably should get around to reporting) -- page sends http unless you set Accept to 'application/json' which urllib3 doesn't let you do.
- gkya 9y agoYou should be able to add headers to your request. I set my user agent as such: import urllib.request as rq req = rq.Request(url, headers={'User-Agent': useragent}) request = rq.urlopen(req, timeout=timeout) You should be able to set Accept in a similar fashion (tho IDK if urllib mandates a certain value for that header).
- UncleEntity 9y agoYeah, it works in urllib but not in urllib3. IIRC 'User-Agent' wasn't the problem but trying to set 'Accept' is what failed (and even more strangely they only do User-Agent filtering on API calls but not for http). I started to dig into the code but since I got it to work in urllib I left it as Someone Else's Problem.
- gkya 9y agoOh, alright then. I didn't know urllib3 was a different thing. I'm recently returning to python after some years, still catching up with all the new things.
- haikuginger 9y agourllib3 lead maintainer here. I wasn't able to reproduce your bug, at least with the main development branch, when I passed `Accept` as part of the `headers` dict. If you could file an issue with a reproduction case, I'd definitely appreciate it.
- UncleEntity 9y agoSeems like the problem is due to the way they filter on User-Agent and not with urllib3, sorry.
- UncleEntity 9y agoEh, maybe... http = urllib3.PoolManager() r = http.request('GET', "https://oneom.tk/data/config" https://oneom.tk/data/config", headers={'Accept': 'application/json', 'User-Agent' : "Mozilla/5.0"}) r.headers['Content-Type'] 'application/json' r = http.request('GET', "https://oneom.tk/data/config" https://oneom.tk/data/config", headers={'Accept': 'application/json'}) r.headers['Content-Type'] 'text/html; charset=UTF-8'
- haikuginger 9y agoIt looks like this API in particular expects all requests to have a User-Agent header, and we don't set one by default. Setting any user agent appears to work; the issue isn't with the `Accept` header. You can reproduce with curl with the following command: curl -v 'https://oneom.tk/data/config' -H 'Accept: application/json' -H 'User-Agent:' That'll nullify the default curl user agent and should produce the same results you were seeing with urllib3.
- UncleEntity 9y agoThat makes sense, I kind of figured they were doing something screwy. I'm guessing a blacklist on certain User-Agent settings. When I was messing with it I could get it to work with curl and (eventually) urllib but urllib3 was no bueno until I tried to reproduce the problem and just copied the header with the User-Agent field from the urllib code.