5 ms·

Running a server just to set the user agent header to the googlebot one for some requests feels a bit heavyweight. But perhaps it’s necessary, as it seems Fire

Running a server just to set the user agent header to the googlebot one for some requests feels a bit heavyweight. But perhaps it’s necessary, as it seems Firefox no longer has an about:config option to override the user agent…am I missing it somewhere? Edit: The about:config option general.useragent.override can be created and will be used for all requests (I just tested). I was confused because that config key doesn’t exist in a fresh install of Firefox. The user agent header string from this repo is: "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html http://www.google.com/bot.html)"

darknavi 2y ago
Personally I find it nice for sending articles to friends.
- samstave 2y ago
  You can make a search related function in FF by rightclicking on that box and 'add keyword for this search' https://i.imgur.com/AkMxqIj.png https://i.imgur.com/AkMxqIj.png and then in your browser just type the letter you assign it to: for example, I have 'i' == the searchbox for IMDB, so I type 'i [movie]' in my url and it brings up the IMDB search of that movie . https://i.imgur.com/dXdwsbA.png https://i.imgur.com/dXdwsbA.png So you can just assign 'a' to that search box and type 'a [URL]' in your address bar and it will submit it to your little thing.
- cortesoft 2y ago
  That would mean that your self-hosted install is exposed to the internet. I don't think I want to run a publicly accessible global relay.
  - KennyBlanken 2y ago
    Eh, pretty minimal risk unless you use a guessable hostname and/or the URL gets published somewhere. If the install is under "eixk3.somedomain.com/ladderforfriends" and it sits behind a reverse proxy, it might as well be invisible to the internet, unless your DNS provider is an idiot and allows zone transfers, or you are on a network where someone is snarfing up DNS requests and then distributing that info to third parties. If you restrict it to TLS 1.3, even someone sniffing traffic from one of your friends won't be able to glean anything useful, because the requested hostname is never sent in plaintext. Rotate the hostname if/when it becomes necessary...
    - WayToDoor 2y ago
      Your certificate will however show up in public certificate transparency lists. You could mitigate that with a wildcard cert, but still..
    - paulddraper 2y ago
      Those are all very plausible
    - cortesoft 2y ago
      As soon as you get a cert for that domain, you will start getting requests to it because of certificate transparency reports. Everyone will know immediately the site exists.
Beijinger 2y ago
This does not work anymore? https://addons.mozilla.org/en-US/firefox/addon/random_user_agent/ https://addons.mozilla.org/en-US/firefox/addon/random_user_a...
Zaheer 2y ago
If this is all it's doing then you could also just use this extension: https://requestly.com/ https://requestly.com/ Create a rule to replace user agent with "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html http://www.google.com/bot.html)" I just tried it and seems to work.
- unethical_ban 2y ago
  I tried "User Agent Switcher" since it doesn't require a login. Washingtonpost.com blocked me, and NYT won't load article contents.
- mathfailure 2y ago
  It used to be a good extension. Now it is crapware tied to web services. I don't want any web services, I don't want seeing ads about paid features, I want a free extension working absolutely locally and not phoning home. This piece of crap is, unfortunately, unfit.
NoboruWataya 2y ago
I use this extension which has a decent UI: https://webextension.org/listing/useragent-switcher.html https://webextension.org/listing/useragent-switcher.html
codetrotter 2y ago
> set the user agent header to the googlebot one Also, how effective is this really? Don’t the big news sites check the IP address of the user agents that claim to be GoogleBot?
- mdotk 2y ago
  This. 12ft has never ever worked for me.
  - HKH2 2y ago
    I know one website it works well on, so I still use it, but yes, most others fail.
- dutchmartin 2y ago
  If you would host that server on Google cloud, you would make it a lot harder already.
  - jsheard 2y ago
    https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/... They provide ways to verify Googlebot IPs specifically, anyone who cares to check wouldn't be fooled by running a fake Googlebot on Googles cloud. Likewise with Bingbot: https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26 https://www.bing.com/webmasters/help/how-to-verify-bingbot-3...
    - KennyBlanken 2y ago
      yes, where "cares" means "the lost revenue is greater than the cost of development, QA, and computational/network/storage overhead, and the impact of increased complexity, of a function that figures out whether people are faking their user agent." It's probably orders of magnitude greater than the revenue loss from the tiny minority of people doing such things, especially given not everyone who uses tools like these will become a subscriber if blocked, so that cuts the "lost" revenue down even further.
judge2020 2y ago
I always do DevTools -> Network Conditions to set UA, at least in Chrome.
chefandy 2y ago
It says it blocks ads and other things, too. I imagine the use case is someone wanting this for multiple devices/people so they don't have to set up an extension for every platform/device individually. I have no idea how effective it is

darknavi 2y ago

Personally I find it nice for sending articles to friends.

samstave 2y ago
You can make a search related function in FF by rightclicking on that box and 'add keyword for this search' https://i.imgur.com/AkMxqIj.png https://i.imgur.com/AkMxqIj.png and then in your browser just type the letter you assign it to: for example, I have 'i' == the searchbox for IMDB, so I type 'i [movie]' in my url and it brings up the IMDB search of that movie . https://i.imgur.com/dXdwsbA.png https://i.imgur.com/dXdwsbA.png So you can just assign 'a' to that search box and type 'a [URL]' in your address bar and it will submit it to your little thing.
cortesoft 2y ago
That would mean that your self-hosted install is exposed to the internet. I don't think I want to run a publicly accessible global relay.
- KennyBlanken 2y ago
  Eh, pretty minimal risk unless you use a guessable hostname and/or the URL gets published somewhere. If the install is under "eixk3.somedomain.com/ladderforfriends" and it sits behind a reverse proxy, it might as well be invisible to the internet, unless your DNS provider is an idiot and allows zone transfers, or you are on a network where someone is snarfing up DNS requests and then distributing that info to third parties. If you restrict it to TLS 1.3, even someone sniffing traffic from one of your friends won't be able to glean anything useful, because the requested hostname is never sent in plaintext. Rotate the hostname if/when it becomes necessary...
  - WayToDoor 2y ago
    Your certificate will however show up in public certificate transparency lists. You could mitigate that with a wildcard cert, but still..
  - paulddraper 2y ago
    Those are all very plausible
  - cortesoft 2y ago
    As soon as you get a cert for that domain, you will start getting requests to it because of certificate transparency reports. Everyone will know immediately the site exists.

Beijinger 2y ago

This does not work anymore? https://addons.mozilla.org/en-US/firefox/addon/random_user_agent/ https://addons.mozilla.org/en-US/firefox/addon/random_user_a...

Zaheer 2y ago

If this is all it's doing then you could also just use this extension: https://requestly.com/ https://requestly.com/ Create a rule to replace user agent with "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html http://www.google.com/bot.html)" I just tried it and seems to work.

unethical_ban 2y ago
I tried "User Agent Switcher" since it doesn't require a login. Washingtonpost.com blocked me, and NYT won't load article contents.
mathfailure 2y ago
It used to be a good extension. Now it is crapware tied to web services. I don't want any web services, I don't want seeing ads about paid features, I want a free extension working absolutely locally and not phoning home. This piece of crap is, unfortunately, unfit.

NoboruWataya 2y ago

I use this extension which has a decent UI: https://webextension.org/listing/useragent-switcher.html https://webextension.org/listing/useragent-switcher.html

codetrotter 2y ago

> set the user agent header to the googlebot one Also, how effective is this really? Don’t the big news sites check the IP address of the user agents that claim to be GoogleBot?

mdotk 2y ago
This. 12ft has never ever worked for me.
- HKH2 2y ago
  I know one website it works well on, so I still use it, but yes, most others fail.
dutchmartin 2y ago
If you would host that server on Google cloud, you would make it a lot harder already.
- jsheard 2y ago
  https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/... They provide ways to verify Googlebot IPs specifically, anyone who cares to check wouldn't be fooled by running a fake Googlebot on Googles cloud. Likewise with Bingbot: https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26 https://www.bing.com/webmasters/help/how-to-verify-bingbot-3...
  - KennyBlanken 2y ago
    yes, where "cares" means "the lost revenue is greater than the cost of development, QA, and computational/network/storage overhead, and the impact of increased complexity, of a function that figures out whether people are faking their user agent." It's probably orders of magnitude greater than the revenue loss from the tiny minority of people doing such things, especially given not everyone who uses tools like these will become a subscriber if blocked, so that cuts the "lost" revenue down even further.

judge2020 2y ago

I always do DevTools -> Network Conditions to set UA, at least in Chrome.

chefandy 2y ago

It says it blocks ads and other things, too. I imagine the use case is someone wanting this for multiple devices/people so they don't have to set up an extension for every platform/device individually. I have no idea how effective it is