3 ms·

Many sites behind Cloudflare. If I don't have this line in my .wgetrc, I get 403's sometimes from sites using Cloudflare. user_agent=Mozilla/5.0 (compatible;

• x0 11y ago
  Note: the specific user-agent doesn't matter, I just think it's funny to steal Cloudflare's one.
  • kuschku 11y ago
    I do the same whenever a site requests personal info from me and I don't want to give it. Find the corresponding info from the company or their CEO, use that. And whenever some site wants me to register for a newsletter, I enter their own "critical support" mail address. I hope at some point these sites learn that mandatory anything just hives you useless data.
    • x0 11y ago
      Haha, I usually put root@localhost, or www-data@localhost. It accepts it fairly often too. Sometimes I need to open up the Chrome Dev Tools and delete the input's type attribute (<input type="email"> to just <input>) because there's some client-side thing that tries to stop you inputting invalid emails.
• lmm 11y ago
  Wget uses its own user-agent by default (Wget/x.y.z (linux-gnu)), not no header, so that's more likely cloudflare or sites specifically blocking wget.
  • x0 11y ago
    Actually for the longest time, I had in my .wgetrc, user_agent="Mozilla/5.0 (etc etc etc)". It was a valid user agent. And Cloudflare was blocking that. Turns out if you wrap the actual user agent in quotes in your .wgetrc, wget doesn't strip them off, so cloudflare was seeing User-Agent: "Mozilla/5.0 (etc etc etc)", ie. with the quotes. And it was rejecting that.
    • lmm 11y ago
      Fair enough - but if it rejects that it would probably reject 10 random characters too