4 ms·

Claude code decided to just put my name and email in the User-Agent when scraping docs from the SEC. No clever prompting required. It’s not a terrible idea rea

Claude code decided to just put my name and email in the User-Agent when scraping docs from the SEC. No clever prompting required. It’s not a terrible idea really, but I wish it would’ve asked me first.

mchinen 3mo ago
Why is it not a terrible idea?
- dannyw 3mo ago
  If you’re making automated requests, I consider it a common courtesy to provide an accurate user agent. Some services like Wikimedia will let you browse/download with rate limits IF your user agent is descriptive enough and not misleading.
  - mchinen 3mo ago
    Thanks, I wasn't aware of this. But to put your real name in the field instead of at least a pseudonymous id or more descriptive info but still have more bits of uncertainty user-agent for a public website, is that really a preferred practice?
    - remus 3mo ago
      As a website owner, if I saw someone scraping with a realistic looking name + email address I'd definitely give them more latitude than someone trying to hide the fact they're scraping. In my experience people who are hiding the fact are much more likely to be doing something nefarious.
- mirekrusin 3mo ago
  Sound legit to me as long as it's prompted to use hardcoded "dario amodei".
throw101010 3mo ago
How have you noticed that it did that?
- FriedPickles 3mo ago
  I could see it right there in the tool call: Bash(curl -A "<my name> <my email>" "sec.gov...")
threecheese 3mo ago
That’s the fault of the SEC EDGAR tool. The Edgar MCP docs show setting an env var: ‘SEC_EDGAR_USER_AGENT="Your Name (name@domain.com)"’. Claude just followed the instructions. Which is maybe even more risky?
- FriedPickles 3mo ago
  Interesting theory. The Claude Code instance says it's using this user agent based on the guidelines in the EDGAR Fair Access Policy: https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data https://www.sec.gov/search-filings/edgar-search-assistance/a... ...which indeed requests that bots supply `User-Agent: Sample Company Name AdminContact@<sample company domain>.com`. So in this case, the site just asked nicely and Claude Code complied. Note that this wouldn't have worked for Claude Chat, which the author was testing, because the `web_fetch` tool cannot set the User-Agent or other headers.
  - killix 3mo ago
    [flagged]

mchinen 3mo ago

Why is it not a terrible idea?

dannyw 3mo ago
If you’re making automated requests, I consider it a common courtesy to provide an accurate user agent. Some services like Wikimedia will let you browse/download with rate limits IF your user agent is descriptive enough and not misleading.
- mchinen 3mo ago
  Thanks, I wasn't aware of this. But to put your real name in the field instead of at least a pseudonymous id or more descriptive info but still have more bits of uncertainty user-agent for a public website, is that really a preferred practice?
  - remus 3mo ago
    As a website owner, if I saw someone scraping with a realistic looking name + email address I'd definitely give them more latitude than someone trying to hide the fact they're scraping. In my experience people who are hiding the fact are much more likely to be doing something nefarious.
mirekrusin 3mo ago
Sound legit to me as long as it's prompted to use hardcoded "dario amodei".

throw101010 3mo ago

How have you noticed that it did that?

FriedPickles 3mo ago
I could see it right there in the tool call: Bash(curl -A "<my name> <my email>" "sec.gov...")

threecheese 3mo ago

That’s the fault of the SEC EDGAR tool. The Edgar MCP docs show setting an env var: ‘SEC_EDGAR_USER_AGENT="Your Name (name@domain.com)"’. Claude just followed the instructions. Which is maybe even more risky?

FriedPickles 3mo ago
Interesting theory. The Claude Code instance says it's using this user agent based on the guidelines in the EDGAR Fair Access Policy: https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data https://www.sec.gov/search-filings/edgar-search-assistance/a... ...which indeed requests that bots supply `User-Agent: Sample Company Name AdminContact@<sample company domain>.com`. So in this case, the site just asked nicely and Claude Code complied. Note that this wouldn't have worked for Claude Chat, which the author was testing, because the `web_fetch` tool cannot set the User-Agent or other headers.
- killix 3mo ago
  [flagged]