6 ms·
It seems that OpenAI is scraping [certificate transparency] logs
- drwhyandhow 11mo agoThis has been long the case! I think there whole business model is based off scraping lol
- Aurornis 11mo agoThis could be OpenAI, or it could be another company using their header pattern. It has long been common for scrapers to adopt the header patterns of search engine crawlers to hide in logs and bypass simple filters. The logical next step is for smaller AI players to present themselves as the largest players in the space. Some search engines provide a list of their scraper IP ranges specifically so you can verify if scraper activity is really them or an imitator. EDIT: Thanks to the comment below for looking this up and confirming this IP matches OpenAI’s range.
- jsheard 11mo agoIn this case it is actually OpenAI, the IP (74.7.175.182) is in one of their published ranges (74.7.175.128/25). https://openai.com/searchbot.json https://openai.com/searchbot.json I don't know if imitating a major crawler is really worth it, it may work against very naive filters, but it's easy to definitively check whether you're faking so it's just handing ammo to more advanced filters which do check. $ curl -I https://www.cloudflare.com HTTP/2 200 $ curl -I -H "User-Agent: Googlebot" https://www.cloudflare.com HTTP/2 403
- Aurornis 11mo agoThanks for looking it up!
- btown 11mo agoI don't have a statistic here, but I'm always surprised how many websites I come across that do limited user-agent and origin/referrer checks, but don't maintain any kind of active IP based tracking. If you're trying to build a site-specific scraper and are getting blocked, mimicking headers is an easy and often sufficient step.
- xyzzy_plugh 11mo agoIf you can't tell the difference between active tracking and inspecting request headers, then it's worth committing a bit of time to ponder. Particularly the costs associated with IP tracking at scale.
- ccgreg 11mo ago> Some search engines provide a list of their scraper IP ranges Common Crawl's CCBot has published IP ranges. We aren't a search engine (although there are search engines using our data) and we like to describe our crawler as a crawler, not a "scraper".
- bobsmooth 11mo ago>The logical next step is for smaller AI players to present themselves as the largest players in the space. We think we're so different from animals https://en.wikipedia.org/wiki/Mimicry https://en.wikipedia.org/wiki/Mimicry
- 827a 11mo agoThousands of systems, from Google to script kiddies to OpenAI to nigerian call scammers to cybersecurity firms, actively watch the certificate transparency logs for exactly this reason. Yawn.
- H8crilA 11mo agoFor those that never looked at the CT logs: https://crt.sh/?q=ycombinator.com https://crt.sh/?q=ycombinator.com (the site may occasionally fail to load)
- Eikon 11mo agoShameless plug :) https://www.merklemap.com/search?query=ycombinator.com&page=0 https://www.merklemap.com/search?query=ycombinator.com&page=... Entries are indexed by subdomain instead of by certificate (click an entry to see all certificates for that subdomain). Also, you can search for any substring (that was quite the journey to implement so it's fast enough across almost 5B entries): https://www.merklemap.com/search?query=ycombi&page=0 https://www.merklemap.com/search?query=ycombi&page=0
- chuckadams 11mo agoAny insights you can share on how you made search so fast? What kind of resources does it take to implement it?
- gmerc 11mo agoLet's prompt inject it
- mxlje 11mo agoSo? It’s public information and a somewhat easily consumable stream of websites to scrape, if my job was to scrape the entire internet I’d probably start there, too.
- jcims 11mo agoGiven these are trivially forged, presumably they aren't really using a Mac for scraping, right? Just to elicit a 'standard' end user response from the server? >useragent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; robots.txt;
- benjojo12 11mo agothe ip address the this comes from is a OpenAI search bot range: > "ipv4Prefix": "74.7.175.128/25" from https://openai.com/searchbot.json https://openai.com/searchbot.json
- Hrun0 11mo agoYes, it is very common to change your useragent for web scraping. Mainly because there are websites which will block you just based on that alone
- snowwrestler 11mo agoRight. Crawler user agent strings in general tend to include all sorts of legacy stuff for compatibility. This actually is a well-behaved crawler user agent because it identifies itself at the end.
- deathanatos 11mo ago… the UA is malformed, even. Makes me want to reconfigure my servers to just drop such traffic. If you can't be arsed to send a well-formed UA, I have doubts that you'll obey other conventions like robots.txt.
- throwaway613745 11mo agoOpenAI is scraping everything that is publicly accessible. Everything.
- Aachen 11mo agoYet they provide the user agents and IP address ranges which they scrape from, and say they respect robots.txt I run a web server and so see a lot of scrapers, but OpenAI is one of the ones that appear to respect limits that you set. A lot of (if not most) others don't even have that ethics standard so I'd not say that "OpenAI scrapes everything they can access. Everything" without qualification, as that doesn't seem to be true, at least not until someone puts a file behind a robots deny page and finds that chatgpt (or another of openai's products) has knowledge of it
- immibis 11mo agoThere's no evidence the barrage of residentially-proxied bot accesses hitting every public website have anything to do with OpenAI, but then again, there's also no evidence they don't.
- warkdarrior 11mo agoSo do Google, Microsoft/Bing, Yandex, etc. How else would they make sure their search/chatbot/q&a products are up to date?
- _pdp_ 11mo agoI wonder if this can be used to contaminate OpenAI search indexes?
- bombcar 11mo agoIf you somewhat want to avoid this, get a wildcard certificate (LE supports them: https://community.letsencrypt.org/t/acme-v2-production-environment-wildcards/55578 https://community.letsencrypt.org/t/acme-v2-production-envir... Then all they know is the main domain, and you can somewhat hide in obscurity.
- lysace 11mo agoUnfortunately they are a bit extra bothersome to automate (depending on your DNS provider/setup) because of the DNS CNAME-method validation requirement.
- jsheard 11mo agoYep, but next year they intend to launch an alternative DNS challenge which doesn't require changing DNS records with every renewal. Instead you'll create a persistent TXT record containing a public key, and then any ACME client which has the private key can keep requesting new certs forever. https://letsencrypt.org/2025/12/02/from-90-to-45#making-automation-easier-with-a-new-dns-challenge-type https://letsencrypt.org/2025/12/02/from-90-to-45#making-auto...
- deleted 11mo ago[deleted]
- Ajedi32 11mo agoOh, sweet! I didn't know about this. I have no need of wildcard certs, but this will greatly simplify the process of issuing certificates for internal services behind my local firewall. No need to maintain an acme-dns server; just configure the ACME client, set the DNS record and you're done? Very nice.
- 8cvor6j844qw_d6 11mo agoGreat to hear, one less API keys needed for the DNS records.
- cortesoft 11mo ago
- basilikum 11mo agoThey definitely do. Before this comment CT logs – aside from DNS queries – were the only way to know about https://onion.basilikum.monster https://onion.basilikum.monster and you have to send the hostname in the SNI, otherwise you get another certificate back. Of course I get some bot traffic including the OpenAI bot – although I just trust the user agent there, I have not confirmed that the origin IP address of the requests actually belongs to OpenAI. That's just the internet. I like the bots, especially the wp-admin or .env ones. It's fun watching them doing their thing like little ants.
- xpe 11mo agoLooking around at the comments, I have a birds-eye view. People are quite skilled at jumping to conclusions or assuming their POV is the only one. Consider this simplified scenario to illustrate: - X happened - Person P says "Ah, X happened." - Person Q interprets this in a particular way and says "Stop saying X is BAD!" - Person R, who already knows about X... (and indifferent to what others notice or might know or be interested in) ...says "(yawn)". - Person S narrowly looks at Person R and says "Oh, so you think Repugnant-X is ok?" What a train wreck. Such failure modes are incredibly common. And preventable.* What a waste of the collective hours of attention and thinking we are spending here that we could be using somewhere else. See also: the difference between positive and normative; charitable interpretations; not jumping to conclusions; not yucking someone else's yum * So preventable that I am questioning the wisdom of spending time with any communication technology that doesn't actively address these failures. There is no point at blaming individuals when such failures are a near statistical certainty.
- 47282847 11mo agoI agree with your analysis but try to not agree with your conclusion, purely for my own metal hygiene: I believe one can retrain the pattern matching of one’s brain for happier outcomes. If I let my brain judge this as a “failure“ (judgment “it is wrong“), I will either get sad about it (judgment “… and I can’t change it“) or angry (… and I can do something about it“). In cases such as this I prefer to accept it as is, so I try to rewrite my brain rule to consider it a necessary part of life (judgment “true/good/correct“).
- xpe 11mo agoAh, in case it didn't come across clearly, my conclusion isn't to blame the individuals. My assessment is to seek out better communication patterns, which is partly about "technology" and partly about culture (expectations). People could indeed learn not to act this way with a bit of subtle nudging, feedback, and mechanism design. I'm also pretty content accepting the unpleasant parts of reality without spin or optimism. Sometimes the better choice is still crappy, after all ;) I think Oliver Burkeman makes a fun and thoughtful case in "The Antidote: Happiness for People Who Can't Stand Positive Thinking" https://www.goodreads.com/book/show/13721709-the-antidote https://www.goodreads.com/book/show/13721709-the-antidote
- matt3210 11mo agoYour content is stolen for training the moment you put it up
- jfindper 11mo agoIt is an _incredible_ stretch to frame certificate transparency logs as "content" in the creative sense. The whole purpose of this data is to be consumed by 3rd-parties.
- integralid 11mo agoI don't see issue with OAI scraping public logs. But what GP probably meant is that OAI definitely uses this log to get a list of new websites in order to scrap then later. This is a pretty standard way to use CT logs - you get a list of domains to scrap instead of relying solely on hyperlinks.
- advisedwang 11mo agomatt3210 clearly means that the content of the website (revealed by the CT log) is what is being stolen, not the data in the CT log
- ang_cire 11mo agoI think their point is that the people registering certs may not intend their sites to be immediately scraped, but now OpenAI is bypassing e.g. google indexing or web spidering, and using your cert provider's CT entries to find you immediately for scraping.
- 0xdeadbeefbabe 11mo agoIt would be funny if your content disappeared when it was stolen.
- prepend 11mo agoIf I give my content away for free, it can’t be stolen. The point of putting up a public web site is so the public can view it (including OpenAI/google/etc). If I don’t want people viewing it, then I don’t make it public. Saying that things are stolen when they aren’t clouds the issue.
- poormathskills 11mo agoIs it still “scraping” when the purpose of these transparency logs is to be used for this purpose?
- LeifCarrotson 11mo agoThe ostensible purpose of the certificate transparency logs is to allow validation of a certificate you're looking at - I browse to https://poormathskills.com https://poormathskills.com and want to figure out the details of when its cert was issued. The (presumably) unintended, unexpected purpose of the logs is to provide public notification of a website coming online for scrapers, search engines, and script kiddies to attack it: I could register https://verylongrandomdomainnameyoucantguess7184058382940052829204504849.com https://verylongrandomdomainnameyoucantguess7184058382940052... and unwisely expect it to be unguessable, but as it turns out OpenAI is going to scrape it seconds after the certificate is issued.
- mh- 11mo agoUnintended: agreed. Unexpected: plenty of us called out this inevitability when the CT proposal was circulated.
- rcxdude 11mo agoThe main thing isn't validating the cert you're looking at, per se, it's to validate the activities of the issuers. Mainly that they aren't issuing certificates they aren't supposed to (i.e. you can monitor the logs for your domain to check some random CA you've never done business hasn't issued a cert for it). This is mainly enforced by any violations (i.e. any certificates found that don't show up in the logs) being grounds for being removed from browser's trusted list.
- 8cvor6j844qw_d6 11mo agoAnyone went with wildcard certificates to avoid disclosing subdomains in certificate transparency logs?
- toddgardner 11mo agoIf you want to learn more about Certificate Transparency Logs, how to pull and search them, we just did a 3 part series about how we did this at CertKit: https://www.certkit.io/blog/searching-ct-logs https://www.certkit.io/blog/searching-ct-logs
- kirito1337 11mo agoyawn, i saw this more than 1000 times privacy doesnt exist in this world
- nephihaha 11mo agoOf course it doesn't exist if you keep handing it away.
- throwaway150 11mo agoI don't understand the outrage in some of the comments. The certificate transparency logs are literally meant to be read by absolutely whoever wants to read them. The clue is right in the name. It's transparency logs! Transparency! I just don't understand how people with no clue whatsoever about what's going on feel so confident to express outrage over something they don't even understand! I don't mind someone not knowing something. Everybody has to learn things somewhere for the first time. But couldn't they just keep their outrage to themselves and take some time to educate themselves, to find out whether that outrage is actually well placed? Some of the comments in the OP are also misinformed or illogical. But there's one guy there correcting them so that's good. I mean I'd say that https://en.wikipedia.org/wiki/Certificate_Transparency https://en.wikipedia.org/wiki/Certificate_Transparency or literally any other post about CT is going to be far more informative than this OP!
- prepend 11mo agoPeople are just raging and want an outlet. They aren’t thinking logically. It’s been going on forever (remember how companies were reading files off your computer aka cookies in 1999?) This seems like a total non-issue and expect that any public files are scraped by OpenAI and tons others. If I don’t want something scraped, I don’t make it public.
- bigbuppo 11mo agoFor many years now. The crawlers, scanners, and bots start hammering a website within a minute of a certificate being issued. Remember to get your garbage WCM installed and secured before installing the real certificate as you have about a 15 second window before they're hammering around for fresh wordpress installs. Granted, you people are all smart enough to have all that automated using a CI/CD pipeline so that you just commit a single file with the domain name to a git repo and all that magic happens.
- accelbred 11mo agoSeems like you could set up a cert for a honeypot domain to collect ips of bots running off of the certificate transparency logs. If domain isnt linked from anywhere, then its pretty sure to be a bot isn't it?
- bigiain 11mo agoI usually get a cert for my public domain (root and usually with www. as a Subject Alternate Name (SAN)) and if I'm going to use subdomains I don't intend to become widely public, I'll add a wildcard SAN of *.example.com so I don't have to expose subdomains in transparency logs. There's some security downside there if my web servers get hacked and my certs exfiltrated, but for a lot of stuff that tradeoff seems reasonable. I wouldn't recommend this approach of you were a bank or a government security agency or a drug cartel.
- Bender 11mo agoIt seems that OpenAI is scraping [certificate transparency] logs They would be far from first. Any time I create a Wildcard cert in LE I immediately see a ton of sub-domain enumeration in my DNS query logs. Just for fun I create a bunch of wildcard certs for domains I do not even use just to keep their bots busy ... not used as in parked domains. This has been going on about as long as the CT logs have existed.