ccgreg

3 ms·

1.
by ccgreg 2mo ago
How does that work for non-profit web crawls like the Common Crawl Foundation?
2.
by ccgreg 2mo ago
Common Crawl is text-only.
3.
by ccgreg 2mo ago
Are you talking about the 8087 transcendental instructions? Which are well known and well-written software doesn't have a problem with it.
4.
by ccgreg 2mo ago
Huh. A quick google says that's a feature bit and there's also an OS call to turn it on.
5.
by ccgreg 2mo ago
There are? We aren't talking about feature bits, and the only known software that checks the vendor are Intel's math libraries.
6.
by ccgreg 2mo ago
Another aspect of the x86_64 architectural swamp is that Intel deliberately makes their optimized math libraries fail if run on a chip that is not 'INTEL INSIDE'. That block totally ignores cpu feature bits. I've always wonde
7.
by ccgreg 2mo ago
How is this different from x86_64 and aarch64? Or even the various Alpha and Mips64 chips.
8.
by ccgreg 3mo ago
David recently announced he was leaving MLCommons, so maybe we'll get him back as an industry analyst.
9.
by ccgreg 3mo ago
Common Crawl has very little content from Reddit and 4chan -- both blocked us years ago.
10.
by ccgreg 3mo ago
Common Crawl is about 1 petabyte per year, compressed. Uncompressed is 4X.
11.
by ccgreg 3mo ago
Both IA and CCF are private foundations and accept donations.
12.
by ccgreg 3mo ago
> It's a shame that CCBot is caught in the cross fire, but that's life. We're used to it. Sadly.
13.
by ccgreg 3mo ago
The author blocked CCBot even though CCBot isn't part of the high traffic problem -- apparently he trusted Cloudflare labeling us as an "AI Bot".
14.
by ccgreg 3mo ago
That’s already a big business!
15.
by ccgreg 3mo ago
Thanks for the heads up -- this isn't popular yet, and it requires some work to avoid polluting things like the Internet Archive Wayback Machine.
16.
by ccgreg 3mo ago
The post says: > Because Common Crawl stores only the first 1 MB of each PDF That limit became 5 MB in March 2025.
17.
by ccgreg 3mo ago
Appreciate you double-checking.
18.
by ccgreg 3mo ago
That isn't true. You're welcome to peruse our index to prove or disprove your claim.
19.
by ccgreg 4mo ago
It's kind of interesting that you contradict much of what the article concludes, even though the article gives a lot of examples. Maybe your prediction will be true.
20.
by ccgreg 4mo ago
We do. First off we have a public parquet-format index of all of the urls we crawl every month. And then that also lives in a HDFS table that determines when we want to recrawl a page we've crawled before.
21.
by ccgreg 4mo ago
Appreciate your kind words! Many people have worked at Common Crawl over the years, and it's been a labor of love fueled by positive comments like yours and the large list of PhD theses helped by our public web dataset.
22.
by ccgreg 4mo ago
If you're referring to Common Crawl, which has existed since 2008, indeed your predictions are somewhat accurate. It's easy to opt out or limit what is collected. The crawling itself is inexpensive to us and the hosting is from th
23.
by ccgreg 4mo ago
We aren't sure if that really made a significant difference in Common Crawl's data quality. It does hurt our dataset from a humanities point of view, alas.
24.
by ccgreg 4mo ago
Common Crawl's archive has metadata that says when each record (html file) was crawled.
25.
by ccgreg 4mo ago
Common Crawl's dataset was downloaded in full 100 times in 2025. We agree that it would be great if it was even more widely used.
26.
by ccgreg 4mo ago
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
27.
by ccgreg 5mo ago
Good timing, I'm about to release that dataset.
28.
by ccgreg 6mo ago
Common Crawl is working hard to improve diversity in our crawl.
29.
by ccgreg 7mo ago
I don't know of anyone who uses Common Crawl as pre-training data without filtering it. We have an annotation system that lets people pick and choose which subsets they'd like to use.
30.
by ccgreg 7mo ago
Common Crawl is a sample of the web, so it's not that directly helpful for someone wanting to make a product price dataset.

More ›