3 ms·

> The largest portion of all languages in Common Crawl https://commoncrawl.github.io/cc-crawl-statistics/plots/languages.html https://commoncrawl.github.io/cc-

• Svoka 1y ago
  Thanks! I wonder where this discrepancy comes from
  • tough 1y ago
    probably under-indexing of non-english sources by these crawlers. would be interesting if yandex opened some data sets!
    • simongray 1y ago
      And lots of people write on the web using English as a second language, which both reduces the presence of their native language and increases the presence of English.
      • tough 1y ago
        yep not a native english speaker here and yet my online footprint is mostly english due to software pushing me to learn it
  • numpad0 1y ago
    My guess is that reference counting at depth=1 only capture non-$LANG content which text parts don't matter a lot, e.g. photo galleries.