7 ms·

They are highly dependent on web traffic for revenue. And their costs are even increasing because while human viewers are decreasing they are getting hugged to

They are highly dependent on web traffic for revenue. And their costs are even increasing because while human viewers are decreasing they are getting hugged to death by AI scrapes.

johnnyanmac 1y ago
scraping Wikipedia feels like the stupidest possible move. You can in fact download the entire encyclopedia at any time and take all the time in the world parsing offline. For such purposes, I'd naively just setup some weekly job to download Wikipedia and then run a "scrape" on that. Even weekly may be overkill; a monthly snapshot may do more than enough.
- yorwba 1y ago
  You can download twice-monthly database dumps, but they consist of the raw wikitext, so you need to do a bunch of extra work to render templates and stuff. Meanwhile, if you write a generic scraper, it can connect to Wikipedia like it connects to any other website and get the correctly-rendered HTML. People who aren't interested in Wikipedia specifically but want to download pretty much the entire internet unsurprisingly choose the latter option.
  - parpfish 1y ago
    as somebody that has wrassled with the wikipedia dumps a number of times, i don't understand why wiki doesn't release some sort of sdk that gives you the 'official' parse
    - skywal_l 1y ago
      This. I tried that a few years ago and fell off my chair when I started to realized how DYI the thing is. It's a bunch of unofficial scripts and half-assed out of date help pages. At the time I though, well it's a bunch of hippies with a small budget, who can blame them? Now I learn that there is 600 of them with a budget in the hundreds of millions?? This is becoming another Mozilla foundation...
    - philipkglass 1y ago
      I have wrestled with it too. I believe it's because wikitext is an ad-hoc format that evolved so that the only 100% correct parser/renderer is the MediaWiki implementation. It's like asking for an SDK that correctly parses Perl. Only Perl can do that. There are a bunch of mainly-compatible third party parsers in various languages. The best one I've found so far is Sweble but even it mishandles a small percentage of rare cases.
  - Gander5739 1y ago
    There are also dumps of Wikipedia in html format.
    - yorwba 1y ago
      Do you mean the discontinued Enterprise HTML dumps https://dumps.wikimedia.org/other/enterprise_html/ https://dumps.wikimedia.org/other/enterprise_html/ or the even older discontinued static HTML dumps https://dumps.wikimedia.org/other/static_html_dumps/current/en/ https://dumps.wikimedia.org/other/static_html_dumps/current/... or is there another set of dumps I'm not aware of?
cm2012 1y ago
Isn't it true that only around 10% of Wikipedia massive budget is used to actually run the core website? The rest goes to bloated initiatives in the Wikimedia foundations orbit.
- vrindavan1 1y ago
  [flagged]
- ethmarks 1y ago
  Page 21 of their 2024 annual report[1] has expenses listed. About $3,000,000 for web hosting, about $100,000,000 for salaries and benefits out of $178,000,000 total. [1]: https://wikimediafoundation.org/wp-content/uploads/2025/04/WMF-Annual-Report-23-24.pdf https://wikimediafoundation.org/wp-content/uploads/2025/04/W...
  - cm2012 1y ago
    That includes all of the Wikimedia websites and non-profit activities though, not just the functioning of Wikipedia.org proper. That is a much lower percentage of the total.
  - bawolff 1y ago
    A web server is useless if you aren't paying someone to plug it in. There is probably a lot to criticize, but you need to go deeper than "salaries" are bad. You need some of those to actually run the website.
    - ThrowawayTestr 1y ago
      Even at a quarter million a year that's 400 salaries. How many people are needed to maintain a website?
      - ryan_lane 1y ago
        How many people are needed to maintain instagram, facebook, etc? Wikipedia isn't just a collection of static content. The page contents include static and dynamic content (via a lua scriptable set of templates), semantic data (wikidata), multimedia management (wikimedia commons), editing tools (WYSIWYG editing, with full support of the wiki markup and templating), global caching infrastructure, multiple datacenters for HA/DR, etc. There's also the need to support the staff and volunteer developers, which includes wikimedia cloud services, git hosting, config management and orchestration, CI, community hosted tool/bot services, etc. WMF has ~600 employees, and that's quite lean, for a service of their complexity.
crmd 1y ago
How is their revenue traffic-dependent?
- AstroBen 1y ago
  Their traffic is potential donations Something tells me a person is way less likely to donate if they're consuming the content through an LLM middleman
  - crazygringo 1y ago
    I don't know -- as I said in another comment, my Wikipedia usage has gone down 90% thanks to LLMs. But that means I'm still using it. Especially for more reference stuff like lists of episodes, filmographies, etc. As well as equations, math techniques, etc. If you're the kind of person who donates to Wikipedia, you're probably still using it some even if less, and continue to recognize its importance. Possibly even more, as a kind of collaboratively-edited authority like Wikipedia only becomes more important as AI "slop" becomes more prevalent across blogs etc.
    - arbol 1y ago
      But were you originally convinced to donate by one of those giant Jimmy Wales banners that come up once a year? People won't see them anymore if they're using AI summaries.
      - crazygringo 1y ago
        My point is, using Wikipedia 10% as much is still using Wikipedia. Does it matter if you see the banner 10 times or 100 times in a month?
        arbol 1y ago
        In terms of the effectiveness of the banner, yes it absolutely does. Multiple exposures increase your propensity to convert.
khamidou 1y ago
If you look up their latest annual report (https://wikimediafoundation.org/wp-content/uploads/2025/04/WMF-Annual-Report-23-24.pdf https://wikimediafoundation.org/wp-content/uploads/2025/04/W...) you can see that they're allocating ~1.7% of their expenses towards hosting. I doubt that they're getting "hugged to death" by AI scrapers.
- Liquix 1y ago
  this is a very eye-opening read on their financials: https://en.wikipedia.org/wiki/User:Guy_Macon/Wikipedia_has_Cancer https://en.wikipedia.org/wiki/User:Guy_Macon/Wikipedia_has_C...
  - skeaker 1y ago
    The easy counters to this article are: 1. I think their spending is a good thing. Charitable scholarships for kids and initiatives to have a more educated populous in general are things that I am happy to donate to. 2. As stated in the article, hosting is still a relatively simple expenditure compared to the rest of their operation. If Wikipedia really eats a huge loss, falling back to just hosting wouldn't be unrealistic, especially since the actual operations of Wikipedia are mostly volunteer run anyways. In the absolute worst case, their free data exports would lead to someone making a successor that can be moved to more or less seamlessly. The only real argument in my eyes is that their donation campaigns can seem manipulative. I still think it's fine at the end of the day given that Wikipedia is a free service and donating at all is entirely optional.
    - mminer237 1y ago
      AFAIK, they don't do any scholarships or really do any educational activities. By far their biggest spending item is just $105 million for salaries, mainly for all of its leadership, which is a majority of its expenses. The second biggest line item is grants at $25 million, primarily for users to travel to meet up. Then $10 million for legal fees, $7 million for Wikipedia-hosted travel. I think it's pretty unethical to say you have to donate to keep Wikipedia running when you're practically paying for C-suite raises and politically-aligned contributors' vacations.
fart-fart-FART 1y ago
[flagged]
- 113 1y ago
  They also serve images.
  - fart-fart-FART 1y ago
    [flagged]
    - _def 1y ago
      > In this test, the framework responds with the simplest of responses: a "Hello, World" message rendered as plain text. What a ridiculous comparison. You either don't know what you are talking about, or are a troll. Maybe even both.
      - fart-fart-FART 1y ago
        there are more complex tests in that benchmark. my point was, a decent server can serve tens or hundreds of thousands requests per second when it's a few kilobytes of static content from a RAM database. as long as your web service isn't horribly mismanaged, even to a $5 VPS the bot traffic is infinitesimal background radiation.
  - lxgr 1y ago
    Some of them even fairly often: https://news.ycombinator.com/item?id=26072025 https://news.ycombinator.com/item?id=26072025
- bawolff 1y ago
  Bandwidth costs aren't the dominating cost.
Rebelgecko 1y ago
Bandwidth is a ridiculously tiny portion of Wiki foundation's spending
- ryan_lane 1y ago
  Especially considering a considerably amount of their bandwidth is free, via peering agreements.
WatchDog 1y ago
Infrastructure has long been a tiny portion of wikipedias costs. I think Wikipedia even makes it easy to export all of its data, I don’t think AI scrapers would be a significant new cost
- _def 1y ago
  These are two very different cost factors. Scrapers don't use the available data dumps, that's why they scrape. They are also kind of dumb as they get lost in link structures constantly, which leads to unecessary traffic spikes.
Aurornis 1y ago
> they are getting hugged to death by AI scrapes. Wikipedia is not getting hugged to death by AI scrapers. The source letter shows a relatively small portion of traffic was reclassified as bot traffic. They get a lot of page views globally. It’s a popular website. The bot traffic is not crushing their servers.

They are highly dependent on web traffic for revenue. And their costs are even increasing because while human viewers are decreasing they are getting hugged to death by AI scrapes.

johnnyanmac 1y ago

scraping Wikipedia feels like the stupidest possible move. You can in fact download the entire encyclopedia at any time and take all the time in the world parsing offline. For such purposes, I'd naively just setup some weekly job to download Wikipedia and then run a "scrape" on that. Even weekly may be overkill; a monthly snapshot may do more than enough.

yorwba 1y ago
You can download twice-monthly database dumps, but they consist of the raw wikitext, so you need to do a bunch of extra work to render templates and stuff. Meanwhile, if you write a generic scraper, it can connect to Wikipedia like it connects to any other website and get the correctly-rendered HTML. People who aren't interested in Wikipedia specifically but want to download pretty much the entire internet unsurprisingly choose the latter option.
- parpfish 1y ago
  as somebody that has wrassled with the wikipedia dumps a number of times, i don't understand why wiki doesn't release some sort of sdk that gives you the 'official' parse
  - skywal_l 1y ago
    This. I tried that a few years ago and fell off my chair when I started to realized how DYI the thing is. It's a bunch of unofficial scripts and half-assed out of date help pages. At the time I though, well it's a bunch of hippies with a small budget, who can blame them? Now I learn that there is 600 of them with a budget in the hundreds of millions?? This is becoming another Mozilla foundation...
  - philipkglass 1y ago
    I have wrestled with it too. I believe it's because wikitext is an ad-hoc format that evolved so that the only 100% correct parser/renderer is the MediaWiki implementation. It's like asking for an SDK that correctly parses Perl. Only Perl can do that. There are a bunch of mainly-compatible third party parsers in various languages. The best one I've found so far is Sweble but even it mishandles a small percentage of rare cases.
- Gander5739 1y ago
  There are also dumps of Wikipedia in html format.
  - yorwba 1y ago
    Do you mean the discontinued Enterprise HTML dumps https://dumps.wikimedia.org/other/enterprise_html/ https://dumps.wikimedia.org/other/enterprise_html/ or the even older discontinued static HTML dumps https://dumps.wikimedia.org/other/static_html_dumps/current/en/ https://dumps.wikimedia.org/other/static_html_dumps/current/... or is there another set of dumps I'm not aware of?

cm2012 1y ago

Isn't it true that only around 10% of Wikipedia massive budget is used to actually run the core website? The rest goes to bloated initiatives in the Wikimedia foundations orbit.

vrindavan1 1y ago
[flagged]
ethmarks 1y ago
Page 21 of their 2024 annual report[1] has expenses listed. About $3,000,000 for web hosting, about $100,000,000 for salaries and benefits out of $178,000,000 total. [1]: https://wikimediafoundation.org/wp-content/uploads/2025/04/WMF-Annual-Report-23-24.pdf https://wikimediafoundation.org/wp-content/uploads/2025/04/W...
- cm2012 1y ago
  That includes all of the Wikimedia websites and non-profit activities though, not just the functioning of Wikipedia.org proper. That is a much lower percentage of the total.
- bawolff 1y ago
  A web server is useless if you aren't paying someone to plug it in. There is probably a lot to criticize, but you need to go deeper than "salaries" are bad. You need some of those to actually run the website.
  - ThrowawayTestr 1y ago
    Even at a quarter million a year that's 400 salaries. How many people are needed to maintain a website?
    - ryan_lane 1y ago
      How many people are needed to maintain instagram, facebook, etc? Wikipedia isn't just a collection of static content. The page contents include static and dynamic content (via a lua scriptable set of templates), semantic data (wikidata), multimedia management (wikimedia commons), editing tools (WYSIWYG editing, with full support of the wiki markup and templating), global caching infrastructure, multiple datacenters for HA/DR, etc. There's also the need to support the staff and volunteer developers, which includes wikimedia cloud services, git hosting, config management and orchestration, CI, community hosted tool/bot services, etc. WMF has ~600 employees, and that's quite lean, for a service of their complexity.

crmd 1y ago

How is their revenue traffic-dependent?

AstroBen 1y ago
Their traffic is potential donations Something tells me a person is way less likely to donate if they're consuming the content through an LLM middleman
- crazygringo 1y ago
  I don't know -- as I said in another comment, my Wikipedia usage has gone down 90% thanks to LLMs. But that means I'm still using it. Especially for more reference stuff like lists of episodes, filmographies, etc. As well as equations, math techniques, etc. If you're the kind of person who donates to Wikipedia, you're probably still using it some even if less, and continue to recognize its importance. Possibly even more, as a kind of collaboratively-edited authority like Wikipedia only becomes more important as AI "slop" becomes more prevalent across blogs etc.
  - arbol 1y ago
    But were you originally convinced to donate by one of those giant Jimmy Wales banners that come up once a year? People won't see them anymore if they're using AI summaries.
    - crazygringo 1y ago
      My point is, using Wikipedia 10% as much is still using Wikipedia. Does it matter if you see the banner 10 times or 100 times in a month?
      - arbol 1y ago
        In terms of the effectiveness of the banner, yes it absolutely does. Multiple exposures increase your propensity to convert.

khamidou 1y ago

If you look up their latest annual report (https://wikimediafoundation.org/wp-content/uploads/2025/04/WMF-Annual-Report-23-24.pdf https://wikimediafoundation.org/wp-content/uploads/2025/04/W...) you can see that they're allocating ~1.7% of their expenses towards hosting. I doubt that they're getting "hugged to death" by AI scrapers.

Liquix 1y ago
this is a very eye-opening read on their financials: https://en.wikipedia.org/wiki/User:Guy_Macon/Wikipedia_has_Cancer https://en.wikipedia.org/wiki/User:Guy_Macon/Wikipedia_has_C...
- skeaker 1y ago
  The easy counters to this article are: 1. I think their spending is a good thing. Charitable scholarships for kids and initiatives to have a more educated populous in general are things that I am happy to donate to. 2. As stated in the article, hosting is still a relatively simple expenditure compared to the rest of their operation. If Wikipedia really eats a huge loss, falling back to just hosting wouldn't be unrealistic, especially since the actual operations of Wikipedia are mostly volunteer run anyways. In the absolute worst case, their free data exports would lead to someone making a successor that can be moved to more or less seamlessly. The only real argument in my eyes is that their donation campaigns can seem manipulative. I still think it's fine at the end of the day given that Wikipedia is a free service and donating at all is entirely optional.
  - mminer237 1y ago
    AFAIK, they don't do any scholarships or really do any educational activities. By far their biggest spending item is just $105 million for salaries, mainly for all of its leadership, which is a majority of its expenses. The second biggest line item is grants at $25 million, primarily for users to travel to meet up. Then $10 million for legal fees, $7 million for Wikipedia-hosted travel. I think it's pretty unethical to say you have to donate to keep Wikipedia running when you're practically paying for C-suite raises and politically-aligned contributors' vacations.

fart-fart-FART 1y ago

[flagged]

113 1y ago
They also serve images.
- fart-fart-FART 1y ago
  [flagged]
  - _def 1y ago
    > In this test, the framework responds with the simplest of responses: a "Hello, World" message rendered as plain text. What a ridiculous comparison. You either don't know what you are talking about, or are a troll. Maybe even both.
    - fart-fart-FART 1y ago
      there are more complex tests in that benchmark. my point was, a decent server can serve tens or hundreds of thousands requests per second when it's a few kilobytes of static content from a RAM database. as long as your web service isn't horribly mismanaged, even to a $5 VPS the bot traffic is infinitesimal background radiation.
- lxgr 1y ago
  Some of them even fairly often: https://news.ycombinator.com/item?id=26072025 https://news.ycombinator.com/item?id=26072025
bawolff 1y ago
Bandwidth costs aren't the dominating cost.

Rebelgecko 1y ago

Bandwidth is a ridiculously tiny portion of Wiki foundation's spending

ryan_lane 1y ago
Especially considering a considerably amount of their bandwidth is free, via peering agreements.

WatchDog 1y ago

Infrastructure has long been a tiny portion of wikipedias costs. I think Wikipedia even makes it easy to export all of its data, I don’t think AI scrapers would be a significant new cost

_def 1y ago
These are two very different cost factors. Scrapers don't use the available data dumps, that's why they scrape. They are also kind of dumb as they get lost in link structures constantly, which leads to unecessary traffic spikes.

Aurornis 1y ago

> they are getting hugged to death by AI scrapes. Wikipedia is not getting hugged to death by AI scrapers. The source letter shows a relatively small portion of traffic was reclassified as bot traffic. They get a lot of page views globally. It’s a popular website. The bot traffic is not crushing their servers.