5 ms·

> The modern Wikipedia hosts 11–12 times as many pages as it did in 2005, but the WMF is spending 33 times as much on hosting... Wikipedia also keeps "revision

by film42 9y ago

> The modern Wikipedia hosts 11–12 times as many pages as it did in 2005, but the WMF is spending 33 times as much on hosting... Wikipedia also keeps "revision history" and a "talk" section for each page. Those original pages created before 2005 can still be modified, potentially adding to the overall cost. Breaking news stories can cause massive amounts of churn which I imagine has increased with the site's popularity [1]. So really 11x pages = 33x higher hosting costs doesn't seem unrealistic considering how much metadata is associated with each page. That's not to say there isn't a problem, but "page count" might not be the best metric. I wonder what average number of revisions per page looks like over the last 12 years. [1] https://xtools.wmflabs.org/articleinfo/en.wikipedia.org/Talk:2017_Las_Vegas_Strip_shooting https://xtools.wmflabs.org/articleinfo/en.wikipedia.org/Talk...

revelation 9y ago
You're right, page count is not the best metric. Instead you can just look up the hosting costs in their yearly report and find out it's a pittance compared to all their other spending and declining in relation.
- QAPereo 9y ago
  Could you link me to that?
  - hk__2 9y ago
    https://upload.wikimedia.org/wikipedia/foundation/4/43/Wikimedia_Foundation_Audit_Report_-_FY15-16.pdf https://upload.wikimedia.org/wikipedia/foundation/4/43/Wikim... Page 3. In 2016 they spent $2M in hosting vs. e.g. $2.2M in "travel and conferences" and $3.6M in "donations processing expenses".
    - QAPereo 9y ago
      Well, shit. That’s horrendous, half of the money is just going to salaries.
      - Alex3917 9y ago
        Given that the entire point of WMF is that they pay people to do stuff, it would be alarming if salary wasn’t their largest expense. Why would anyone donate if not to pay for them to work on stuff?
        QAPereo 9y ago
        Most of what they do is host the work of others, who are not paid. It seems more than a little obscene that a corps 300 employees are splitting 31 million, for the task of overseeing a 2mil hosting operation. They spend more on processing donations than hosting. They give away about 5x as much as they do on hosting. What value are these people adding beyond the cost of hosting the repository built by people who are not paid?
        revelation 9y ago
        I mean, that's what the complaint is about. People think when they donate it's to ensure the Wikipedia keeps running and is maintained. In reality most of that money is feeding an insatiable beast that has very little to do with "running Wikipedia".
        Alex3917 9y ago
        > Most of what they do is host the work of others, who are not paid. Have you actually read their annual report lately? I don't follow what they do at this point, but the last time I checked many of their contributors were paid.
        QAPereo 9y ago
        Unless they’re one of the 300 employees splitting that salary, winning awards, or under the ~3mil of specialist services.... If the latter, then they still account for a fraction of the budget. Bull. Shit. An operation which could be run for about 5 mil is running an order of magnitude higher for no discernible change in service.
        tonyztan 9y ago
jchw 9y ago
Since 2005, hosting has gotten much cheaper, and techniques for storing extremely large amounts of data have gotten much more efficient. But as far as I know, MediaWiki is no better positioned to begin taking advantage of those facts. It's still a PHP app accompanied by a huge, vertically scaling MySQL. There's no telling what might work better for Wikipedia, but there's plenty of avenues for experimentation. Maybe revision history could be more effectively stored in Amazon S3-like storage, instead of MySQL. Maybe not, but there's many such possible optimizations I doubt have ever seriously been considered.
- toomuchtodo 9y ago
  I think the savings would be in having an efficient core with lots of caching, similar to the Stackoverflow architecture. Push as much off to the CDN as possible, and dynamically reduce the cache time as edit velocity increases on a page (with cache time increasing again as edits fall off).
  - zrm 9y ago
    You can do even better than that with something read-mostly like Wikipedia. Have a bunch of caches (or a cache hierarchy) which request pages on first use and then cache them indefinitely. Then when a page actually changes, you push notify all the caches to immediately evict that page.
    - mpartel 9y ago
      Sounds error-prone if the notification ever fails to reach some subset of the caches. A long expiration time (e.g. several hours) is probably almost as good, but also "self-healing".
      - zrm 9y ago
        Giving out the wrong page for several hours is obviously better than giving out the wrong page forever, but it's still broken. Better would be to re-request the page after several hours and compare it to the cached copy. Then if they're different, update the cache, but also generate an alert to notify the administrator that something "impossible" happened and there is a bug somewhere that needs to be fixed.

by film42 9y ago

revelation 9y ago

You're right, page count is not the best metric. Instead you can just look up the hosting costs in their yearly report and find out it's a pittance compared to all their other spending and declining in relation.

QAPereo 9y ago
Could you link me to that?
- hk__2 9y ago
  https://upload.wikimedia.org/wikipedia/foundation/4/43/Wikimedia_Foundation_Audit_Report_-_FY15-16.pdf https://upload.wikimedia.org/wikipedia/foundation/4/43/Wikim... Page 3. In 2016 they spent $2M in hosting vs. e.g. $2.2M in "travel and conferences" and $3.6M in "donations processing expenses".
  - QAPereo 9y ago
    Well, shit. That’s horrendous, half of the money is just going to salaries.
    - Alex3917 9y ago
      Given that the entire point of WMF is that they pay people to do stuff, it would be alarming if salary wasn’t their largest expense. Why would anyone donate if not to pay for them to work on stuff?
      - QAPereo 9y ago
        Most of what they do is host the work of others, who are not paid. It seems more than a little obscene that a corps 300 employees are splitting 31 million, for the task of overseeing a 2mil hosting operation. They spend more on processing donations than hosting. They give away about 5x as much as they do on hosting. What value are these people adding beyond the cost of hosting the repository built by people who are not paid?
        revelation 9y ago
        I mean, that's what the complaint is about. People think when they donate it's to ensure the Wikipedia keeps running and is maintained. In reality most of that money is feeding an insatiable beast that has very little to do with "running Wikipedia".
        Alex3917 9y ago
        > Most of what they do is host the work of others, who are not paid. Have you actually read their annual report lately? I don't follow what they do at this point, but the last time I checked many of their contributors were paid.
        QAPereo 9y ago
        Unless they’re one of the 300 employees splitting that salary, winning awards, or under the ~3mil of specialist services.... If the latter, then they still account for a fraction of the budget. Bull. Shit. An operation which could be run for about 5 mil is running an order of magnitude higher for no discernible change in service.
        tonyztan 9y ago

jchw 9y ago

Since 2005, hosting has gotten much cheaper, and techniques for storing extremely large amounts of data have gotten much more efficient. But as far as I know, MediaWiki is no better positioned to begin taking advantage of those facts. It's still a PHP app accompanied by a huge, vertically scaling MySQL. There's no telling what might work better for Wikipedia, but there's plenty of avenues for experimentation. Maybe revision history could be more effectively stored in Amazon S3-like storage, instead of MySQL. Maybe not, but there's many such possible optimizations I doubt have ever seriously been considered.

toomuchtodo 9y ago
I think the savings would be in having an efficient core with lots of caching, similar to the Stackoverflow architecture. Push as much off to the CDN as possible, and dynamically reduce the cache time as edit velocity increases on a page (with cache time increasing again as edits fall off).
- zrm 9y ago
  You can do even better than that with something read-mostly like Wikipedia. Have a bunch of caches (or a cache hierarchy) which request pages on first use and then cache them indefinitely. Then when a page actually changes, you push notify all the caches to immediately evict that page.
  - mpartel 9y ago
    Sounds error-prone if the notification ever fails to reach some subset of the caches. A long expiration time (e.g. several hours) is probably almost as good, but also "self-healing".
    - zrm 9y ago
      Giving out the wrong page for several hours is obviously better than giving out the wrong page forever, but it's still broken. Better would be to re-request the page after several hours and compare it to the cached copy. Then if they're different, update the cache, but also generate an alert to notify the administrator that something "impossible" happened and there is a bug somewhere that needs to be fixed.