3 ms·

I'm curious what the strategy is to get companies like Google to pay for it. Is it "all carrots" or are there any "sticks"? Like maybe rate limiting certain c

by tyingq 5y ago

I'm curious what the strategy is to get companies like Google to pay for it. Is it "all carrots" or are there any "sticks"? Like maybe rate limiting certain consumers?

BiteCode_dev 5y ago
Rate limiting for all would be fair. You can torrent the whole db anyhow.
marginalia_nu 5y ago
They already don't want you to just scrape stuff off Wikipedia because mediawiki apparently doesn't do proper caching and does a stupidly expensive re-rendering of every page that is fetched. So the option you have is using stale data dumps. They're reasonably workable, I even host low-markup versions of every wikipedia article myself in my home lab and it's like some 100 Gb worth of compressed data, images excluded. Uncompressed I don't know where it ends up, but below 1 Tb at any rate. Manageable but very unwieldy. File systems like ext4 struggle with that number of files in a directory so you need to be a bit crafty, but it's not hard to get around. If all you want is an info box, you could probably squeeze it down to a couple of hundred gigabytes uncompressed. Definitely not something you need a data center for. The catch, as mentioned, is is that the data is a stale snapshot. The only way you are getting current data is on the main site, which may be a problem if you have ambitions of staying up to date. I personally don't mind. I figure they used to sell printed encyclopedias and that worked pretty well so having a 1 year old copy of Wikipedia isn't that bad. It's also a sort of weird domain. I can see how it might not be easy to host in the cloud in a way that's both affordable and performant. Since so many insist on cloud-based apps nowadays, there's probably a market for Wikipedia APIs with a good SLA.
- georgyo 5y ago
  > They already don't want you to just scrape stuff off Wikipedia because mediawiki apparently doesn't do proper caching and does a stupidly expensive re-rendering of every page that is fetched. It does cache pages very aggressively. And it even has solutions for hard problems in computer science, cache invalidation. Pages are cached indefinitely until something pokes it. Proof? Add ?action=purge to a wiki page.
  - marginalia_nu 5y ago
    Maybe I'm misinterpreting them then, they seem quite concerned about automated page visits. Like for example this: https://en.wikipedia.org/wiki/Wikipedia:Mirrors_and_forks#Remote_loading https://en.wikipedia.org/wiki/Wikipedia:Mirrors_and_forks#Re... and this: https://meta.wikimedia.org/wiki/Live_mirrors https://meta.wikimedia.org/wiki/Live_mirrors Something resulting in 50k page fetches of cached data in a day is nothing. A raspberry pi could handle 50k static documents in an hour.
    - hashar 5y ago
      Hi, I have been involved with the Wikimedia infrastructure since roughly 2004. The 50k daily hit on https://meta.wikimedia.org/wiki/Live_mirrors https://meta.wikimedia.org/wiki/Live_mirrors is merely to illustrate a web proxy can dramatically reduce the number of hits to the original site. They could have pick 42 or 7 billions, it is just an example. The text dates from 2009, and in April 2009 this is roughly what we had in term of servers https://commons.wikimedia.org/wiki/File:Wikimedia-servers-2009-04-05.svg https://commons.wikimedia.org/wiki/File:Wikimedia-servers-20... Arguably a little more than a Raspberry pi We already had a few million pages rather than 50k of them, and they were not static since pages get edited and made live as soon as one save their edit. Obviously, we already had million daily users and definitely requires a ton of caching all across the stack. Also good luck serving flat files that keeps being written too, that does not really work with hard drive and static files storage. In September 2021 we have served 21 billion pages and 73 billions media files https://stats.wikimedia.org/#/all-projects https://stats.wikimedia.org/#/all-projects . Almost all of them served primarily served from caches. That being said, the wiki pages you have mentioned are a decade old. We definitely had trouble with people having the smart idea of scrapping the whole website, often using either uncacheable content or hitting barely served articles or history that were not cached and thus caused a full rendering of the page. Even with the servers we had, a single user could cause major havoc on the infrastructure and keep a good part of them busy just to serve that single person. Hence the recommendation to use the database dump which are still used: https://dumps.wikimedia.org/ https://dumps.wikimedia.org/ cheers
- boublepop 5y ago
  > because mediawiki apparently doesn't do proper caching and does a stupidly expensive re-rendering of every page Don’t worry, they are hard at work at making workarounds for all their idiotic design decisions that seem to make absolutely no sense.. and those workarounds will be provided quite cheaply!! Reminds me of the saying: “ It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It” We can be quite sure that Wikipedias engineering will never improve if their income becomes dependent on providing workarounds for deficiencies in their solutions.
- nonameiguess 5y ago
  The NSA mirrors all of Wikipedia and Stack Overflow once every 24 hours from the public Internet to the classified Internet, so developers of classified applications can have references without needing to pivot to a separate unclassified workstation that needs to be at least six feet away. I never found it to be a big deal. Once a day updates were perfectly fine for reference usage.
Hamuko 5y ago
FWIW, Google is one of the main sponsors of MusicBrainz and the core MusicBrainz data is (effectively) in the public domain. https://musicbrainz.org/doc/About/Data_License https://musicbrainz.org/doc/About/Data_License https://metabrainz.org/sponsors https://metabrainz.org/sponsors
- H8crilA 5y ago
  Not only that it's one of the top donors to the Wikimedia Foundation.
  - tyingq 5y ago
    I mean, they should be. Google uses Wikipedia content extensively on search results. Google USED TO also have lots of prominent links that would drive traffic (and hence, donations) to Wikipedia. Over time, it has removed a lot of those links and replaced them with ones that lead to more Google hosted content. There's a distinct drop in Google-referred traffic to Wikipedia over time that's in stark contrast to the increased number of eyeballs on Wikipedia sourced content on Google's pages. It's not clear to me whether Google's sponsorship reflects the value they've gained. And, yes, I get that they are under no legal obligation to do anything at all, since the content license allows for all of it. But as with rich snippets, you can't just "take", or eventually, there won't be anything to "take".

by tyingq 5y ago

I'm curious what the strategy is to get companies like Google to pay for it. Is it "all carrots" or are there any "sticks"? Like maybe rate limiting certain consumers?

BiteCode_dev 5y ago

Rate limiting for all would be fair. You can torrent the whole db anyhow.

marginalia_nu 5y ago

They already don't want you to just scrape stuff off Wikipedia because mediawiki apparently doesn't do proper caching and does a stupidly expensive re-rendering of every page that is fetched. So the option you have is using stale data dumps. They're reasonably workable, I even host low-markup versions of every wikipedia article myself in my home lab and it's like some 100 Gb worth of compressed data, images excluded. Uncompressed I don't know where it ends up, but below 1 Tb at any rate. Manageable but very unwieldy. File systems like ext4 struggle with that number of files in a directory so you need to be a bit crafty, but it's not hard to get around. If all you want is an info box, you could probably squeeze it down to a couple of hundred gigabytes uncompressed. Definitely not something you need a data center for. The catch, as mentioned, is is that the data is a stale snapshot. The only way you are getting current data is on the main site, which may be a problem if you have ambitions of staying up to date. I personally don't mind. I figure they used to sell printed encyclopedias and that worked pretty well so having a 1 year old copy of Wikipedia isn't that bad. It's also a sort of weird domain. I can see how it might not be easy to host in the cloud in a way that's both affordable and performant. Since so many insist on cloud-based apps nowadays, there's probably a market for Wikipedia APIs with a good SLA.

georgyo 5y ago
> They already don't want you to just scrape stuff off Wikipedia because mediawiki apparently doesn't do proper caching and does a stupidly expensive re-rendering of every page that is fetched. It does cache pages very aggressively. And it even has solutions for hard problems in computer science, cache invalidation. Pages are cached indefinitely until something pokes it. Proof? Add ?action=purge to a wiki page.
- marginalia_nu 5y ago
  Maybe I'm misinterpreting them then, they seem quite concerned about automated page visits. Like for example this: https://en.wikipedia.org/wiki/Wikipedia:Mirrors_and_forks#Remote_loading https://en.wikipedia.org/wiki/Wikipedia:Mirrors_and_forks#Re... and this: https://meta.wikimedia.org/wiki/Live_mirrors https://meta.wikimedia.org/wiki/Live_mirrors Something resulting in 50k page fetches of cached data in a day is nothing. A raspberry pi could handle 50k static documents in an hour.
  - hashar 5y ago
    Hi, I have been involved with the Wikimedia infrastructure since roughly 2004. The 50k daily hit on https://meta.wikimedia.org/wiki/Live_mirrors https://meta.wikimedia.org/wiki/Live_mirrors is merely to illustrate a web proxy can dramatically reduce the number of hits to the original site. They could have pick 42 or 7 billions, it is just an example. The text dates from 2009, and in April 2009 this is roughly what we had in term of servers https://commons.wikimedia.org/wiki/File:Wikimedia-servers-2009-04-05.svg https://commons.wikimedia.org/wiki/File:Wikimedia-servers-20... Arguably a little more than a Raspberry pi We already had a few million pages rather than 50k of them, and they were not static since pages get edited and made live as soon as one save their edit. Obviously, we already had million daily users and definitely requires a ton of caching all across the stack. Also good luck serving flat files that keeps being written too, that does not really work with hard drive and static files storage. In September 2021 we have served 21 billion pages and 73 billions media files https://stats.wikimedia.org/#/all-projects https://stats.wikimedia.org/#/all-projects . Almost all of them served primarily served from caches. That being said, the wiki pages you have mentioned are a decade old. We definitely had trouble with people having the smart idea of scrapping the whole website, often using either uncacheable content or hitting barely served articles or history that were not cached and thus caused a full rendering of the page. Even with the servers we had, a single user could cause major havoc on the infrastructure and keep a good part of them busy just to serve that single person. Hence the recommendation to use the database dump which are still used: https://dumps.wikimedia.org/ https://dumps.wikimedia.org/ cheers
boublepop 5y ago
> because mediawiki apparently doesn't do proper caching and does a stupidly expensive re-rendering of every page Don’t worry, they are hard at work at making workarounds for all their idiotic design decisions that seem to make absolutely no sense.. and those workarounds will be provided quite cheaply!! Reminds me of the saying: “ It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It” We can be quite sure that Wikipedias engineering will never improve if their income becomes dependent on providing workarounds for deficiencies in their solutions.
nonameiguess 5y ago
The NSA mirrors all of Wikipedia and Stack Overflow once every 24 hours from the public Internet to the classified Internet, so developers of classified applications can have references without needing to pivot to a separate unclassified workstation that needs to be at least six feet away. I never found it to be a big deal. Once a day updates were perfectly fine for reference usage.

Hamuko 5y ago

FWIW, Google is one of the main sponsors of MusicBrainz and the core MusicBrainz data is (effectively) in the public domain. https://musicbrainz.org/doc/About/Data_License https://musicbrainz.org/doc/About/Data_License https://metabrainz.org/sponsors https://metabrainz.org/sponsors

H8crilA 5y ago
Not only that it's one of the top donors to the Wikimedia Foundation.
- tyingq 5y ago
  I mean, they should be. Google uses Wikipedia content extensively on search results. Google USED TO also have lots of prominent links that would drive traffic (and hence, donations) to Wikipedia. Over time, it has removed a lot of those links and replaced them with ones that lead to more Google hosted content. There's a distinct drop in Google-referred traffic to Wikipedia over time that's in stark contrast to the increased number of eyeballs on Wikipedia sourced content on Google's pages. It's not clear to me whether Google's sponsorship reflects the value they've gained. And, yes, I get that they are under no legal obligation to do anything at all, since the content license allows for all of it. But as with rich snippets, you can't just "take", or eventually, there won't be anything to "take".