10 ms·

Wikimedia Enterprise: paid commercial product for reusing Wikipedia content

99 points by rvnx 5y ago

dredmorbius 5y ago
Related discussion from 7 months ago announcing API: https://news.ycombinator.com/item?id=26484080 https://news.ycombinator.com/item?id=26484080
dredmorbius 5y ago
https://nitter.kavin.rocks/Wikimedia/status/1452686760111185924 https://nitter.kavin.rocks/Wikimedia/status/1452686760111185... Principal link is: https://wikimediafoundation.org/news/2021/10/25/wikimedia-foundation-launches-wikimedia-enterprise-the-new-opt-in-product-for-companies-and-organizations-to-easily-reuse-content-from-wikipedia-and-wikimedia-projects/ https://wikimediafoundation.org/news/2021/10/25/wikimedia-fo...
- dang 5y ago
  Changed from https://twitter.com/Wikimedia/status/1452686760111185924 https://twitter.com/Wikimedia/status/1452686760111185924. Thanks!
tyingq 5y ago
I'm curious what the strategy is to get companies like Google to pay for it. Is it "all carrots" or are there any "sticks"? Like maybe rate limiting certain consumers?
- BiteCode_dev 5y ago
  Rate limiting for all would be fair. You can torrent the whole db anyhow.
- marginalia_nu 5y ago
  They already don't want you to just scrape stuff off Wikipedia because mediawiki apparently doesn't do proper caching and does a stupidly expensive re-rendering of every page that is fetched. So the option you have is using stale data dumps. They're reasonably workable, I even host low-markup versions of every wikipedia article myself in my home lab and it's like some 100 Gb worth of compressed data, images excluded. Uncompressed I don't know where it ends up, but below 1 Tb at any rate. Manageable but very unwieldy. File systems like ext4 struggle with that number of files in a directory so you need to be a bit crafty, but it's not hard to get around. If all you want is an info box, you could probably squeeze it down to a couple of hundred gigabytes uncompressed. Definitely not something you need a data center for. The catch, as mentioned, is is that the data is a stale snapshot. The only way you are getting current data is on the main site, which may be a problem if you have ambitions of staying up to date. I personally don't mind. I figure they used to sell printed encyclopedias and that worked pretty well so having a 1 year old copy of Wikipedia isn't that bad. It's also a sort of weird domain. I can see how it might not be easy to host in the cloud in a way that's both affordable and performant. Since so many insist on cloud-based apps nowadays, there's probably a market for Wikipedia APIs with a good SLA.
  - georgyo 5y ago
    > They already don't want you to just scrape stuff off Wikipedia because mediawiki apparently doesn't do proper caching and does a stupidly expensive re-rendering of every page that is fetched. It does cache pages very aggressively. And it even has solutions for hard problems in computer science, cache invalidation. Pages are cached indefinitely until something pokes it. Proof? Add ?action=purge to a wiki page.
    - marginalia_nu 5y ago
      Maybe I'm misinterpreting them then, they seem quite concerned about automated page visits. Like for example this: https://en.wikipedia.org/wiki/Wikipedia:Mirrors_and_forks#Remote_loading https://en.wikipedia.org/wiki/Wikipedia:Mirrors_and_forks#Re... and this: https://meta.wikimedia.org/wiki/Live_mirrors https://meta.wikimedia.org/wiki/Live_mirrors Something resulting in 50k page fetches of cached data in a day is nothing. A raspberry pi could handle 50k static documents in an hour.
- Hamuko 5y ago
  FWIW, Google is one of the main sponsors of MusicBrainz and the core MusicBrainz data is (effectively) in the public domain. https://musicbrainz.org/doc/About/Data_License https://musicbrainz.org/doc/About/Data_License https://metabrainz.org/sponsors https://metabrainz.org/sponsors
  - H8crilA 5y ago
    Not only that it's one of the top donors to the Wikimedia Foundation.
    - tyingq 5y ago
      I mean, they should be. Google uses Wikipedia content extensively on search results. Google USED TO also have lots of prominent links that would drive traffic (and hence, donations) to Wikipedia. Over time, it has removed a lot of those links and replaced them with ones that lead to more Google hosted content. There's a distinct drop in Google-referred traffic to Wikipedia over time that's in stark contrast to the increased number of eyeballs on Wikipedia sourced content on Google's pages. It's not clear to me whether Google's sponsorship reflects the value they've gained. And, yes, I get that they are under no legal obligation to do anything at all, since the content license allows for all of it. But as with rich snippets, you can't just "take", or eventually, there won't be anything to "take".
mmaunder 5y ago
English Wikipedia is 20 gigs which takes a few mins to download on a fiber connection. How is this product compelling without Wikipedia actively throttling downloads and access to unpaid users?
- Zhyl 5y ago
  From the article: >In most cases, commercial entities that reuse Wikimedia content at a high volume have product, service, and system requirements that go beyond what Wikimedia freely provides through publicly available APIs and data dumps.
- shock-value 5y ago
  I don't know if this initiative includes Wikimedia Commons but I'm sure they've got substantially more data than that, and higher typical bandwidth usage. (https://commons.wikimedia.org/wiki/Main_Page https://commons.wikimedia.org/wiki/Main_Page)
  - mgdlbp 5y ago
    Files hosted on Commons currently total just under 320 TB (excluding old versions, which are retained). https://commons.wikimedia.org/wiki/Special:MediaStatistics https://commons.wikimedia.org/wiki/Special:MediaStatistics This MediaWiki special page is also available for other wikis; the English Wikipedia itself hosts 152 GB of files that were uploaded there instead of to Commons for one reason or another (e.g. non-free material under fair use).
- IAmNotAFix 5y ago
  Wikipedia averages 598 new articles per day et 2 edits per second. https://en.wikipedia.org/wiki/Wikipedia:Statistics https://en.wikipedia.org/wiki/Wikipedia:Statistics
  - boublepop 5y ago
    Sure but you only get access to 10 of them unless you pay for Wikipedia premium. /comment from slightly in the future
londons_explore 5y ago
I've got a feeling it won't be long before wikimedia stop making database dumps available, and throttling people crawling the website... A bit of a shame really!
- infofarmer 5y ago
  Doesn’t have to be that way. For a much smaller, but relevant example - see MusicBrainz / MetaBrainz https://metabrainz.org/supporters/account-type https://metabrainz.org/supporters/account-type Wikimedia is simply productizing commercial access to make it easier for enterprises to support it in a commercially justifiable way (vs pure charity) + setting up processes to improve enterprise-only features.
londons_explore 5y ago
> This will not impact smaller content reusers who can continue to leverage Wikimedia’s data dumps and APIs freely for their own use. This is another way to say "if you are a big business, we're going to block you downloading these unless you pay". Won't be long before those data dumps require a "free account" or captcha to download...
- afandian 5y ago
  Seems like an odd speculation. Can you provide any evidence for that?
webmobdev 5y ago
This is quite confusing. According to Wikipedia itself: > Most text in Wikipedia, excluding quotations, has been released under the Creative Commons Attribution-Sharealike 3.0 Unported License (CC-BY-SA) and the GNU Free Documentation License (GFDL) (unversioned, with no invariant sections, front-cover texts, or back-cover texts) and can therefore be reused only if you release any derived work under the Creative Commons Attribution/Share-Alike License or the GFDL. This requires that, among other things, you attribute the authors and allow others to freely copy your work ... > Some text has been imported only under CC-BY-SA and CC-BY-SA-compatible licenses and cannot be reused under GFDL; such text will be identified either on the page footer, in the page history or the discussion page of the article that utilizes the text. All text published before June 15th, 2009 on Wikipedia was released under the GFDL, and you may also use the page history to retrieve content published before that date to ensure GFDL compatibility. > If you are unwilling or unable to use the Creative Commons Attribution/Share-Alike License or the GNU Free Documentation License for your work, use of Wikipedia content is unauthorized. Source: https://en.wikipedia.org/wiki/Wikipedia:FAQ/Copyright#Can_I_reuse_Wikipedia's_content_somewhere_else https://en.wikipedia.org/wiki/Wikipedia:FAQ/Copyright#Can_I_...? So what exactly is Wikipedia offering here? They can't change the copyright. They still have to be attribued. And any deriavative using such content will still have to be released under the original viral license. When anyone can download the whole of Wikipedia, with adequate bandwidth and storage space, what advantages does this service really offer?
- IAmNotAFix 5y ago
  I think it's more the way you access the information than the information itself.
softwaredoug 5y ago
Good. So much of the recent progress in natural language understanding in commercial products like Siri and Alexa is due to the richness and structure in Wikipedia and wiki data. Big tech should be paying for this stuff…
- boublepop 5y ago
  > Big tech should be paying for this stuff… I don’t see anything about anyone paying the content creators. What fairness is it to har one platform currently freeloading of of the work that another platform is freeloading of its users (while begging them for donations mind you) suddenly have to pay the first freeloading platform? I can’t speak for others naturally, but every vontribution I have made to Wikipedia has been made on the basis of contributing you our common knowledge base, not to try to provide a free service for a profit driven corporation.
  - arsome 5y ago
    This is still being run by the non-profit, so I suspect that these can go into ongoing Wikimedia operating costs. They're incredibly efficient for the scale they operate at and I'd say this might improve things even further. The less reliance on giant donation banners, the better.
    - jon-wood 5y ago
      Wikipedia doesn’t rely on giant donation banners - the Wikipedia foundation had $180M dollars available at the end of 2020, and an annual hosting cost of $2M/year. The vast majority of there expenses are $54M/year of salaries, which I doubt are focused on Wikipedia itself. If you want a charitable cause to donate to find one that needs the money.
      - bawolff 5y ago
        Ah yes, all those servers can just putter off by themselves without any attention at all. For a site called hacker news, its a wonder that people think system admins are a superfluous job, totally unnessary at scale* * and yes im aware that wmf employees a lot of people, most aren't sysadmins, and some are more critical than others. I just think its rediculous people think that servers just run themselves and hosting is the only core cost.
sam0x17 5y ago
So as a script kiddie / tween I used keyword-related wikipedia content to get my domain parking network to pass Google's "must have content" requirement (that is until Google changed their TOS at midnight one day specifically to ban users like myself). What is an example of an actually legitimate use of something like this, and in particular, a use-case that wouldn't be properly served by simply downloading the current data archive on a ~daily or ~hourly basis as people tend to, or just by straight-up crawling as I did back in the day?
- treesknees 5y ago
  As someone else mentioned in a commment, Wikipedia averages about 2 edits per second and almost 600 new articles daily. A legitimate use of something like this is having Google search results lookup Wikipedia pages more often and in a timely manner. As the article states, it also provides SLAs for the APIs/service. So let's for example say Big Movie Star X dies, but the free API for downloading content is down for the day for maintenance, well now Google's flagship search product is serving stale/outdated information. This product provides businesses a means of reliably depending on an updating source of data.
nickvincent 5y ago
Worth emphasizing that this is described as basically just adding new ways to consume Wikipedia at scale, not taking anything away (not that that's impossible in the future, but unlikely IMO). In theory, systems that use a lot of Wikipedia data (search engines, "AI" assistant/question answering systems, language models) can keep doing what they're doing with Wikipedia dumps, but new APIs could make these services better (e.g., faster reaction time to current events). A big upside of testing this out is the opportunity to keep better track of what services are Wikipedia-dependent. While search results often send people to Wikipedia (see e.g. https://diff.wikimedia.org/2021/09/23/searching-for-wikipedia-duckduckgo-and-the-wikimedia-foundation-share-new-research-on-how-people-use-search-engines-to-get-to-wikipedia/ https://diff.wikimedia.org/2021/09/23/searching-for-wikipedi...), voice assistants or Q&A systems might be (i.e. likely are) surfacing Wikipedia content in more subtle fashion. Same with large language models. Under status quo, it will be increasingly hard to track these relationships, so exploring alternatives is well worth it.
- chaps 5y ago
  This probably won't do that much on that side of things -- it'd be trivial to mask relationships through proxy orgs and such.
achn 5y ago
Rant: Wikipedia would get my donations easily if they simply included a couple of line items on what they are actually doing to improve Wikipedia! How is Wikipedia’s focus not on building semantic context into their pages? Or adding topical prerequisites? Or any number of meaningful features? Or is their goal to just exist in the present form?
- renewiltord 5y ago
  It’s likely that proving this to you to get your donation (and that of similar people) will cost more than it will bring in.
  - achn 5y ago
    Do you mean in the ad-effectiveness sense that many donors wouldn’t care or would be lost?
    - renewiltord 5y ago
      Yeah I suspect RoI will be negative
      - achn 5y ago
        Perhaps, but if there was ever an ad target that could be categorically labeled as “knowledge seekers” it should be Wikipedia donors.
- colechristensen 5y ago
  I would prefer they not "improve", especially so in order to sell donations. Wikipedia has largely stayed the same since its inception and that is part of its power. So many things have "improved" themselves into irrelevance, or wasted enormous amounts of resources changing things that don't need change. There is a deep pit of improving things to satisfy the loudest people who "want" and not realizing until far too late that you're optimizing to a loud majority while your overall appeal shrinks to nothing. (Think of it like a grocery store dropping the least popular half of its products every few months... eventually they'd be left with light beer and ketchup confused about why nobody comes by any more)
human 5y ago
This is an interesting (and ethical) way for Wikimedia to finance their operations. I have to say I find it funny that back in the days teachers told us not to use Wikipedia and now we’ll have large corporations paying to train their algos using their data. And then those algos will decide whether you get a loan or not, your insurance premiums, and much more.
- a_e_k 5y ago
  Back in grad school I had a well-respected tenured prof point out that Wikipedia was great for looking up immutable and uncontroversial technical things (e.g., a list of refractive indices) and could even be good as a starting point for learning about a topic to research it. You just can't stop there.
- staticman2 5y ago
  You think loan decisions and insurance rates will be made based on a computer program trained on a Wikipedia page? That's... not how any of that works.
- IshKebab 5y ago
  Wikipedia doesn't need this to finance their operations. They have so much money from donations they don't know what to do with it. If they had used it wisely they could have built a large enough endowment to be self-sustaining by now, but unfortunately they didn't.
1cvmask 5y ago
There are deliberate distortions in Wikipedia especially in regards to the national surveillance state. Companies should be aware that they might be paying for misinformation and disinformation. https://news.ycombinator.com/item?id=28808516 https://news.ycombinator.com/item?id=28808516 https://www.thecanary.co/discovery/analysis-discovery/2018/05/23/the-philip-cross-scandal-how-wikipedia-is-being-used-against-left-wing-journalists/ https://www.thecanary.co/discovery/analysis-discovery/2018/0... https://www.bbc.com/news/blogs-trending-44495696 https://www.bbc.com/news/blogs-trending-44495696
- RogerSamson 5y ago
  I am currently researching an long-term editor/abuser who has a rather colorful history. He runs a couple of attack websites targeting universities both in the US and Switzerland. These are current revisions to a past couple of websites which still appear as sources in numerous articles on Wikipedia. He supposedly volunteered at one of the universities before he was dropped because he lied about his credentials. He ran in the 2018 Italian elections. His political party was suggested by some Italian and American sources as a fraud. He had tried for several years to inject articles and tidbits about the party into Wikipedia. The sources which negatively reported on his political party have often been attacked by him. He managed to get one online newspaper's article removed from en-wiki and it-wiki. He appeared to do this for SEO impact to push up sources which are good about him and hide those which are not. He recently tried to incorporate a dating website promoting himself and his search for a wife. He tried to present the site as a "mirror" of Wikipedia. He used to run a service to circumvent anti-plagiarism tools like Turnitin. He did this while volunteering at the university mentioned above. There are several places where his service and his LinkedIn stories come up in the history of Wikipedia articles. He wrote a book about Second Life and a couple of faux academic articles. He tried several times to integrate his book as a source on it-wiki. The faux academic articles were uploaded to Commons, and they were even cited in a few legit academic publications. He operated a school in Second Life. He created articles about the school on Wikipedia, and one of the faux academic articles was about it. These were eventually discovered and removed. He also created several articles about himself on en-wiki, it-wiki, simple-wiki, and ja-wiki. These have all been removed because several of his accounts had been blocked for sockpuppetry. The story grows crazier the I dig deeper.
trhway 5y ago
Breaking net neutrality is bad while doing that to Wikipedia is good. How come?
gojomo 5y ago
Wikimedia should also let users opt-into (small, tasteful) ads instead of fundraising appeals - a way to contribute indirectly, rather than an explicit donation.
- cogburnd02 5y ago
  > (small, tasteful) ads These did used to exist. They were called 120x60 px buttons.
  - gojomo 5y ago
    The original AdWords & AdSense pure-text formats weren't so bad, either.
kbsspl 5y ago
Was just thinking about exactly the whole Wiki ecosystem. It's created by individuals like us and should be available via maybe archive.org as a complete dump for individuals to play with. Are they available ?
- humanistbot 5y ago
  https://dumps.wikimedia.org https://dumps.wikimedia.org
  - kbsspl 5y ago
    Thank you. Will explore.
smsm42 5y ago
I must admit to a mixed feeling about it. On one side, a lot of large companies already are using Wikipedia data at large scale, already in a disorganized manner and not giving back anything (well, sometimes giving back donations but not in any relation to their use). Creating a venue for proper large-scale use that has a channel to give back money looks good. On the other hand, a lot of Wikipedia is created by many, many hours of work of many, many volunteers, who aren't (usually) paid a dime for it. Capitalizing on that, and having clients that pay $big_money for the result, creates misaligned incentives, and however noble and pure the motives of everybody are now, incentives always matter and always create pressure. What if some company says "we'll pay millions of dollars, but you're got to remove some of the worst examples of content we don't like - it's not really much, compared to the whole mass of information, nobody would even notice it!" Right now I don't think the community and Wikimedia would agree, but what would happen when they're used to the cash inflow that this program creates and grown to depend on it?

99 points by rvnx 5y ago

Related discussion from 7 months ago announcing API: https://news.ycombinator.com/item?id=26484080 https://news.ycombinator.com/item?id=26484080

dredmorbius 5y ago

https://nitter.kavin.rocks/Wikimedia/status/1452686760111185924 https://nitter.kavin.rocks/Wikimedia/status/1452686760111185... Principal link is: https://wikimediafoundation.org/news/2021/10/25/wikimedia-foundation-launches-wikimedia-enterprise-the-new-opt-in-product-for-companies-and-organizations-to-easily-reuse-content-from-wikipedia-and-wikimedia-projects/ https://wikimediafoundation.org/news/2021/10/25/wikimedia-fo...

dang 5y ago
Changed from https://twitter.com/Wikimedia/status/1452686760111185924 https://twitter.com/Wikimedia/status/1452686760111185924. Thanks!

tyingq 5y ago

I'm curious what the strategy is to get companies like Google to pay for it. Is it "all carrots" or are there any "sticks"? Like maybe rate limiting certain consumers?

BiteCode_dev 5y ago
Rate limiting for all would be fair. You can torrent the whole db anyhow.
marginalia_nu 5y ago
They already don't want you to just scrape stuff off Wikipedia because mediawiki apparently doesn't do proper caching and does a stupidly expensive re-rendering of every page that is fetched. So the option you have is using stale data dumps. They're reasonably workable, I even host low-markup versions of every wikipedia article myself in my home lab and it's like some 100 Gb worth of compressed data, images excluded. Uncompressed I don't know where it ends up, but below 1 Tb at any rate. Manageable but very unwieldy. File systems like ext4 struggle with that number of files in a directory so you need to be a bit crafty, but it's not hard to get around. If all you want is an info box, you could probably squeeze it down to a couple of hundred gigabytes uncompressed. Definitely not something you need a data center for. The catch, as mentioned, is is that the data is a stale snapshot. The only way you are getting current data is on the main site, which may be a problem if you have ambitions of staying up to date. I personally don't mind. I figure they used to sell printed encyclopedias and that worked pretty well so having a 1 year old copy of Wikipedia isn't that bad. It's also a sort of weird domain. I can see how it might not be easy to host in the cloud in a way that's both affordable and performant. Since so many insist on cloud-based apps nowadays, there's probably a market for Wikipedia APIs with a good SLA.
- georgyo 5y ago
  > They already don't want you to just scrape stuff off Wikipedia because mediawiki apparently doesn't do proper caching and does a stupidly expensive re-rendering of every page that is fetched. It does cache pages very aggressively. And it even has solutions for hard problems in computer science, cache invalidation. Pages are cached indefinitely until something pokes it. Proof? Add ?action=purge to a wiki page.
  - marginalia_nu 5y ago
    Maybe I'm misinterpreting them then, they seem quite concerned about automated page visits. Like for example this: https://en.wikipedia.org/wiki/Wikipedia:Mirrors_and_forks#Remote_loading https://en.wikipedia.org/wiki/Wikipedia:Mirrors_and_forks#Re... and this: https://meta.wikimedia.org/wiki/Live_mirrors https://meta.wikimedia.org/wiki/Live_mirrors Something resulting in 50k page fetches of cached data in a day is nothing. A raspberry pi could handle 50k static documents in an hour.
Hamuko 5y ago
FWIW, Google is one of the main sponsors of MusicBrainz and the core MusicBrainz data is (effectively) in the public domain. https://musicbrainz.org/doc/About/Data_License https://musicbrainz.org/doc/About/Data_License https://metabrainz.org/sponsors https://metabrainz.org/sponsors
- H8crilA 5y ago
  Not only that it's one of the top donors to the Wikimedia Foundation.
  - tyingq 5y ago
    I mean, they should be. Google uses Wikipedia content extensively on search results. Google USED TO also have lots of prominent links that would drive traffic (and hence, donations) to Wikipedia. Over time, it has removed a lot of those links and replaced them with ones that lead to more Google hosted content. There's a distinct drop in Google-referred traffic to Wikipedia over time that's in stark contrast to the increased number of eyeballs on Wikipedia sourced content on Google's pages. It's not clear to me whether Google's sponsorship reflects the value they've gained. And, yes, I get that they are under no legal obligation to do anything at all, since the content license allows for all of it. But as with rich snippets, you can't just "take", or eventually, there won't be anything to "take".

mmaunder 5y ago

English Wikipedia is 20 gigs which takes a few mins to download on a fiber connection. How is this product compelling without Wikipedia actively throttling downloads and access to unpaid users?

Zhyl 5y ago
From the article: >In most cases, commercial entities that reuse Wikimedia content at a high volume have product, service, and system requirements that go beyond what Wikimedia freely provides through publicly available APIs and data dumps.
shock-value 5y ago
I don't know if this initiative includes Wikimedia Commons but I'm sure they've got substantially more data than that, and higher typical bandwidth usage. (https://commons.wikimedia.org/wiki/Main_Page https://commons.wikimedia.org/wiki/Main_Page)
- mgdlbp 5y ago
  Files hosted on Commons currently total just under 320 TB (excluding old versions, which are retained). https://commons.wikimedia.org/wiki/Special:MediaStatistics https://commons.wikimedia.org/wiki/Special:MediaStatistics This MediaWiki special page is also available for other wikis; the English Wikipedia itself hosts 152 GB of files that were uploaded there instead of to Commons for one reason or another (e.g. non-free material under fair use).
IAmNotAFix 5y ago
Wikipedia averages 598 new articles per day et 2 edits per second. https://en.wikipedia.org/wiki/Wikipedia:Statistics https://en.wikipedia.org/wiki/Wikipedia:Statistics
- boublepop 5y ago
  Sure but you only get access to 10 of them unless you pay for Wikipedia premium. /comment from slightly in the future

londons_explore 5y ago

I've got a feeling it won't be long before wikimedia stop making database dumps available, and throttling people crawling the website... A bit of a shame really!

infofarmer 5y ago
Doesn’t have to be that way. For a much smaller, but relevant example - see MusicBrainz / MetaBrainz https://metabrainz.org/supporters/account-type https://metabrainz.org/supporters/account-type Wikimedia is simply productizing commercial access to make it easier for enterprises to support it in a commercially justifiable way (vs pure charity) + setting up processes to improve enterprise-only features.

londons_explore 5y ago

> This will not impact smaller content reusers who can continue to leverage Wikimedia’s data dumps and APIs freely for their own use. This is another way to say "if you are a big business, we're going to block you downloading these unless you pay". Won't be long before those data dumps require a "free account" or captcha to download...

afandian 5y ago
Seems like an odd speculation. Can you provide any evidence for that?

webmobdev 5y ago

This is quite confusing. According to Wikipedia itself: > Most text in Wikipedia, excluding quotations, has been released under the Creative Commons Attribution-Sharealike 3.0 Unported License (CC-BY-SA) and the GNU Free Documentation License (GFDL) (unversioned, with no invariant sections, front-cover texts, or back-cover texts) and can therefore be reused only if you release any derived work under the Creative Commons Attribution/Share-Alike License or the GFDL. This requires that, among other things, you attribute the authors and allow others to freely copy your work ... > Some text has been imported only under CC-BY-SA and CC-BY-SA-compatible licenses and cannot be reused under GFDL; such text will be identified either on the page footer, in the page history or the discussion page of the article that utilizes the text. All text published before June 15th, 2009 on Wikipedia was released under the GFDL, and you may also use the page history to retrieve content published before that date to ensure GFDL compatibility. > If you are unwilling or unable to use the Creative Commons Attribution/Share-Alike License or the GNU Free Documentation License for your work, use of Wikipedia content is unauthorized. Source: https://en.wikipedia.org/wiki/Wikipedia:FAQ/Copyright#Can_I_reuse_Wikipedia's_content_somewhere_else https://en.wikipedia.org/wiki/Wikipedia:FAQ/Copyright#Can_I_...? So what exactly is Wikipedia offering here? They can't change the copyright. They still have to be attribued. And any deriavative using such content will still have to be released under the original viral license. When anyone can download the whole of Wikipedia, with adequate bandwidth and storage space, what advantages does this service really offer?

IAmNotAFix 5y ago
I think it's more the way you access the information than the information itself.

softwaredoug 5y ago

Good. So much of the recent progress in natural language understanding in commercial products like Siri and Alexa is due to the richness and structure in Wikipedia and wiki data. Big tech should be paying for this stuff…

boublepop 5y ago
> Big tech should be paying for this stuff… I don’t see anything about anyone paying the content creators. What fairness is it to har one platform currently freeloading of of the work that another platform is freeloading of its users (while begging them for donations mind you) suddenly have to pay the first freeloading platform? I can’t speak for others naturally, but every vontribution I have made to Wikipedia has been made on the basis of contributing you our common knowledge base, not to try to provide a free service for a profit driven corporation.
- arsome 5y ago
  This is still being run by the non-profit, so I suspect that these can go into ongoing Wikimedia operating costs. They're incredibly efficient for the scale they operate at and I'd say this might improve things even further. The less reliance on giant donation banners, the better.
  - jon-wood 5y ago
    Wikipedia doesn’t rely on giant donation banners - the Wikipedia foundation had $180M dollars available at the end of 2020, and an annual hosting cost of $2M/year. The vast majority of there expenses are $54M/year of salaries, which I doubt are focused on Wikipedia itself. If you want a charitable cause to donate to find one that needs the money.
    - bawolff 5y ago
      Ah yes, all those servers can just putter off by themselves without any attention at all. For a site called hacker news, its a wonder that people think system admins are a superfluous job, totally unnessary at scale* * and yes im aware that wmf employees a lot of people, most aren't sysadmins, and some are more critical than others. I just think its rediculous people think that servers just run themselves and hosting is the only core cost.

sam0x17 5y ago

So as a script kiddie / tween I used keyword-related wikipedia content to get my domain parking network to pass Google's "must have content" requirement (that is until Google changed their TOS at midnight one day specifically to ban users like myself). What is an example of an actually legitimate use of something like this, and in particular, a use-case that wouldn't be properly served by simply downloading the current data archive on a ~daily or ~hourly basis as people tend to, or just by straight-up crawling as I did back in the day?

treesknees 5y ago
As someone else mentioned in a commment, Wikipedia averages about 2 edits per second and almost 600 new articles daily. A legitimate use of something like this is having Google search results lookup Wikipedia pages more often and in a timely manner. As the article states, it also provides SLAs for the APIs/service. So let's for example say Big Movie Star X dies, but the free API for downloading content is down for the day for maintenance, well now Google's flagship search product is serving stale/outdated information. This product provides businesses a means of reliably depending on an updating source of data.

nickvincent 5y ago

Worth emphasizing that this is described as basically just adding new ways to consume Wikipedia at scale, not taking anything away (not that that's impossible in the future, but unlikely IMO). In theory, systems that use a lot of Wikipedia data (search engines, "AI" assistant/question answering systems, language models) can keep doing what they're doing with Wikipedia dumps, but new APIs could make these services better (e.g., faster reaction time to current events). A big upside of testing this out is the opportunity to keep better track of what services are Wikipedia-dependent. While search results often send people to Wikipedia (see e.g. https://diff.wikimedia.org/2021/09/23/searching-for-wikipedia-duckduckgo-and-the-wikimedia-foundation-share-new-research-on-how-people-use-search-engines-to-get-to-wikipedia/ https://diff.wikimedia.org/2021/09/23/searching-for-wikipedi...), voice assistants or Q&A systems might be (i.e. likely are) surfacing Wikipedia content in more subtle fashion. Same with large language models. Under status quo, it will be increasingly hard to track these relationships, so exploring alternatives is well worth it.

chaps 5y ago
This probably won't do that much on that side of things -- it'd be trivial to mask relationships through proxy orgs and such.

achn 5y ago

Rant: Wikipedia would get my donations easily if they simply included a couple of line items on what they are actually doing to improve Wikipedia! How is Wikipedia’s focus not on building semantic context into their pages? Or adding topical prerequisites? Or any number of meaningful features? Or is their goal to just exist in the present form?

renewiltord 5y ago
It’s likely that proving this to you to get your donation (and that of similar people) will cost more than it will bring in.
- achn 5y ago
  Do you mean in the ad-effectiveness sense that many donors wouldn’t care or would be lost?
  - renewiltord 5y ago
    Yeah I suspect RoI will be negative
    - achn 5y ago
      Perhaps, but if there was ever an ad target that could be categorically labeled as “knowledge seekers” it should be Wikipedia donors.
colechristensen 5y ago
I would prefer they not "improve", especially so in order to sell donations. Wikipedia has largely stayed the same since its inception and that is part of its power. So many things have "improved" themselves into irrelevance, or wasted enormous amounts of resources changing things that don't need change. There is a deep pit of improving things to satisfy the loudest people who "want" and not realizing until far too late that you're optimizing to a loud majority while your overall appeal shrinks to nothing. (Think of it like a grocery store dropping the least popular half of its products every few months... eventually they'd be left with light beer and ketchup confused about why nobody comes by any more)

human 5y ago

This is an interesting (and ethical) way for Wikimedia to finance their operations. I have to say I find it funny that back in the days teachers told us not to use Wikipedia and now we’ll have large corporations paying to train their algos using their data. And then those algos will decide whether you get a loan or not, your insurance premiums, and much more.

a_e_k 5y ago
Back in grad school I had a well-respected tenured prof point out that Wikipedia was great for looking up immutable and uncontroversial technical things (e.g., a list of refractive indices) and could even be good as a starting point for learning about a topic to research it. You just can't stop there.
staticman2 5y ago
You think loan decisions and insurance rates will be made based on a computer program trained on a Wikipedia page? That's... not how any of that works.
IshKebab 5y ago
Wikipedia doesn't need this to finance their operations. They have so much money from donations they don't know what to do with it. If they had used it wisely they could have built a large enough endowment to be self-sustaining by now, but unfortunately they didn't.

1cvmask 5y ago

There are deliberate distortions in Wikipedia especially in regards to the national surveillance state. Companies should be aware that they might be paying for misinformation and disinformation. https://news.ycombinator.com/item?id=28808516 https://news.ycombinator.com/item?id=28808516 https://www.thecanary.co/discovery/analysis-discovery/2018/05/23/the-philip-cross-scandal-how-wikipedia-is-being-used-against-left-wing-journalists/ https://www.thecanary.co/discovery/analysis-discovery/2018/0... https://www.bbc.com/news/blogs-trending-44495696 https://www.bbc.com/news/blogs-trending-44495696

RogerSamson 5y ago
I am currently researching an long-term editor/abuser who has a rather colorful history. He runs a couple of attack websites targeting universities both in the US and Switzerland. These are current revisions to a past couple of websites which still appear as sources in numerous articles on Wikipedia. He supposedly volunteered at one of the universities before he was dropped because he lied about his credentials. He ran in the 2018 Italian elections. His political party was suggested by some Italian and American sources as a fraud. He had tried for several years to inject articles and tidbits about the party into Wikipedia. The sources which negatively reported on his political party have often been attacked by him. He managed to get one online newspaper's article removed from en-wiki and it-wiki. He appeared to do this for SEO impact to push up sources which are good about him and hide those which are not. He recently tried to incorporate a dating website promoting himself and his search for a wife. He tried to present the site as a "mirror" of Wikipedia. He used to run a service to circumvent anti-plagiarism tools like Turnitin. He did this while volunteering at the university mentioned above. There are several places where his service and his LinkedIn stories come up in the history of Wikipedia articles. He wrote a book about Second Life and a couple of faux academic articles. He tried several times to integrate his book as a source on it-wiki. The faux academic articles were uploaded to Commons, and they were even cited in a few legit academic publications. He operated a school in Second Life. He created articles about the school on Wikipedia, and one of the faux academic articles was about it. These were eventually discovered and removed. He also created several articles about himself on en-wiki, it-wiki, simple-wiki, and ja-wiki. These have all been removed because several of his accounts had been blocked for sockpuppetry. The story grows crazier the I dig deeper.

trhway 5y ago

Breaking net neutrality is bad while doing that to Wikipedia is good. How come?

gojomo 5y ago

Wikimedia should also let users opt-into (small, tasteful) ads instead of fundraising appeals - a way to contribute indirectly, rather than an explicit donation.

cogburnd02 5y ago
> (small, tasteful) ads These did used to exist. They were called 120x60 px buttons.
- gojomo 5y ago
  The original AdWords & AdSense pure-text formats weren't so bad, either.

kbsspl 5y ago

Was just thinking about exactly the whole Wiki ecosystem. It's created by individuals like us and should be available via maybe archive.org as a complete dump for individuals to play with. Are they available ?

humanistbot 5y ago
https://dumps.wikimedia.org https://dumps.wikimedia.org
- kbsspl 5y ago
  Thank you. Will explore.

smsm42 5y ago

I must admit to a mixed feeling about it. On one side, a lot of large companies already are using Wikipedia data at large scale, already in a disorganized manner and not giving back anything (well, sometimes giving back donations but not in any relation to their use). Creating a venue for proper large-scale use that has a channel to give back money looks good. On the other hand, a lot of Wikipedia is created by many, many hours of work of many, many volunteers, who aren't (usually) paid a dime for it. Capitalizing on that, and having clients that pay $big_money for the result, creates misaligned incentives, and however noble and pure the motives of everybody are now, incentives always matter and always create pressure. What if some company says "we'll pay millions of dollars, but you're got to remove some of the worst examples of content we don't like - it's not really much, compared to the whole mass of information, nobody would even notice it!" Right now I don't think the community and Wikimedia would agree, but what would happen when they're used to the cash inflow that this program creates and grown to depend on it?