5 ms·

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable for

by paxys 2y ago

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

ks2048 2y ago
I don't have a problem with having paid special services, but the "machine parsable formats" is a bit troubling since I think that should be a core part of the open wikipedia project. I submit this link after coming across this site while Googling for info on parsing wikipedia "infoboxes". I plan to check out their "Article Structured Contents (BETA)" API. Improving infoboxes to be machine-readable seems important. And it would be bad if didn't do this because it's a revenue stream for them.
- codetrotter 2y ago
  There’s also Wikidata which has machine readable data for everyone https://m.wikidata.org/wiki/Wikidata:Main_Page https://m.wikidata.org/wiki/Wikidata:Main_Page > Wikidata is a free and open knowledge base that can be read and edited by both humans and machines. > Wikidata acts as central storage for the structured data of its Wikimedia sister projects including Wikipedia, Wikivoyage, Wiktionary, Wikisource, and others.
  - riku_iki 2y ago
    Yes, but it is different dataset, many(majority?) infoboxes are not in wikidata
- bawolff 2y ago
  Just fyi, i think the plan for machine readable infoboxes is stick the data in wikidata, which has its own dumps in machine readable format as well as a publicly accessible SPARQL query endpoint. Its actually quite cool. If you have never played with it i encourage checking out some of the example queries (there is a button labelled examples) on https://query.wikidata.org/ https://query.wikidata.org/
- zozbot234 2y ago
  DBPedia has been providing machine-readable info drawn from Wikipedia infoboxes for a long time, and that's absolutely an open project. But there are drawbacks to that approach, nowadays Wikipedia users are broadly expected to input that info into Wikidata (a language-independent project, providing its data under the CC0 license) and the Wikipedias are gradually moving towards fetching the information from there as well. The English Wikipedia is somewhat behind on that, so DBPedia might still be useful for parsing info from it.
- kemayo 2y ago
  Machine-readable infoboxes are surprisingly painful because an "infobox" isn't a top-level thing in mediawiki. Rather, it's something that emerged from community practice -- a template that's slightly-standardized, and has a sprawling network of subtemplates for specific categories of article. Then the assorted other language wikipedias have their own variants of these templates with their own names, etc. Pulling it into a mediawiki extension (or core) and making it part of the page-level metadata gets suggested pretty frequently, but it's a bit contentious amongst the hardcore editors who'd need to actually adopt such things. The template-based nature of the current infoboxes mean that they're very accessible to the community, and it's easy to spin off new variants or make changes without getting programmers to help you. There's slow movement towards getting the sort of data that winds up in infoboxes into wikidata, but it's still somewhat spotty. (If you've never done it, it can be quite edifying to install mediawiki for yourself and seeing how much of the surrounding infrastructure of wikipedia is absent because it's all templates.)
  - PokestarFan 2y ago
    I once tried to export a simple infobox template with dependencies and it exported 30-40 different template/module pages.
elric 2y ago
CC-BY-SA to be specific. How does attribution of derivative works work in LLMs (etc)? When is the content they produce required to be CC-BY-SA as well (by virtue of the -SA part of the license)?
- schlauerfox 2y ago
  LLM output can't be copywritten as it's not the work of a human. The company would presumably have to include attribution to every piece of data the system is trained on for every query? Seems absurd, but then how would attribution work? Not really considered when the license was written.
  - elric 2y ago
    Licenses go beyond copyright though, they tell you what you can and can't do with something someone else made. They're contracts. I'm sure we'll be seeing more lawsuits (and likely new regulation and licenses) around this. Are you saying LLMs are not derivatives of their source material? Why not?
    - grape_surgeon 2y ago
      I agree on lawsuits being likely and needed to establish precedent. CC-BY-SA specifies "other rights such as publicity, privacy, or moral rights may limit how you use the material". At some point, maybe poorly-trained chatbots that consistently produce what's seen as avoidably/negligantly poor results may become regulated. Like how if a company poorly trains its employees, it is on the hook for their employees' behavior.
btown 2y ago
It also seems Wikimedia isn't trying to relicense the content in any way that strips its e.g. CC-SA status, but rather providing the licenses as context alongside each API call. https://helpcenter.enterprise.wikimedia.com/hc/en-us/articles/6175800381851-Does-Wikimedia-Enterprise-include-the-license-of-the-pages https://helpcenter.enterprise.wikimedia.com/hc/en-us/article... It's worth noting that https://creativecommons.org/faq/#artificial-intelligence-and-cc-licenses https://creativecommons.org/faq/#artificial-intelligence-and... itself takes the general stance that "as a general matter text and data mining in the United States is considered a fair use and does not require permission under copyright." But as a practical matter, I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training by the very platform that solicited their volunteer services, regardless of their works' legal status and Wikimedia's technical respect of that legal status.
- emw 2y ago
  > I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training As someone who avidly edited Wikipedia for 6-8 years, I am happy to see my volunteer work used for LLM training. I also agree some other editors likely aren't.
  - ZunarJ5 2y ago
    As another editor, I think they might be a vocal minority. :)
  - lambdaone 2y ago
    Given that all Wikipedia editors have explicitly consented to their content being released under the Creative Commons Attribution-ShareAlike 4.0 License, they don't get a choice about their content being used for any purpose. Redistribution of content is an entirely different matter, and the legal status of copyrighted material in relation to LLM training is an open issue that is currently the subject of litigation.
    - 2y ago
nextaccountic 2y ago
> machine parsable formats, Isn't the point of wikidata to offer machine-parseable formats? Also: how can those formats be locked into a paywall?
- riku_iki 2y ago
  wikidata is not the same as wikipedia infoboxes. wikipedia infoboxes API is already paywalled, pricing starts from $0.01/req.
  - nextaccountic 2y ago
    What I am proposing is that Wikipedia infoboxes should be included in Wikidata
    - riku_iki 2y ago
      this is good proposal, but it is far from what we have and not clear if it will happen.
      - nextaccountic 2y ago
        But now unfortunately the Wikimedia foundation has a motive to block such a proposal, in order to not threaten their revenue stream And that's the problem with making money off convenience. If the free thing becomes better, then it cannibalizes the paid stuff. So if you take this approach to funding, you will want the free tier to be perpetually bad as much as possible.
        riku_iki 2y ago
        I think sorting current infoboxes mess is not trivial problem. We will see what will be quality of their solution, it could be lower than actual wikidata.

by paxys 2y ago

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

ks2048 2y ago

I don't have a problem with having paid special services, but the "machine parsable formats" is a bit troubling since I think that should be a core part of the open wikipedia project. I submit this link after coming across this site while Googling for info on parsing wikipedia "infoboxes". I plan to check out their "Article Structured Contents (BETA)" API. Improving infoboxes to be machine-readable seems important. And it would be bad if didn't do this because it's a revenue stream for them.

codetrotter 2y ago
There’s also Wikidata which has machine readable data for everyone https://m.wikidata.org/wiki/Wikidata:Main_Page https://m.wikidata.org/wiki/Wikidata:Main_Page > Wikidata is a free and open knowledge base that can be read and edited by both humans and machines. > Wikidata acts as central storage for the structured data of its Wikimedia sister projects including Wikipedia, Wikivoyage, Wiktionary, Wikisource, and others.
- riku_iki 2y ago
  Yes, but it is different dataset, many(majority?) infoboxes are not in wikidata
bawolff 2y ago
Just fyi, i think the plan for machine readable infoboxes is stick the data in wikidata, which has its own dumps in machine readable format as well as a publicly accessible SPARQL query endpoint. Its actually quite cool. If you have never played with it i encourage checking out some of the example queries (there is a button labelled examples) on https://query.wikidata.org/ https://query.wikidata.org/
zozbot234 2y ago
DBPedia has been providing machine-readable info drawn from Wikipedia infoboxes for a long time, and that's absolutely an open project. But there are drawbacks to that approach, nowadays Wikipedia users are broadly expected to input that info into Wikidata (a language-independent project, providing its data under the CC0 license) and the Wikipedias are gradually moving towards fetching the information from there as well. The English Wikipedia is somewhat behind on that, so DBPedia might still be useful for parsing info from it.
kemayo 2y ago
Machine-readable infoboxes are surprisingly painful because an "infobox" isn't a top-level thing in mediawiki. Rather, it's something that emerged from community practice -- a template that's slightly-standardized, and has a sprawling network of subtemplates for specific categories of article. Then the assorted other language wikipedias have their own variants of these templates with their own names, etc. Pulling it into a mediawiki extension (or core) and making it part of the page-level metadata gets suggested pretty frequently, but it's a bit contentious amongst the hardcore editors who'd need to actually adopt such things. The template-based nature of the current infoboxes mean that they're very accessible to the community, and it's easy to spin off new variants or make changes without getting programmers to help you. There's slow movement towards getting the sort of data that winds up in infoboxes into wikidata, but it's still somewhat spotty. (If you've never done it, it can be quite edifying to install mediawiki for yourself and seeing how much of the surrounding infrastructure of wikipedia is absent because it's all templates.)
- PokestarFan 2y ago
  I once tried to export a simple infobox template with dependencies and it exported 30-40 different template/module pages.

elric 2y ago

CC-BY-SA to be specific. How does attribution of derivative works work in LLMs (etc)? When is the content they produce required to be CC-BY-SA as well (by virtue of the -SA part of the license)?

schlauerfox 2y ago
LLM output can't be copywritten as it's not the work of a human. The company would presumably have to include attribution to every piece of data the system is trained on for every query? Seems absurd, but then how would attribution work? Not really considered when the license was written.
- elric 2y ago
  Licenses go beyond copyright though, they tell you what you can and can't do with something someone else made. They're contracts. I'm sure we'll be seeing more lawsuits (and likely new regulation and licenses) around this. Are you saying LLMs are not derivatives of their source material? Why not?
  - grape_surgeon 2y ago
    I agree on lawsuits being likely and needed to establish precedent. CC-BY-SA specifies "other rights such as publicity, privacy, or moral rights may limit how you use the material". At some point, maybe poorly-trained chatbots that consistently produce what's seen as avoidably/negligantly poor results may become regulated. Like how if a company poorly trains its employees, it is on the hook for their employees' behavior.

btown 2y ago

It also seems Wikimedia isn't trying to relicense the content in any way that strips its e.g. CC-SA status, but rather providing the licenses as context alongside each API call. https://helpcenter.enterprise.wikimedia.com/hc/en-us/articles/6175800381851-Does-Wikimedia-Enterprise-include-the-license-of-the-pages https://helpcenter.enterprise.wikimedia.com/hc/en-us/article... It's worth noting that https://creativecommons.org/faq/#artificial-intelligence-and-cc-licenses https://creativecommons.org/faq/#artificial-intelligence-and... itself takes the general stance that "as a general matter text and data mining in the United States is considered a fair use and does not require permission under copyright." But as a practical matter, I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training by the very platform that solicited their volunteer services, regardless of their works' legal status and Wikimedia's technical respect of that legal status.

emw 2y ago
> I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training As someone who avidly edited Wikipedia for 6-8 years, I am happy to see my volunteer work used for LLM training. I also agree some other editors likely aren't.
- ZunarJ5 2y ago
  As another editor, I think they might be a vocal minority. :)
- lambdaone 2y ago
  Given that all Wikipedia editors have explicitly consented to their content being released under the Creative Commons Attribution-ShareAlike 4.0 License, they don't get a choice about their content being used for any purpose. Redistribution of content is an entirely different matter, and the legal status of copyrighted material in relation to LLM training is an open issue that is currently the subject of litigation.
  - 2y ago

nextaccountic 2y ago

> machine parsable formats, Isn't the point of wikidata to offer machine-parseable formats? Also: how can those formats be locked into a paywall?

riku_iki 2y ago
wikidata is not the same as wikipedia infoboxes. wikipedia infoboxes API is already paywalled, pricing starts from $0.01/req.
- nextaccountic 2y ago
  What I am proposing is that Wikipedia infoboxes should be included in Wikidata
  - riku_iki 2y ago
    this is good proposal, but it is far from what we have and not clear if it will happen.
    - nextaccountic 2y ago
      But now unfortunately the Wikimedia foundation has a motive to block such a proposal, in order to not threaten their revenue stream And that's the problem with making money off convenience. If the free thing becomes better, then it cannibalizes the paid stuff. So if you take this approach to funding, you will want the free tier to be perpetually bad as much as possible.
      - riku_iki 2y ago
        I think sorting current infoboxes mess is not trivial problem. We will see what will be quality of their solution, it could be lower than actual wikidata.