8 ms·
Wikimedia Enterprise – APIs for LLMs, AI Training, and More
- mort96 2y agoI give it at most a year before they start using this LLM to generate content for Wikipedia
- dangoodmanUT 2y agoGood so they can stop shilling for donations on wikipedia
- miffy900 2y agoI'm reading the API docs https://enterprise.wikimedia.com/docs/ https://enterprise.wikimedia.com/docs/ And they don't have an OpenAPI spec available to download? So they seriously expect developers to just manually write their own client code by reading and translating those example CURL commands by hand??! Seriously it's 2024! Not having a spec to download and insepct for any API is a sign of incompetence. When tools like Postman or https://github.com/OpenAPITools/openapi-generator-cli https://github.com/OpenAPITools/openapi-generator-cli exist and save hours of time, you can't seriously expect devs to write all this connecting code by hand anymore.
- michaelt 2y agoIf only there was some code generation technology that could take in human-readable specifications, not just machine-readable ones....
- croes 2y agoThen create one, share it and save us some time.
- trustno2 2y agoso enterprise very wow
- chuckreynolds 2y agolink at bottom of https://enterprise.wikimedia.com/docs/ https://enterprise.wikimedia.com/docs/ page
- schoenobates 2y agoBottom of the page ... https://api.enterprise.wikimedia.com/spec/spec.yaml https://api.enterprise.wikimedia.com/spec/spec.yaml
- krainboltgreene 2y ago> And they don't have an OpenAPI spec available to download? So they seriously expect developers to just manually write their own client code by reading and translating those example CURL commands by hand??! This is baby work to literally every programmer I know.
- causal 2y agoNot to mention that if it's really needed, the LLMs this is made for also excel at converting CURL commands to whatever spec pretty trivially. I've used them this way on a few occasions.
- miffy900 2y agoIt's also a solved problem that literally no human should have to bother with anymore. Like it was solved with SOAP APIs back in the 2000's. Not to mention the fact that documentation is still always hand-written - which can have mistakes or typos in it, then downstream developers will have to translate that into hand-written client code, which can also have those same mistakes or bugs in it. And when the mistake is corrected, the dev needs to go back to where the correction was made in the docs, re-read and interpret the correction and then correct their client code. With API specs and code generators that entire process is automated now. Now imagine bringing together approximately 6 different APIs into a single app, 5 have API specs while 1 doesn't. Now 80% of your time is spent on that 1 API where you have to read the docs to keep your client code up to date. Oh that 1 API vendor doesn't bother letting you know when their API is updated! So you have to F5 their website every week! This is literally my life right now. Anyway, in this case, it was my mistake not to read until the very bottom where they had an OpenAPI yaml spec link there.
- krainboltgreene 2y ago> It's also a solved problem that literally no human should have to bother with anymore. This is such a hilariously wrong statement I can't even begin to approach the rest of this comment with any serious attention.
- rattray 2y agoIf someone from the wikimedia enterprise team is here and interested in providing those SDKs, but finds the openapi generator linked above to be lacking, I'd be happy to help (my company does the SDKs for OpenAI, Cloudflare, etc). I've been a monthly donor to wikimedia for a long time and it'd be quite a privilege to support them with nice API wrappers too. Email in profile.
- causal 2y agoWhat's up with HN's general tone of hostility toward Wikimedia foundation? It's weird how the more free a service is the more entitled our attitudes seem to become.
- ThrowawayTestr 2y agoI think the hostility comes from Wikipedia constantly asking for money when most of the money goes to things that aren't Wikipedia.
- paulette449 2y agoHiding behind a throwaway account allows you to make false statements, unless you can substantiate the "most of the money" comment. Wikimedia Foundation breaks down its expenditure here: https://wikimediafoundation.org/support/where-your-money-goes/ https://wikimediafoundation.org/support/where-your-money-goe... I'm sure what poster is referring to was "discussed" previously in the link below. I use parentheses because it descended into flames pretty quickly: https://news.ycombinator.com/item?id=33170710 https://news.ycombinator.com/item?id=33170710
- ThrowawayTestr 2y agoMy account is two years old and your profile is empty so we're equally as anonymous. That aside, the first heading in your link is "43% Direct support to websites" so yes, most of the money does not go to Wikipedia.
- Vuizur 2y agoWikimedia is unfortunately becoming one of the worst places to give your money to. They have their closed-source infrastructure, which now for years has not been able to generate HTML dumps without a significant percentage of articles missing. They have know of the bug for ages, but still... WMF don't care about their products at all, they give much more funding to vanity workshops in Africa while utterly ignoring the requests of the Wiktionary community, for example.
- mushufasa 2y agoYet this also exists: https://en.wikipedia.org/wiki/Wikipedia:Database_download https://en.wikipedia.org/wiki/Wikipedia:Database_download
- dangoodmanUT 2y agosee you in a year when that finishes
- StableAlkyne 2y agoIt does exist, it is just incredibly painful to use -- it's a raw SQL dump of each individual page, which is missing some of the dynamically generated parts of the page you might care about (like categories, unless you separately download that table too). You also have to then run it through wiki software locally if you want the rendered page instead of markup. Don't get me wrong, it is wonderful that the Wikipedia team offers this, and I am grateful they give anything at all for offline usage. It just feels like it's intended more as a side product of their backup process, rather than something you're really supposed to use.
- riku_iki 2y agofor now
- itishappy 2y agoWikimedia is Wikipedia's parent entity, but it's also the parent of: MediaWiki Wikibooks Wikidata Wikifunctions Wikimedia Commons Wikinews Wikiquote Wikisource Wikispecies Wikiversity Wikivoyage Wiktionary https://en.wikipedia.org/wiki/Wikimedia_Foundation https://en.wikipedia.org/wiki/Wikimedia_Foundation
- violet13 2y agoIt sounds like an impressive list, but many of these are ghost towns and are of little value to machine learning. Wikiversity is a mess... Wikipedia is the crown jewel and probably the only thing of unique commercial value for ML.
- nickpsecurity 2y agoI was hoping more groups did stuff like this. The free sites doing it could handle some copyright issues if their EULA had a built-in license for distribution. In my previous analysis, (IIRC) I found that Wikipedia articles were under a copyleft license with attribution requirements. Does how Wikipedia Enterprise delivers this bypass that where neither use nor derivatives have those requirements? Or are they ignoring the licensing requirements which still legally apply to enterprise customers? Or are they not even claiming to address it?
- kemayo 2y agoMy understanding is that Enterprise is mostly about smoother delivery of content. E.g. Google has those informational summaries in search results, and it's easier for it to keep those current if there's a stream of updates it can subscribe to, rather than having to constantly download the full Wikipedia database dumps and parse them out. It also puts it all into conveniently formatted responses, and tries to do some signaling of content reliability. The API responses do include information about the content license, which differs a bit between different wikimedia properties: https://helpcenter.enterprise.wikimedia.com/hc/en-us/articles/6175800381851-Does-Wikimedia-Enterprise-include-the-license-of-the-pages https://helpcenter.enterprise.wikimedia.com/hc/en-us/article... (I work for the WMF. I don't work on the Enterprise stuff, or have any insider knowledge of it. E.g. I used Google as an illustrative example, but I have no idea whether they're actually using this service. :D)
- bawolff 2y ago> to keep those current if there's a stream of updates it can subscribe to, rather than having to constantly download the full Wikipedia database dumps and parse them out. I mean presumably prior to this they were using the [free] parsoid rest api along with the [free] event stream api. I highly doubt they were parsing the dumps. Its not even clear to me what the core value proposition of the new api is over the old api from the perspective of google. Maybe SLAs? A way to justify donating money without it looking like a donation?
- 2y ago
- deegles 2y agoLooking forward to seeing the details on how they will handle revenue sharing with all of the people who contribute to them.
- riku_iki 2y agoare they obligated?..
- cdme 2y agoNo, but nobody is obligated to contribute to or support them either.
- riku_iki 2y agothat's not really true. Because of network capture, those who wants to contribute into some public free human knowledge, obligated to do it through wikimedia and contribute to wikimedia revenue streams.
- cdme 2y agoOr this simply discourages contributions for folks not wanting to volunteer effort to the gaping maw of yet another monolithic content vacuum/LLM.
- paxys 2y agoIf you want monetary compensation you should not be contributing to Wikipedia. Otherwise apply to work there as an employee.
- tux3 2y agoAfter seeing this comment, onlookers report that the Wikimedia Foundation appears to have just vanished in a puff of volunteerism
- 2y ago
- s1k3s 2y ago> Real-time access to Knowledge Wikipedia knowledge is basically "democracy" knowledge, i.e. the more people decided to support an idea, the "truer" it gets. That's not knowledge at all!
- verdverm 2y agoThat politicians get to scrub their pages shows there are cracks in places, but overall it's generally pretty ok
- Tao3300 2y ago> overall it's generally pretty ok That's the kind of glowing praise I'd get from my 8th grade Geometry teacher when I got a C on a test.
- verdverm 2y agolol, I find wikipedia to be a bit all over the place or lack structure. It also is largely a giant block of text, though I do think the new LHS menu has improved things
- tux3 2y agoThat's exactly what almost all knowledge is. When was it that you last verified something by yourself, with an experiment? You didn't test the things you know. You know things because you could see they were the consensus, and so you had no reason to challenge them. If an idea is disputed, then you trust it less. If it comes from a small number of reputable sources, then you trust it more than a large numbers of unreliable people. So with the Wiki. Human knowledge isn't from the platonic realm. Human knowledge isn't checked by a theorem prover. You get almost all of your knowledge from other people, and you have no choice but to trust them for almost all of it.
- nickpsecurity 2y ago“That's exactly what almost all knowledge is.” It’s worse than that. Much knowledge comes from authorities or peers. The sources could be unpopular or barely reviewed. Yet, people are likely to believe specific types of sources. It seems to be hardwired for some purpose.
- bawolff 2y agoWhile i am sympathetic to wmf finding alternative funding streams, I do get nervous about these sort of things due to the inherent conflict of interest and incentives to canabalize the free offerings. I'm not saying that is happening now, but will it happen eventually? Additionally, originally it was promised this would all be open source, and officially they are sticking with that, but it seems like they are going with the model of throwing code over the fence like once a year, which does not really meet my expectations.
- admissionsguy 2y agoWMF goes the way of bureaucracies. It finds multiple ways to enlarge itself over the years and just cannot ever get any smaller. I suppose at some point it will overextend itself and harm Wikipedia, the only[1] thing of value, in the process. [1] Yeah there are Wikidata and some other projects. But it's like saying Google is not an ad company.
- toomuchtodo 2y agoI’d rather Wikipedia offer it versus a for profit enterprise using their dataset. Cut out the middleman and vertically integrate.
- kemayo 2y agoThere's a whole statement-of-principles thing that at least implies that the intention isn't to cannibalize the existing offerings: https://meta.wikimedia.org/wiki/Wikimedia_Enterprise/Principles https://meta.wikimedia.org/wiki/Wikimedia_Enterprise/Princip... Though I imagine that only works so far as you feel you can trust the Foundation to stick to those principles, so that's complicated. :D There's also a bunch of FAQs here that sort of get at how the funding streams are supposed to integrate into the existing structures and how it's supposed to avoid pushing out the free services: https://meta.wikimedia.org/wiki/Wikimedia_Enterprise/FAQ https://meta.wikimedia.org/wiki/Wikimedia_Enterprise/FAQ (As I said elsewhere, I work for the WMF but I don't work on anything related to this so I'm just commenting as someone who has more experience searching through our public info than most HN commenters would...)
- 2y ago
- thih9 2y agoIs it’s ok to train AIs on CC-licensed data? What about attribution, should the AI be distributed with a note about the dataset’s origin? I could imagine a poorly trained model that returns content nearly identical to original; who would enforce CC rules then?
- yreg 2y agoIt shouldn't be much of a problem to ship the LLM along with attributions. (List of all sources used in the dataset - not a problem, unless they are secret, shady or illegal.) Wikipedia is one of the easy ones, since you need to attribute just 'Wikipedia' for the entire corpus instead of many individual users. The bigger issue is when a user uses the model to generate some text. Should they attribute it when using it somewhere? That doesn't seem very practical, since it seems that soon most of the text will be edited by LLMs and those seem to be trained on most of the web -> so pretty much everything would need to be attributed to everything. Unless someone puts a stop to this, which I find improbable.
- sp332 2y agoIncluding a list of every Wikipedia author is possible but very inconvenient. https://en.m.wikipedia.org/wiki/Wikipedia:Reusing_Wikipedia_content https://en.m.wikipedia.org/wiki/Wikipedia:Reusing_Wikipedia_...
- killingtime74 2y agoI wonder if it can be like those California proposition notices, just include it everywhere just in case
- pwarner 2y agohttps://creativecommons.org/2023/08/18/understanding-cc-licenses-and-generative-ai/ https://creativecommons.org/2023/08/18/understanding-cc-lice...
- deleted 2y ago[deleted]
- kristianpaul 2y agoInteresting, the current DB dump is not for everyone, but if they also offer an LLM trained on Wikipedia data that answers and provide actual valida citation , please do. (not sure if duckduckgo stop offering that)
- flipbrad 2y agoNot quite what you're looking for, but something to play around and give feedback on in this area is https://meta.m.wikimedia.org/wiki/Future_Audiences/Experiment:Citation_Needed https://meta.m.wikimedia.org/wiki/Future_Audiences/Experimen...
- yreg 2y agoAFAIK a good way to provide better answers and avoid hallucinations would be to compute embeddings for all sections of text in Wikipedia and then when a user asks a question create an embedding from that question. Use it to find the X closest embeddings to the question being posed, lookup their original articles, feed them all into context of an LLM and then ask it to answer the question based on that context (alone). Contexts are becomming quite large so it's possible to put a lot of stuff in there. LLMs answering questions based on a giben text seem to be more reliable than those that are simply trained/fine tuned on some library of texts. p
- jahewson 2y agoUnfortunately this still results in plenty of hallucinations.
- falcor84 2y agoWhat do you mean? Have you or someone else already followed this exact approach?
- yreg 2y agoSupabase did: https://supabase.com/blog/chatgpt-supabase-docs https://supabase.com/blog/chatgpt-supabase-docs I've attempted it as well a year ago (mostly for fun) for our project. Yes, it can still hallucinate. But I would say it's much much much better in this regard than fine-tuning. When I did it, the main issue was that our documentation wasn't exhaustive enough. There are plenty of things that are clear to our users (other teams in the company), but not at all clear to the LLM from the few text excerpts it receives. Also, our context was quite limited back then to just a few paragraphs of text.
- low_tech_punk 2y agoNitpick, the purple blue gradient is overused, especially as the background of source code examples. It hinders readability and feels off-brand for a Wikimedia project. This might be foreshadowing a descent into chasing trends. I just hope Wikimedia does not follow Mozilla's track in spending budget on fluff work.
- deleted 2y ago[deleted]
- adultSwim 2y agoI support this and think it should be a model for other organizations. They already publish the data for free. It's already being used to train models. Companies are already using those models to profit. I'm not sure I understand the concern. I hope to see their APIs expand in future work, successfully tying together machine learning with their prior work on publishing a knowledge graph enriched with high quality structured metadata.