3 ms·

> its just matching which title in language X matches which title in language Y Like I said, gross overengineering. "You can save cents and seconds of time up

by j-pb 5y ago

> its just matching which title in language X matches which title in language Y Like I said, gross overengineering. "You can save cents and seconds of time updating knowledge boxes, by spending millions and hours developing federated SPARQL query capabilities!"

bawolff 5y ago
More like, we put the facts and knowledge in a central (mysql) db, so that instead of updating them in 300 places (Wikipedia has a lot of languages), you just update them in one place. And since the data happened to be there anyways, we added an RDF export feature, and threw that into a BlazeGraph DB so that people could do SPARQL queries if they want. The (not really federated) SPARQL queries have basically nothing to do with the central-place-to-store-facts that are used in multiple places, feature, but it was pretty cheap to do once everything was in a central place.
- lifeisstillgood 5y ago
  Is there a entry point for "how to make FOSS contributions to wiki-projects". I at the very least would be interested in finding out how all the pieces fit together, and maybe browsing the big list to see if anything was in my wheelhouse. Edit: And thank you for some insight into the back end of wikipedia - I did not know multiple languages was such a big push and it is such a sensible direction . Keep up the good work and keep pestering me for cash each year.
  - bawolff 5y ago
    https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hac... is probably what you are looking for. There's a very old architecture doc from 2012: https://www.mediawiki.org/wiki/Manual:MediaWiki_architecture https://www.mediawiki.org/wiki/Manual:MediaWiki_architecture its kind of outdated and missing a lot of the newer pieces. p.s. I used to work for the Wikimedia foundation about 1.5 years ago, but don't anymore. Prior to working there I was a volunteer for the FOSS project, and in theory I still am but I honestly don't do much anymore. My opinions are my own and don't represent WMF or anyone else.
- j-pb 5y ago
  As if thowing the messy complexities of MySQL queries into the serene simplicity of static hypertext is somehow better. How often is numerical (which is the only knowledge you can really automate this way) knowledge updated? How often is said knowledge dependent on cultural idiosyncracies? How much effort is the code maintenance of the specialised editing system? How much effort is maintainance of the centralised infrastructure. How much effort is it really to change the data in multiple places, given that enough eyes every bug is shallow? What are the oppirtunity costs of not being able to properly distribute and share articles because they are suddenly tied to a very specific codebase? How much infrastructure cost do I have because I can't get volunteers to host my data? Overengineering solving the wrong problems at it's finest, no wonder the software has stayed this bad for decades now.
  - bawolff 5y ago
    > How often is numerical (which is the only knowledge you can really automate this way) knowledge updated? Numerical knowledge is not the only knowledge that is stored there. I'm going to go with update rate of often. It has a higher edit rate than english wikipedia does. > How often is said knowledge dependent on cultural idiosyncracies? Sometimes that is true, other times it isn't. The system is used where it makes sense, and not used where it doesn't. > How much effort is the code maintenance of the specialised editing system? How much effort is maintainance of the centralised infrastructure. There is certainly some. I wouldn't say its dominating, but it certainly exists. There is plenty of centralized infrastructure for other things too. Prior to the introduction of wikidata, more complex fragile systems existed to try and keep things in sync in a more manual fashion. They also had a cost. > How much effort is it really to change the data in multiple places, given that enough eyes every bug is shallow? A lot. And we don't have that many eyes in lots of languages. There's a lot after all: https://en.wikipedia.org/wiki/Special:SiteMatrix https://en.wikipedia.org/wiki/Special:SiteMatrix > What are the oppirtunity costs of not being able to properly distribute and share articles because they are suddenly tied to a very specific codebase Not sure what this has to do with it. Articles are written in a custom markup language. That's already tying it to a specific code base quite heavily. The wikidata dependency is pretty trivial, and there are open apis to get info out of wikidata, and regularly made dumps of all the data in wikidata if you want to copy it. If that's not good enough, just copy the rendered version instead of the source. > How much infrastructure cost do I have because I can't get volunteers to host my data I'm not sure what you're talking about. Wikipedia was never hosted in some magical peer-to-peer fashion on volunteer web servers, nor is there any situation where it is likely to be hosted that way. The goals of (near) instant update time, as well as some central quality control and user blocking, makes distributed hosting pretty impossible.
auggierose 5y ago
Sounds like you have no clue what engineering is.
- jbuhbjlnjbn 5y ago
  I wonder why you have nothing substantial to argue and feel the need to resort to ad hominem
nickik 5y ago
As somebody who has used these facilities I highly disagree. They are really awesome. And if you have some magically technical solution that is much better and much easier, then please share it. And don't just complain that everybody else is doing everything wrong.

by j-pb 5y ago

bawolff 5y ago

More like, we put the facts and knowledge in a central (mysql) db, so that instead of updating them in 300 places (Wikipedia has a lot of languages), you just update them in one place. And since the data happened to be there anyways, we added an RDF export feature, and threw that into a BlazeGraph DB so that people could do SPARQL queries if they want. The (not really federated) SPARQL queries have basically nothing to do with the central-place-to-store-facts that are used in multiple places, feature, but it was pretty cheap to do once everything was in a central place.

lifeisstillgood 5y ago
Is there a entry point for "how to make FOSS contributions to wiki-projects". I at the very least would be interested in finding out how all the pieces fit together, and maybe browsing the big list to see if anything was in my wheelhouse. Edit: And thank you for some insight into the back end of wikipedia - I did not know multiple languages was such a big push and it is such a sensible direction . Keep up the good work and keep pestering me for cash each year.
- bawolff 5y ago
  https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hac... is probably what you are looking for. There's a very old architecture doc from 2012: https://www.mediawiki.org/wiki/Manual:MediaWiki_architecture https://www.mediawiki.org/wiki/Manual:MediaWiki_architecture its kind of outdated and missing a lot of the newer pieces. p.s. I used to work for the Wikimedia foundation about 1.5 years ago, but don't anymore. Prior to working there I was a volunteer for the FOSS project, and in theory I still am but I honestly don't do much anymore. My opinions are my own and don't represent WMF or anyone else.
j-pb 5y ago
As if thowing the messy complexities of MySQL queries into the serene simplicity of static hypertext is somehow better. How often is numerical (which is the only knowledge you can really automate this way) knowledge updated? How often is said knowledge dependent on cultural idiosyncracies? How much effort is the code maintenance of the specialised editing system? How much effort is maintainance of the centralised infrastructure. How much effort is it really to change the data in multiple places, given that enough eyes every bug is shallow? What are the oppirtunity costs of not being able to properly distribute and share articles because they are suddenly tied to a very specific codebase? How much infrastructure cost do I have because I can't get volunteers to host my data? Overengineering solving the wrong problems at it's finest, no wonder the software has stayed this bad for decades now.
- bawolff 5y ago
  > How often is numerical (which is the only knowledge you can really automate this way) knowledge updated? Numerical knowledge is not the only knowledge that is stored there. I'm going to go with update rate of often. It has a higher edit rate than english wikipedia does. > How often is said knowledge dependent on cultural idiosyncracies? Sometimes that is true, other times it isn't. The system is used where it makes sense, and not used where it doesn't. > How much effort is the code maintenance of the specialised editing system? How much effort is maintainance of the centralised infrastructure. There is certainly some. I wouldn't say its dominating, but it certainly exists. There is plenty of centralized infrastructure for other things too. Prior to the introduction of wikidata, more complex fragile systems existed to try and keep things in sync in a more manual fashion. They also had a cost. > How much effort is it really to change the data in multiple places, given that enough eyes every bug is shallow? A lot. And we don't have that many eyes in lots of languages. There's a lot after all: https://en.wikipedia.org/wiki/Special:SiteMatrix https://en.wikipedia.org/wiki/Special:SiteMatrix > What are the oppirtunity costs of not being able to properly distribute and share articles because they are suddenly tied to a very specific codebase Not sure what this has to do with it. Articles are written in a custom markup language. That's already tying it to a specific code base quite heavily. The wikidata dependency is pretty trivial, and there are open apis to get info out of wikidata, and regularly made dumps of all the data in wikidata if you want to copy it. If that's not good enough, just copy the rendered version instead of the source. > How much infrastructure cost do I have because I can't get volunteers to host my data I'm not sure what you're talking about. Wikipedia was never hosted in some magical peer-to-peer fashion on volunteer web servers, nor is there any situation where it is likely to be hosted that way. The goals of (near) instant update time, as well as some central quality control and user blocking, makes distributed hosting pretty impossible.

auggierose 5y ago

Sounds like you have no clue what engineering is.

jbuhbjlnjbn 5y ago
I wonder why you have nothing substantial to argue and feel the need to resort to ad hominem

nickik 5y ago

As somebody who has used these facilities I highly disagree. They are really awesome. And if you have some magically technical solution that is much better and much easier, then please share it. And don't just complain that everybody else is doing everything wrong.