13 ms·

Wikipedia starts work on $2.5M internet search engine project to rival Google [pdf]

• udkl 11y ago
  The grant amount is $250,000
  • olh 11y ago
    Correct. As per page 9 the most of the budget allocated for the project comes from wikimedia themselves, totaling 2,445,873.00 USD in for the fiscal year 2015-2016.
    • mh- 11y ago
      do these funds come from the donations they solicit on wikipedia.org?
      • TazeTSchnitzel 11y ago
        Yes, it's the main (only?) source of income for the Wikimedia Foundation.
        • maxerickson 11y ago
          There are substantial donations from other foundations and company match programs (Huuuge page): https://annual.wikimedia.org/2014/#s-5 https://annual.wikimedia.org/2014/#s-5 I guess the foundations aren't reasonably a response to the Jimmy banners, the company matches probably are.
• cmarschner 11y ago
  This is a very, very, very small amount of money if you want to build a search engine, let alone one "to rival Google" (source?). Looks like the goals are realistic, though - look how wikipedia search could be extended beyond results from wikipedia.org, build some test sets. And get a better idea what it really is that is supposed to be built.
  • Lanari 11y ago
    Google started with less, and so almost every project on the internet.
  • tuvalie 11y ago
    I run a knowledge engine project that actively mines facts from web-based sources and third-party data dumps. It was featured on the front page of HN a while ago, and has a total of $10 in funding (from a single donation; not a typo). I have, however, put a ton of time into it, and it's something I'm very passionate about. I'm fairly confident Wikipedia can have success making initial headway on their grant objectives with $250,000. If anyone's interested, here's a demo of my own project: https://tuvalie.com/fae/?q=Albert%20Einstein https://tuvalie.com/fae/?q=Albert%20Einstein
    • metasean 11y ago
      Over 15 years ago, for my undergraduate thesis I set up a "Hypermedia Textbook" on the history of my field. I had to manually collate all the info, manually scan in every photo, and type in every last bit of text and html. The end result was a couple hundred pages that looked very, very similar to what your Einstein page looks like! At the time, I knew a better way would emerge, but didn't know how or when. It's moments like this that I (a) feel old :( and (b) am amazed by the times we live in and the speed at which things are happening! :) Thank you for providing such a wonderful, if unintentional, moment of self-reflection!
      • tuvalie 11y ago
        Thank you for checking it out! And if you have any ideas for how things can be improved, I'd love to hear them :)
    • cJ0th 11y ago
      So apparently Einstein has his own, official website (einstein.biz). Who knew? It has got a proper shop and everything! Treat yourself to this relatively expensive USB stick: http://www.shopeinstein.com/mimobot-16gb-usb-flash-drive/details/117610702?feature-name=toys-gadgets&feature=117611424 http://www.shopeinstein.com/mimobot-16gb-usb-flash-drive/det...
• wanderingstan 11y ago
  Given the terrible state of advertising, I would welcome a search engine that penalizes pages with popovers, animated ads, auto playing audio, and so on. Google would never build this, given its business model. I hope Wikipedia brings some innovation to search, untethered from advertising revenue.
  • aabbccddee 11y ago
    Google does do this AFAIK. They dont get any money when people turn on adblock and have always leaned towards simple ads as well as filtering out sites with crummy ux in search
  • manigandham 11y ago
    Google does actually penalize this. It's also not that easy to catch given the layers involved.
• kristianp 11y ago
  This is a link to the actual pdf: https://upload.wikimedia.org/wikipedia/foundation/a/a7/Knowledge_engine_grant_agreement.pdf https://upload.wikimedia.org/wikipedia/foundation/a/a7/Knowl...
• talles 11y ago
  I don't know how should I feel about all those donation campaigns they usually do after this.
  • Mizza 11y ago
    Looks like this is a grant specifically for this work (at least in part.) I think this is an incredibly good use of their money. Google is the world's biggest surveillance machine, I hope that Wikipedia can do to them what they've already done to Encarta.
• xojoc 11y ago
  This is exciting. Recently I started to work on an answer engine/search engine. It still sucks but it's a good project to work on when bored http://kairos.xyz/ http://kairos.xyz/ In a few weeks I'll publish the source code and do a Show HN. I wish a lot of luck to https://www.mojeek.com/ https://www.mojeek.com/ and http://www.lexxe.com/ http://www.lexxe.com/ too. DuckDuckGo also started to crawl the web with its own bot (right now they're using Yandex's api). We need more competition from different countries. Just think about the censorship done by Baidu or how Google never plays by its own rules. It's also interesting to think about a way to monetize a search engine. For kairos.xyz I was thinking about paid accounts (1 euro per month) providing more features, like the ability to search from the command line. For example you write "kairos Richard Stallman" and it prints basic information about Richard Stallman on your terminal.
  • seven-dev 11y ago
    (Default nginx page showing up on your website)
    • xojoc 11y ago
      It works for me. Care to show me a screenshot?
      • seven-dev 11y ago
        "Welcome to nginx on Debian! If you see this page, the nginx web server is successfully installed and working on Debian. Further configuration is required. For online documentation and support please refer to nginx.org Please use the reportbug tool to report bugs in the nginx package with Debian. However, check existing bug reports before reporting a new bug. Thank you for using debian and nginx." I meant here: http://kairos.xyz/ http://kairos.xyz/
      • seven-dev 11y ago
        When I make a request from another IP, it gives me the right page. " (...) Try... Who is Richard Stallman? define bravado 10 USD to EUR RFC 2460 generate username help - about"
• jayadeeptp 11y ago
  Misleading title. I don't think they want the result of the grant to rival Google
• seven-dev 11y ago
  I support it 100%. I love Google but they have too much power and I'm sure they'll start taking advantage of that soon (like they did with Google+ and Youtube).
  • adventured 11y ago
    Google has been the dominant search engine for a solid 12 years. They have a practical monopoly. They're under constant anti-trust scrutiny - of one form another - in all of their major markets. And you think they're going to start taking advantage (as in particularly egregious behavior) of their position soon? No need. It's generating $23 billion per year in operating income and growing. There are no serious challengers. It's far more likely their search engine will be constrained under piles of government oversight in the coming years. About the time governments start worrying about products like this in tech, is about the time they're just beginning to become less central. The exact same thing happened to both IBM and Microsoft.
    • ntakasaki 11y ago
      They already take advantage of their in several different ways. https://www.linkedin.com/pulse/ftc-says-google-skewed-search-results-news-insurance https://www.linkedin.com/pulse/ftc-says-google-skewed-search... http://www.searchenginejournal.com/yelp-complains-outranked-google-local-listings/111539/ http://www.searchenginejournal.com/yelp-complains-outranked-... http://marketingland.com/once-deemed-evil-google-now-embraces-paid-inclusion-13138 http://marketingland.com/once-deemed-evil-google-now-embrace... http://searchengineland.com/google-results-too-ad-heavy-166226 http://searchengineland.com/google-results-too-ad-heavy-1662... http://wallstcheatsheet.com/stocks/ftc-googles-ad-practice-is-deceptive.html/?a=viewall http://wallstcheatsheet.com/stocks/ftc-googles-ad-practice-i... http://www.benedelman.org/adlabeling/google-colors-oct2014.html http://www.benedelman.org/adlabeling/google-colors-oct2014.h...
  • dghughes 11y ago
    I can't see it being much better I find Wikipedia impartiality lacking it's very US-centric. For example something like the history of the Alaskan panhandle as seen from the US perspective is totally different when seen from a Canadian perspective. I never use Wikipedia as a primary source of info even the linked sources I try to use as least three independent sources. I would certainly like to see "Reliable and trustworthy information" but who do I trust who is reliable?
• deleted 11y ago
  [deleted]
• sparkzilla 11y ago
  Google already is the search engine for Wikipedia. And Wikipedia is the content provider for Google. Why mess up such a beautiful arrangement? http://newslines.org/blog/google-and-wikipedia-best-friends-forever/ http://newslines.org/blog/google-and-wikipedia-best-friends-...
• timClicks 11y ago
  Link to Wikimedia's wiki page on the project includes a decent FAQ: https://meta.wikimedia.org/wiki/Knowledge_Engine https://meta.wikimedia.org/wiki/Knowledge_Engine
• bato 11y ago
  Surprised no one has been mentioning qwant.com yet.
• pavanlimo 11y ago
  Doesn't it say 250k in the letter?
• nitrix 11y ago
  The title is misleading. It's 250k, not 2.5m and the goal is a knowledge engine, not a search engine.
  • lucagold 11y ago
    http://i.imgur.com/w89dQ4i.png http://i.imgur.com/w89dQ4i.png sounds like a search engine to me...
    • orionblastar 11y ago
      It is a search engine called the knowledge engine. That is what I read of it.
  • chris_wot 11y ago
    Not only is it a search engine, but it is a grant application that has had WMF staff leaving in droves, and has greatly upset many, many others - who will quite likely also leave. It's very, very sad. And it's also a shameful moment for the WMF. edit: and don't just think it's me saying it. The WMF has had a mass exodus of staff in the last week or so. If you speak to any WMF non-executive staff members directly, you'll quickly find out that morale is at an all time low, and confidence in the WMF Board is sitting at something like 12%.
    • xojoc 11y ago
      Care to explain? Do you have some links/sources?
      • chris_wot 11y ago
        Sure. I'll start off with the following email from Liam Wyatt: https://lists.wikimedia.org/pipermail/wikimedia-l/2016-February/081835.html https://lists.wikimedia.org/pipermail/wikimedia-l/2016-Febru... The grant application you are looking at was only revealed due to a MASSIVE amount of controversy and pressure within the Wikimedia Foundation. The community representative (James Heilman) on the board was let go the other day, in part because of concerns around this grant. You might want to look at the Wikipedia Signpost article he wrote about this: https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2016-02-03/In_focus https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2... Many people have questioned this. Lila, their Executive Director, seems to have conjured this up out of thin air, without consultation of any WMF staff members, or anyone in any of the various communities. Even highly influential, well respected people like Tim Starling appear to have been broadsided by this. Here is what Lila Tretikov wrote about the search engine: It was my mistake to not initiate this ideation on-wiki. Quite honestly, I really wish I could start this discussion over in a more collaborative way, knowing what I know today. Of course, that’s retrospecting with a firmer understanding of what the ideas are, and what is worthy of actually discussing. In the staff June Metrics meeting in 2015, the ideation was beginning to form in my mind from what I was learning through various conversations with staff. I had begun visualizing open knowledge existing in the shape of a universe. I saw the Wikimedia movement as the most motivated and sincere group of beings, united in their mission to build a rocket to explore Universal Free Knowledge. The words “search” and “discovery” and “knowledge” swam around in my mind with some rocket to navigate it. However, “rocket” didn’t seem to work, but in my mind, the rocket was really just an engine, or a portal, a TARDIS, that transports people on their journey through Universal Free Knowledge. From the perspective I had in June, however, I was unprepared for the impact uttering the words “Knowledge Engine” would have. Can we all just take a moment and mercifully admit: it’s a catchy name. Perhaps not a great one or entirely appropriate in our context (hence we don’t use it any more). I was motivated. I didn’t yet know exactly what we needed to build, or how we would end up building it. I could’ve really used your insight and guidance to help shape the ideas, and model the improvements, and test and verify the impacts. However, I was too afraid of engaging the community early on. Why do you think that was? I have a few thoughts, and would like to share them with you separately, as a wider topic. Either way, this was a mistake I have learned enormously from. (this can be found here: https://meta.wikimedia.org/wiki/User_talk:LilaTretikov_(WMF)#Knowledge_Engine_grant https://meta.wikimedia.org/wiki/User_talk:LilaTretikov_(WMF)...) That's a very, very real problem. An executive director of the Wikimedia Foundation should never have felt too afraid of engaging with the wider community on an issue as fundamental as this one. It's even more concerning that a half-thought through idea didn't get discussed and yet a grant application was made. All those who say that the application is only for $250,000 are entirely missing the point - the entire project would be $2.5 million, this is just the first, initial stage. It's even worse when Jimmy Wales states that: "To make this very clear, no one in top positions has proposed or is proposing that WMF should get into the general "searching" or "try to be Google". [1] Yet that is precisely what is being done here. The WMF appear to have known about this, because they seem to have done a large number of hirings to be dedicated to search - which I hear through contacts was questioned at the time as it seemed an odd way to allocate WMF resources. There have been, in the last week, I believe 5 or 6 influential staff leave the WMF. IN fact, they appear to be haemorrhaging staff currently, with no real sign of any abatement. None of this is at all satisfying to me. I was very, very involved in Wikipedia years ago. I started their Admin Noticeboard, and I did lots of article work, and helped kick off some key things, one of which was the [citation needed] tag which I have to admit I have some mixed feelings about. But for such an important project, it saddens me greatly to say that as an outsider now, it looks like things are being badly mismanaged. I hope for everyone's sake (and not just the folks at the WMF) that this can be resolved. It's not like governance issues can't be addressed - when Sue Gardener was in charge of the WMF, things not only ran like clock-work, but she ensured maximum transparency and we all trusted her implicitly because she earned that trust. I can't say the same for the current Executive team. 1. http://www.theregister.co.uk/2016/02/12/wikipedia_grant_build_search_engine_knight_foundation/ http://www.theregister.co.uk/2016/02/12/wikipedia_grant_buil...
• Rauchg 11y ago
  The biggest problem is the lack of data about what people are searching. It's a catch-22 that's very hard to break in the face of Google's search dominance and ubiquity. By Google being the best, it only becomes better, and introduces a huge barrier to entry to competitors. It used to be possible to know what people were searching for to end up in a given Wikipedia article, but the process is now only asynchronous (and limited) through Webmaster tools[1] In my mind, the most interesting aspect of the announcement should not be how much money they have to spend, but how they plan on solving this paradox. [1] http://webmasters.stackexchange.com/a/60350 http://webmasters.stackexchange.com/a/60350
  • rgovind 11y ago
    I think facebook should be able to build a search engine. Don't know why they don't have one yet
    • dexwiz 11y ago
      Facebook wants a single platform to provide the Web through. If they built a search engine, it would only search Facebook.
      • elmar 11y ago
        Facebook search is broken, twitter search works much better.
        • patrickaljord 11y ago
          It's only broken because most people share with their friends only and search cannot expose their posts. This makes real time search way less efficient than twitter.
          • tallanvor 11y ago
            That's what security trimming is for. You index everything, including the permissions for each item and provide the searcher's permissions (whether it's an acl, a claim, or some other form) as part of the query. Anonymous or unauthenticated users get a "public" claim that only returns results available to everyone.
• abalone 11y ago
  Summary of the approach (p10): "1) Public curation mechanisms for quality; 2) Transparency, telling users exactly how the information originated; 3) Open data access to metadata, giving users the exact date source of the information; 4) Protected user privacy, with their searching protected by strict privacy controls; 5) No advertising, which assures the free flow of information and a complete separation from commercial interests; 6) Internalization, which emphasizes community building and the sharing of information instead of a top-down approach." My first thought: How will transparency impact SEO? Will spammers be able to better game the algorithm when they know its internals? However I am excited at the prospect of a wikipedia-like public curation system for the entire web. I admit I'm flabbergasted that the whole thing ever worked, but it does.
  • jondubois 11y ago
    I'm pretty sure that the spammer argument is just an excuse used by Google to allow them to keep their business practices out of public scrutiny. Google search results are biased in favour of content produced by those who have money and power. Google ranks everything based on popularity - Not based on quality. Popularity and quality are two independent concepts and not necessarily related. That's something which Wikimedia understands but which Google doesn't.
    • bitmapbrother 11y ago
      I'd ask you to cite your claims, but we both know you can't. It's a pity your issues with Google cause you to pollute discussions with BS.
      • userbinator 11y ago
        Do a Google search for anything even slightly obscure, and you're likely to find the first page or so of results filled with highly-SEO'd sites that offer little in the way of deep, detailed content. The smaller sites which do have that content, but just haven't been SEO'd much, have been eclipsed. They're still there, but rendered nearly inaccessible. Interesting discussion on a search engine that does sort of the opposite of Google: https://news.ycombinator.com/item?id=3910304 https://news.ycombinator.com/item?id=3910304
• blairanderson 11y ago
  I support it 100%, but still think they should use search advertising to cover costs and further development instead of asking for donations every year. especially if they can make something that actually does rival google... other companies have spent billions and not gotten very close.
• npguy 11y ago
  Sergey Brin on People Vs Machines - http://statspotting.com/sergey-brin-i-was-wrong-about-semantics/ http://statspotting.com/sergey-brin-i-was-wrong-about-semant...
• qaq 11y ago
  I am financing a $10 project to rival Tesla.
• mabbo 11y ago
  Google's advantage isn't just that they were first, or that their algorithm is the best- it's the CPU resources they have available to keep their data updated faster. Search for any news item and you'll have all articles published more than 2 minutes ago included in your results, all blog posts, everything. They consume it all, and offer the output in near-real-time. Wikimedia don't have the resources to do that. And they especially won't without advertising to pay for it.
• jcrben 11y ago
  This was/is actually an extremely controversial project. The corporation (basically the Executive Director) pursued the grant and the idea without soliciting input or really disclosing it to the community of editors, and eventually one of the community-elected trustees was removed for questioning the lack of transparency. The community has a long list of software improvements that they'd like to see to the core platform. A recent employee survey showed only 10% of WMF staff approved of the Executive Director, probably in large part due to things like this. A critical take on the project as it has been handled: http://permalink.gmane.org/gmane.org.wikimedia.foundation/82112 http://permalink.gmane.org/gmane.org.wikimedia.foundation/82... https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2016-01-13/In_focus https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2... is a pretty good (but dated) overview from Wikipedia's weekly newspaper, but there's a few others in the Signpost and a few blog posts across the web.
  • bravura 11y ago
    A bit more detail is shared in this comment below: https://news.ycombinator.com/user?id=chris_wot https://news.ycombinator.com/user?id=chris_wot
    • yuvipanda 11y ago
      More specific link: https://news.ycombinator.com/item?id=11101262 https://news.ycombinator.com/item?id=11101262
      • chris_wot 11y ago
        Better off here: https://news.ycombinator.com/item?id=11101164 https://news.ycombinator.com/item?id=11101164 :-)
  • deleted 11y ago
    [deleted]
  • runasroot 11y ago
    Wikipedia needs to solve it's management issues before it starts developing new software...
• JayHost 11y ago
  This is great. It feels like we live in an information overload era opposite of North Korea. Search "Are cookies really bad for me" and find an answer that supports what you want to hear. "Live a little" Sponsored by Nesthouse Cookies INC
• grandalf 11y ago
  most of my google searches include a wikipedia result on the first page. I would estimate this could reduce Google's web search revenue by upwards of 40% worldwide.
  • xiphias 11y ago
    Google doesn't make that much money on research quieries. It's mostly your other queries are what enables them to sell ads.
    • grandalf 11y ago
      well nearly everything one might conceivably google has a wikipedia page...
• datashovel 11y ago
  I'm not kidding when I say that if they want to know where to spend the $2.5m I would start with cleaning up their core codebase. IMO Mediawiki open source code is a disaster. EDIT: Not because it's written in PHP. Because it's architected poorly.
  • chris_wot 11y ago
    It's funny you should mention that. That was a point that apparently a number of WMF staff expressed, and it was apparently ignored.
    • yuvipanda 11y ago
      There used to be a team dedicated to making MW Core better / cleaner, but that was lost in a re-org earlier in 2015.
      • chris_wot 11y ago
        That's a darned shame. I'm aware that there are a lot of areas that people want to fix on MW Core. I get really concerned when I hear that the person who holds the vision and direction for the Wikimedia Foundation didn't really participate in it beforehand, and I get even more concerned when I see that she branches off into proposals for search technology that appear to be far outside the scope of Wikimedia projects. Nobody has ever thought search in Wikipedia or the various projects was particularly effective. However, bringing everything together doesn't just involve searching, and frankly there are a number of more pressing governance and community issues that need to be managed. Perhaps I'm being a bit unfair here, but she was profiled when she first joined the WMF Board, and the following was said about her: At the meeting, she described the impact on friends and family of the Chernobyl nuclear disaster, and the difficulty of getting reliable information in the face of “so much secrecy.” Yet we see that this is precisely what happened with this grant proposal. A major grant was applied for and awarded and not even WMF staffers knew about it. You can see on the mailing list that it was a total shock when it was finally revealed. I'm watching this train wreck from afar, but closer than others because some of my friends are deeply involved in Wikipedia and the WMF. I'm always amazed that a leadership change can complete kill an organisation. I've seen it in the corporate world, and I see it all the time in the volunteer world as well. The Wikimedia Foundation seems to be yet another victim of the appointment of a clueless leader, with no experience in the area or with the group they are meant to be leading, thrashing around, making changes without really understanding how systems work, the history of the organisation or relying on the experience and sage advise of the many expert and dedicated people around them, ultimately leading to a great deal of unnecessary turmoil, ill-will and frankly destruction in their wake.
• jccalhoun 11y ago
  I am guessing this has a different focus than their previous attempt at making a search engine, wikia search, which they abandoned fairly quickly https://en.wikipedia.org/wiki/Wikia_Search https://en.wikipedia.org/wiki/Wikia_Search
  • Washuu 11y ago
    Wikia is not part of the Wikimedia. https://en.wikipedia.org/wiki/Wikia#Relationship_with_Wikipedia https://en.wikipedia.org/wiki/Wikia#Relationship_with_Wikipe... http://community.wikia.com/wiki/Help:Wikimedia http://community.wikia.com/wiki/Help:Wikimedia (Edit: Who decided that enter is not equal to enter?)
• languagehacker 11y ago
  As a former Wikia employee, I am somewhat of a MediaWiki insider. I sped Wikia's search engine up by several orders of magnitude and then went on to pilot a number of NLP/machine learning initiatives in the company. Jimmy Wales' already tried to make a "Google Killer" ten years ago. It was tilting at windmills to say the least. Letting individuals help manage algorithmic search results was harder than you could imagine. Let's not even get into the difficulty of building an effective crawler. One of Wikia's former CEOs, Gil Penchina, notoriously undervalued search as a result of this very public gaffe. By the time I came in, it took over five seconds to do a simple on-wiki search. Searching across wikis took so long they actually just sent the search to Google and had you abandon the site. I personally fixed a lot of these problems, and that part was pretty cool. So now let's get to the subject at hand, which is a search feature based on an authoritative knowledge graph. Something like this should adequately surface factual information in an intuitive manner -- optimally based on natural language. Wikia already tried this, too. They brought on a very seasoned advisor who played a crucial role in the semantic web movement far back into the early oughts. I remember going to semantic web meetups in Austin when I was in grad school quite some time ago now to hear this guy talk. This guy was essentially the SF-based manager or lead for a small team located in Poland whose job it was to take some of the "structured data" at Wikia and attempt to build some kind of knowledge graph on top of it. This project was unsuccessful. So why did it fail? We'll start with a lack of product direction. Wikia had and probably still has a very junior product organization that is mostly interested in the site's UI and (recently) a focus on "fandom" (yuck). The team allocated to the project was based in Poland (Poznan, to be exact), and primarily kids coming out of a technical school on their first job. Your assumption about communication being a problem would be correct. However, the subject matter expert was so entrenched in his area of specialization, the problem was even more compounded on the native English-speaker side. There was too much getting in the weeds, and not enough focus on incremental progress. To make things worse, they tried using a proprietary, not-ready-for-primetime data store because it most closely matched the SME's preconceptions on how the data should be structured. There was absolutely not an existing business use case for this data store, and problems getting it to work turned even building a simple demo into a death march. Either way, what I'm saying is, $250,000 is not enough to solve this problem. We have attempted to solve this problem before in the MediaWiki world. It's not going to magically get better. To make something like this work, you need: 1) Best-in-class UX people who would know how a knowledge graph provides a significant improvement over existing solutions 2) Leadership that can bridge the gap between SMEs and implementers 3) Very skilled engineering resources with backgrounds in less conventional technologies This is a massive investment that no one is willing to spend on what is essentially a media play. About six months later, I had built a proof-of-concept that sucked data out of MediaWiki Infobox templates into Neo4j, a well supported graph database. I was able to answer questions like, "Which cartoon characters are rabbits", and "What movie won the most Oscars in 1968" using the Cypher query language. At that point in time, Wikia had decided they were tired of investing in structured data, and wanted to re-skin the site for a third time in as many years to make it look more like BuzzFeed. Structured data is cool. In many cases, unsupervised learning may be what you're actually looking for. But in the end it has to satisfy a real user's needs. Wikipedia has five million English articles. Wikia has over 20 million. As far as capitalizing on this wealth of knowledge, the devil is truly in the details. But it's a real shame that all of that information isn't put to better use than to encourage the socially maladjusted to take quizzes over which anime character they're more like.
• doyoulikeworms 11y ago
  If nothing else, I hope it improves the currently abysmal search features for Wikipedia today.
  • lacksconfidence 11y ago
    turns out that is the only thing this project is about. There is no web crawler, there is no external content. The grant and the money WMF is spending are being spent to improve internal search at wikipedia.
• frik 11y ago
  Good luck. We definitely need more search engines. (That Google announced to lower the PageRank(R)/site score for non-HTTPS sites is a clear indicator that they about to cross the line (monopoly). And no, DDG and most others are "just" meta search engines that rely on Yahoo Boss ($$$) which future is uncertain and relies on Bing.) There was "Wikia Search" by Wikipedia founder Jimmy Wales: "Wikia Search was a short-lived free and open-source Web search engine launched by Wikia, a for-profit wiki-hosting company founded in late 2004 by Jimmy Wales and Angela Beesley. Wikia Search followed other experiments by Wikia into search engine technology and officially launched as a "public alpha" on January 7, 2008. The roll-out version of the search interface was widely criticized by reviewers in mainstream media. After failing to attract an audience, the site closed by 2009." https://en.wikipedia.org/wiki/Wikia_Search https://en.wikipedia.org/wiki/Wikia_Search I used Wikia Search back then, it was good enough (like Bing in comparison to Google back then). It was based on Apache Nutch and Solr/(Hadoop(?)/Lucene ... Maybe you can rely on Lucene or SphinxSearch projects to kick-start.
  • grapehut 11y ago
    Google doesn't want to lower the PageRank of http sites, it just wants to use http vs https as a feature in ranking. That isn't particularly surprising, I would be willing to bet google already uses hundreds of such features (one of which might be PageRank).
• melted 11y ago
  That's how much 5 qualified software engineers would cost to employ for a year (gross, including compensation, benefits, payroll taxes, office space, hardware, etc, and that's on the low end of the range). Good luck with that.
  • treve 11y ago
    It costs half a million to employ one software engineer for a year?
    • exolymph 11y ago
      Cost per employee is much greater than just their salary.
    • melted 11y ago
      A software engineer qualified to work on this kind of thing is worth about $350K in combined compensation on the market right now. Typically half of that is base, while the other half is stock and other taxable benefits. The number can be higher. This is the cost of just compensation to the company, excluding the payroll tax. You can, of course, find someone a lot cheaper, but then you'd be a fool to expect the result to be anywhere near as good as what Google can pull off, because if that someone could do what Google can, why would she work for half the compensation instead of applying to Google or FB or whoever pays competitive salaries these days.
  • bin01 11y ago
    They plan to hire 8 engineers, 2 data analysts, and 4 team leads. Relevant details on page 8. https://twitter.com/dtunkelang/status/699049083806167040 https://twitter.com/dtunkelang/status/699049083806167040
    • melted 11y ago
      Why do they need 4 team leads for 8 engineers?
      • 11y ago
• idibidiart 11y ago
  Remember this? https://evolvingtrends.wordpress.com/2006/06/26/wikipedia-30-the-end-of-google/ https://evolvingtrends.wordpress.com/2006/06/26/wikipedia-30...
• NKCSS 11y ago
  Weird; they have anual drives to raise money to keep the site running; would not expect that the'd have 2.5m lying around to do pet projects like this...
  • marincounty 11y ago
    Most non-profits, especially the ones that are always asking, usually have a lot of funds. Before I give, I go to guidestar, hit free preview(they try to trick you into a paying membership), download last few years of 1040's, and see if everything looks copacetic. I look at who is making the most money. Their is usually one person making a very good living. California non-profits are much easier to scrutinize than Deleware non-profits.
    • ryacko 11y ago
      They are required to give you the 990 form if you email them usually. In anycase, https://wikimediafoundation.org/wiki/Financial_reports https://wikimediafoundation.org/wiki/Financial_reports And the internet archive is a lot more deserving.
      • rpgmaker 11y ago
        I believe in internet's archive mission even if I don't use the site that often but I use wikipedia too much, I can't justify not donating to them when they need it.
    • greglindahl 11y ago
      FWIW the archive has a lot of 990s archived at [1]. We love Wikipedia, why not donate to both? :-) [1] https://archive.org/details/IRS990 https://archive.org/details/IRS990
  • merb 11y ago
    the same thing I was thinking.
  • exDM69 11y ago
    If you look at the papers in the link, a private foundation is donating 850k solely for this purpose. I didn't read all the legalese in the paper, not sure where the rest is coming from. So apparently it's not money of the Wikimedia foundation funding this project.
• franky303 11y ago
  $250000 != $2.5M
  • nkjoep 11y ago
    basically, no one is reading the actual content meh. It's 250k USD not 2,5M USD
  • chris_wot 11y ago
    Part of a 2.5 million dollar grant, this is the first stage.
• jonathankoren 11y ago
  I've read a log of the inside wikimedia links here, and I'm confused about all the talk of gnashing of teeth and rending of cloth. This is controversial because some want to pay down technical debt rather than have a small team do knowledge graph search? Okay...
  • chris_wot 11y ago
    It's not a small team compared to the size if the organisation. And you mischaracterise the situation: this is a problem with engagement, transparency and openness.
    • jonathankoren 11y ago
      Yeah the relative merits of the initiative seem to be besides the point. If you have a toxic environment, even a proposal to cure all disease for everyone for free will attract derision. It's hard to get worked up over some other team's morale. If its such a crappy place, just quit. The could probably literally go across the street and get a new job. I don't really care. It's all way too inside baseball for me.
      • chris_wot 11y ago
        You cared enough to comment. This wasn't just a place of work for me, I volunteered my time because I believed in what they were doing. You might never have contributed in a significant fashion to Wikipedia and other WMF projects, but I did. Sure, I didn't get employed, but then again I know a lot of people I met and have continued to be friends with who are still deeply involved. You may not care about your friends' morale, and you might think it's easy for people to "go across the street and get a new job", but then you seem like a pretty thoughtless person. Of course, you've not understood at all what the larger issues are. You must have a bit of a comprehension issue, because I supplied quite a few links that you apparently read that explained the underlying problems. Just remember though: you don't care :-)
        • jonathankoren 11y ago
• osoti 11y ago
  Top Places to Eat in U.S. According to Yelp http://goo.gl/18pkmx http://goo.gl/18pkmx
• Aissen 11y ago
  Comparison point: it's Bing's budget for about 5 hours.
• veritas213 11y ago
  a JV might be a better idea...$2.5M isnt that much money and i doubt it will even come close to being useful relative to the other search engines
• inaudible 11y ago
  Anyone want to hazard a guess at the technology they plan to implement to get this started? Surely this is not designed to be written from scratch, so.. - Are they using known lexical & semantic scanners? - Is it focused on English language first? - What crawlers will scan content? - I'll assume it's an open platform, but license for contributors? - What database architecture will hold the graph? - How does it know the mark of authority, and is this primarily based on human input learning or machine learning? I'm sure $2.5M wont touch the sides, but maybe if it's a well directed project, with healthy user contribution, based on interesting technologies they might develop a good backbone architecture. Ambitious for sure.
• aaron695 11y ago
  Wikipedia main asset seems based around user contributions and human interactions not programming or hard algorithms, this seems quite a leap into another field with not much money. Bing cost MS $5.5 Billion in their field of expertise http://www.geek.com/news/bing-has-cost-microsoft-5-5-billion-since-launch-1423117/ http://www.geek.com/news/bing-has-cost-microsoft-5-5-billion...
• exDM69 11y ago
  The link headline is highly editorialized, there is no mention of "google" or "rivalry" in the pdf in the link.
• lossolo 11y ago
  This is like competing with Intel in CPU server market, where he has 98% market share. So they are trying to compete using 2.5 mil dollars with software backed by multi billion dollars, hundreds thousands of servers, tons of data, thousands of developers, ML integration etc.? Good luck with that. Many tried backed by x times more resources than this 2.5 mil, unfortunately all failed.
• naveen99 11y ago
  Computing is so cheap now, google isn't going to be so dominant on text search for long. Their money is needed for video and pictures and audio, but the text internet can be cached whole by small entities now. Maybe Wikipedia should launch a video encyclopedia to try to provide a 5 minute video of every article, for people who like videos more than reading.
• chrisra 11y ago
  So this is why they've been asking for donations? Made it seem like they're on the ropes.
• gersh 11y ago
  As a former NLP engineer and former wikiHow engineer, I have some perspective on this. Google has included more and more information from Wikipedia. Furthermore, Google includes snippets of external websites in the knowledge box on more and more pages. How long will be until can algorithmically generate its own Wikipedia articles? Wikipedia relies upon coming to its site for contributions and donations. Without search, Wikipedia risks being subsumed by Google. They have a difficult position of thinking about the future without pissing off Google. Computers are getting more and more powerful. Wikipedia needs to do stay relevant. I think this is the right decision.