5 ms·

We need a Wikipedia for data

• dood 19y ago
  Sounds a lot like what I think Freebase [http://www.freebase.com/ http://www.freebase.com/] are trying to do.
• danohuiginn 19y ago
  http://theinfo.org/get/data http://theinfo.org/get/data http://www.ckan.net/package/list http://www.ckan.net/package/list And Freebase, as mentioned by dood
• cousin_it 19y ago
  Free vector map data is growing: http://openstreetmap.org http://openstreetmap.org . They recently got donated the whole Netherlands.
  • craig-faber 19y ago
    Plus India. (It's not up yet.)
• prakash 19y ago
  Isn't Metaweb's Freebase (http://www.freebase.com/ http://www.freebase.com/) doing this? Check it out if you have not seen this before, these guys are Rockstars!!
  • Readmore 19y ago
    Exactly my thought. Freebase is awesome, I just wish they didn't use JSON.
    • mrtron 19y ago
      Why no JSON?
      • Readmore 19y ago
        I'll be the first to admit it might just be me ;) I prefer XML and haven't used JSON extensively enough to be proficient with it. If there are pluses to using JSON over XML I'm ignorant to them.
        • pistoriusp 19y ago
          "Readmore" perhaps? ;)
          • Readmore 19y ago
            HA, nice one! Any suggestions on where to start?
            • pistoriusp 19y ago
              Here you go: http://json.org http://json.org http://en.wikipedia.org/wiki/JSON http://en.wikipedia.org/wiki/JSON I use JSON when I'm doing Ajax things. Jquery and Django both have methods to encode/ decode JSON.
    • kirubakaran 19y ago
      I like them more because they use JSON.
• jfno67 19y ago
  It's like freebase all right, but the article has something right about adoption. He points out some big company would have to donate a great starting dataset to drive adoption. I think this is one problem with Freebase. Another problem I see is the structuring of the data, it is a hurdle to sharing. Finally the last problem I see is Data is currently very much seen as a competitive advantage. When Google introduce functionalities to make corrections on Google Maps, I don’t think they do it to share with everyone else. They do it because they want Google Maps to be the destination with the best possible data. A wiki of data is a great idea, but I’m not sure we have the solution yet.
  • pius 19y ago
    It's like freebase all right, but the article has something right about adoption. He points out some big company would have to donate a great starting dataset to drive adoption. http://news.ycombinator.com/item?id=157966 http://news.ycombinator.com/item?id=157966
    • jyu 19y ago
      This improves the (easy) technical hurdles, but how about (very difficult) social, economic, and bureaucratic ones? One of the biggest competitive advantages companies have is data. Like the article and previous comment already said, adoption is the hardest barrier. Unless someone can provide a compelling reason for companies/scientists/etc to give data or data access, there really isn't much else to discuss. I think sites like mashery and dapper.net are going in the right direction, by providing good licensing rights and monetization controls that can incent large companies with reliable datasets to participate.
    • jfno67 19y ago
      It's a start for sure and it's great that freebase does this. But what I refer to is more like Google gives all their business location listings to Freebase. I don't think this will happen, but maybe something of less value while being still significant.
• Dylanfm 19y ago
  Regarding (some) stocks: http://blog.infochimps.org/2008/03/20/stock-market-dataset-is-up/ http://blog.infochimps.org/2008/03/20/stock-market-dataset-i...
• cousin_it 19y ago
  Joel had an article on "commoditizing your complements": http://www.joelonsoftware.com/articles/StrategyLetterV.html http://www.joelonsoftware.com/articles/StrategyLetterV.html . Of course we want to commoditize data, to raise the value of hackers (raise the expected return from hacking). On the other side, data companies want to commoditize hackers, to raise the value of data... a process we are sure to resent.
  • allertonm 19y ago
    Absolutely, my first thought on reading this was "Why won't these data companies understand they should throw away their business so us coders can make some money?"
• yters 19y ago
  Have you heard of freebase? From what I've read online, I think they are doing something like this.
• mdemare 19y ago
  Bret is spot on. Open data would unlock a vast amount of wealth. Other suggestions: A collection of various kinds of texts, translated into 20-30 different languages. 100 million words (per language) would be fine. 20 minutes of text, spoken in thousands of different voices/accents.
  • david927 19y ago
    An open translation dictionary is a fantastic idea. You would have to be careful to clarify the context. A word in one language can often translate to several words in another language, for example. But I think it's do-able. And I think you meant 100 thousand words per language. :-)
    • mdemare 19y ago
      I don't mean a dictionary (also a good idea), I meant texts: articles, novels, blog posts, transcripts of conversations, etc. As for dictionaries, there's wiktionary, but it's broken because it's based on words, not meanings, so you'd need 30*29 translations for each word. Mmm, maybe I should do it...
      • david927 19y ago
        But a good translation is really, really tough to do. It's an art form. Just having a basic context isn't usually enough. "I'm going there to see you." Am going on foot or by car? (Gehen vs. fahren; chodit vs jezdit.) And how well do I know you? Each culture has a different point where they go from formal (Vous/Sie/Vy) to informal (tu/du/ty), based on how well you know someone. I'm telling you, it's tough.
• aneesh 19y ago
  "No one really wants factual data accuracy and completeness to be their competitive advantage" Some people actually do build their business on this. It's not the most sustainable business model, but in today's world, good data can be - and is - a competitive advantage. In an ideal world, should it be that way? Maybe not. I think that's what this post is getting at.
• paddy_m 19y ago
  Two years ago I wrote http://formula1db.com http://formula1db.com , to teach myself sql. To get the data, I had to screen scrape formula1.com . Once I had the data, learning SQL became a joy. I haven't done much with the data since I built the site, but I am considering open sourcing it. What I don't understand is, why don't sport leagues open source their data. What do they lose? Its a good think that people are so excited about your sport, that they build custom apps based on it. Sadly sport leagues don't seem to get it, I remember the MLB cracking down on a fan generated datatbase of baseball statistics a while ago.
  • anamax 19y ago
    They think that they can't sell the data if they open source it. They probably also have control issues. They think that a popular application using "their data" is necessarily lucrative and they think that they should get a huge hunk of that money.
  • jsackmann 19y ago
    Actually, MLB data is fairly close to being open-sourced. Historical data IS open-sourced (though not by MLB itself): http://baseball1.com/content/view/57/82/ http://baseball1.com/content/view/57/82/ http://retrosheet.org/ http://retrosheet.org/ Current major and minor league data is available as well, though MLB will crack down on anyone who is trying to make money off of derivative products. Here's where you'll find it, as XML: http://gdx.mlb.com/components/game/ http://gdx.mlb.com/components/game/ One can do pretty cool stuff with all of it, and many people have, despite the fact that we can't make money off of it: http://minorleaguesplits.com/ http://minorleaguesplits.com/ (my site) http://baseball.bornbybits.com/2008/pitchers.html http://baseball.bornbybits.com/2008/pitchers.html (analysis based on detailed pitch speed / break information that MLB started collecting last year.)
• bct 19y ago
  It's so painfully obvious that this is a good idea, and yet people are skeptical of the Semantic Web, which has been promoting this idea for almost a decade (and doesn't require centralization). Why?
  • cousin_it 19y ago
    Here's three anti-SemWeb articles that try to answer your question: http://www.well.com/~doctorow/metacrap.htm http://www.well.com/~doctorow/metacrap.htm http://www.shirky.com/writings/semantic_syllogism.html http://www.shirky.com/writings/semantic_syllogism.html http://blahsploitation.blogspot.com/2005/09/i-always-figured-that-at-least-rdf-was.html http://blahsploitation.blogspot.com/2005/09/i-always-figured...
    • bct 19y ago
      Why is "metacrap" a problem for the semantic web, but not for data-Wikipedia? The Shirky article is a well-known strawman. Thanks for the pointer to the last one, I'll read it when I get a chance.
      • cousin_it 19y ago
        I really shouldn't be doing this... > Why is "metacrap" a problem for the semantic web, but not for data-Wikipedia? Because Wikipedia is centralized, and the SemWeb isn't. > The Shirky article is a well-known strawman. DH3. Contradiction
        • bct 19y ago
          I don't want to turn this into a huge debate either, but those articles (and uncritical readings of them) have set the web back years. > Because Wikipedia is centralized, and the SemWeb isn't. If data-Wikipedia and a television station are both publishing data about when your favorite show is on that station, who are you more likely to believe? Obviously you need to be careful about where your data comes from, but a single centralized source is not necessarily more trustworthy than many carefully selected sources. Blind crawling isn't (and will probably not be) the norm for data collection on the semantic web. > DH3. Contradiction Heh, got me there. Shirky's thesis is based on the idea that making inferences from data is the ultimate purpose of the semantic web. But linked, machine-readable data--that is, the semantic web--is useful even if inferencing is useless. I don't think this is a claim that needs evidence, it should be fairly obvious. Shirky's article's portrayal of the semantic web has little to do with the real thing. Here's a much broader debunking of it: http://www.poorbuthappy.com/ease/semantic/ http://www.poorbuthappy.com/ease/semantic/
• omarseyal 19y ago
  I've thought about this before... Wikipedia covers unstructured data, but we need something to be its analog for structured data. Others have pointed out freebase (which kicks ass IMO), but there's also swivel (http://www.swivel.com/ http://www.swivel.com/). Swivel's correctness model, however, doesn't seem to be the same open idea that freebase is. It seems more to be focused on data "authorities" and "official" data providers as opposed to just an "accuracy of the masses" system that the more open sites rely on.
• redorb 19y ago
  the gate keepers, of this information are profiting off of it. As long as that continues they will resist change.
• asp742 19y ago
  There was a rumor that Google would host scientific data http://blog.wired.com/wiredscience/2008/01/google-to-provi.html http://blog.wired.com/wiredscience/2008/01/google-to-provi.h... I am not sure if this is true or what the status is. Open Data Commons has an interesting license for open datasets http://www.opendatacommons.org http://www.opendatacommons.org
• andreyf 19y ago
  I remember seeing this company that organized data... oh yeah, they also wrote most of my textbook and aquired Reuters: http://en.wikipedia.org/wiki/The_Thomson_Corporation http://en.wikipedia.org/wiki/The_Thomson_Corporation Could they really end up being the Encarta Encyclopedia of data?
• deleted 19y ago
  [deleted]
• neilc 19y ago
  There is also Swivel: http://www.swivel.com/ http://www.swivel.com/
• kirubakaran 19y ago
  http://theinfo.org/ http://theinfo.org/
• mrflip 19y ago
  There's largely no such thing as "closed source" data. Many of the restrictions people claim on publicly distributed data are bogus: you cannot claim copyright on a comprehensive collection of facts. http://www.iusmentis.com/databases/us/ http://www.iusmentis.com/databases/us/ http://blog.infochimps.org/2008/04/02/good-neighbors-and-open-grazing/ http://blog.infochimps.org/2008/04/02/good-neighbors-and-ope... I don't think baseball is cracking down on people making money on this, unless they infringed their (quite reasonable) hot news claims to the real-time data. Baseball is the leading example of why giving away most of your data is the best use of it. The sport of baseball -- the way it's played on the field, the way players are scouted and trained, and the way it's enjoyed as a fan (Moneyball? Fantasy Sports?) -- have been revolutionized by amateurs making use of free open data. If you give out the great bulk of your data, people will be enhancing it with metadata, building tools on top of it, and most importantly connecting it to the rest of humanity's knowledge store and mining it for connections you'd have never conceived. Giving out "up to last month" or "daily intervals" will grow sharply the market for "real time" or "second-by-second". Baseball's mission statement concerns bats, bases, butts and seats -- not visualizing correlations among heterogeneous data stores. By releasing their data for free they let the smartest people in the world have the opportunity to perform that second task for free. We're about to enter the age of ubiquitous information. Drawing these data stores into open formats, making them discoverable, and interconnecting them across knowledge domains presents explosive opportunities. But who will own this data and what access will they allow? If you want to help ensure that the answer is 'everyone' and 'all of it', come join the http://infochimps.org http://infochimps.org project, a free open community effort to build an Allmanac of everything.
• zandorg 19y ago
  My degree thesis was on this subject, except this... It stores the data as English text, parseable into Prolog predicates (ie, key/value data chunks), and you have a special editor which shows you the predicates generated by the English you just wrote. Does this sound useful to anyone? I'd love to pick up on it. The parser for English I wrote is in Lisp, and works very well (it's recursive, etc). And of course, you don't have to make a new database, you just do the usual Wiki editing and just make sure the text you enter is parseable by the latest parser.
• jexe 19y ago
  Finding high quality data is very expensive, and very well-guarded with license restrictions once you do cough up for it.. Seems like factual data will inevitably gravitate toward free, though. The sooner the better.
• nreece 19y ago
  A Dataset Catalog: http://www.datawrangling.com/some-datasets-available-on-the-web.html http://www.datawrangling.com/some-datasets-available-on-the-...
• pius 19y ago
  http://dbpedia.org http://dbpedia.org
• nazgulnarsil 19y ago
  we need a wikipedia for ______ is generally true. the problem is getting all the knowledgeable people involved with ______ to contribute their time.