4 ms·

what would be the benefit of having the web as a dataset when it is rife with copyright laws (see craigslist), monopoly businesses who viciously protect their h

what would be the benefit of having the web as a dataset when it is rife with copyright laws (see craigslist), monopoly businesses who viciously protect their human uploaded content and profiles, and majority of the data from the web being useless without a context and purpose of the searcher? I'm just curious as to how commoncrawl compares with kimonolabs and import.io as they seem to have the same goal of creating an internet as a dataset, or an API. I can't help but feel like it's just solving another 'semantic web' problem that nobody asked for. It is funny that the most demanding customers of semantic web are also the ones who are willing to pay the least amount of time and money.

pasteurquadrant 12y ago
+1
jobposter1234 12y ago
Regarding copyright issues, I believe you can still use copyrighted data as long as it's transformed. E.g., building language models, or doing a search engine like Google. In fact, I can think of more computational uses for copyrighted data, while on the "banned" side, I can only think of... SEO. Regarding point two: "monopoly businesses who viciously protect their human uploaded content". I spend a lot of time scraping these monopoly businesses, and it seems to me they do a decent job of letting their users decide what data is exposed. Facebook, linkedin, and google all are decent about letting me scrape their public info. That's all I have a right to -- private info should stay private, at the behest of the owner (the User in UCG). You are correct regarding the third point, but I don't see that as a problem. This isn't a solution in search of a problem -- it's a problem without a solution at the moment. Here's a toy example of something I'd like to do: calculate the positive / negative sentiment of commenters at particular baseball fan sites, so I can hide the content I don't like, and show that which I do. Having a common crawl of the site would be immensely useful (and is indeed a prereq) for this. I wouldn't need to republish it, just compute on it.

pasteurquadrant 12y ago

jobposter1234 12y ago

Regarding copyright issues, I believe you can still use copyrighted data as long as it's transformed. E.g., building language models, or doing a search engine like Google. In fact, I can think of more computational uses for copyrighted data, while on the "banned" side, I can only think of... SEO. Regarding point two: "monopoly businesses who viciously protect their human uploaded content". I spend a lot of time scraping these monopoly businesses, and it seems to me they do a decent job of letting their users decide what data is exposed. Facebook, linkedin, and google all are decent about letting me scrape their public info. That's all I have a right to -- private info should stay private, at the behest of the owner (the User in UCG). You are correct regarding the third point, but I don't see that as a problem. This isn't a solution in search of a problem -- it's a problem without a solution at the moment. Here's a toy example of something I'd like to do: calculate the positive / negative sentiment of commenters at particular baseball fan sites, so I can hide the content I don't like, and show that which I do. Having a common crawl of the site would be immensely useful (and is indeed a prereq) for this. I wouldn't need to republish it, just compute on it.