3 ms·

It seems like nothing like this currently exists in a centralized, non-distributed way. Why add the complexity of a p2p network into an unproven concept? Is it

by yid 12y ago

It seems like nothing like this currently exists in a centralized, non-distributed way. Why add the complexity of a p2p network into an unproven concept? Is it purely to save on the cost of indexing and serving queries? > Scraping Google is a bad idea, which is quite funny as Google itself is the mother of all scrapers, but I digress. It's not really "funny"/ironic/etc -- Google put capital into scraping websites to build an index, and you're free to do the same, but you shouldn't expect Google to allow you to scrape their index for free. EDIT: just saw this: > Right now, PLOS, eLife, PeerJ and ScienceDirect are supported, so any paper you read from these publishers, while using the extension, will get indexed and added to the network automatically. Yeah, they're not going to like that. You might want to consult a lawyer.

Blahah 12y ago
Actually PLOS, eLife and PeerJ are all Open Access publishers and explicitly condone this kind of scraping of their sites. They want to promote reuse. ScienceDirect is owned by Elsevier and is a different kettle of fish. One we all hope to boil in the very near future. However, the title, authors, etc. are not copyrightable and are explicitly free for indexing in the ToS. This is not crawling, only scraping from an already rendered page. They really can't complain.
- gioele 12y ago
  > This is not crawling, only scraping from an already rendered page. They really can't complain. You may want to have a look at http://en.wikipedia.org/wiki/Sui_generis_database_right http://en.wikipedia.org/wiki/Sui_generis_database_right
  - juretriglav 12y ago
    Thanks for pointing this out, but IANAL; could you briefly explain what this is about in the context of Scholar Ninja?
    - gioele 12y ago
      The problem is that the phrase "This is not crawling, only scraping" is to be taken with a grain of salt. If you find a page with a lot of information and you "just scrape it", you may still be subject to "sui generis" database rights, i.e. you are probably not allowed to reuse the data you just got. You can say "it is only an alphabetical list of names and titles", but you have to remember that the "sui generis" DB rights have been created to protect phone books, and almost every compilation of data is more complex than a phone book.
      - juretriglav 12y ago
        Thanks for that explanation, makes sense now. I really don't want to upset anyone and would like Scholar Ninja to be perceived as an additional value to all involved parties. For publishers specifically, it surely must be in their interest that people are able to find their papers. I'm hoping that because Scholar Ninja is also framed as an open-source, non-profit initiative, I'll upset people even less. Fingers crossed.
juretriglav 12y ago
The P2P network approach here is important for two reasons, one is that I do not have fulltext access to journals and researchers using this extension do, and second, I do not have the resources to run a centralized search engine, which would be expensive, as you say, because of index/server costs. The fact that it's an unproven concept is exciting to me; that and the fact that it's using very modern technologies to solve an existing problem. If nothing else, maybe this project can server as a cool demo for the underlying tech, i.e. WebRTC. To my knowledge it is the only keyword-base search engine based entirely in the browser. I agree with your remark about Google, I was trying to be witty but often my humor fails me, as my family and friend will be eager to confirm :) Given that Google doesn't have public APIs, even paid ones, for any of their search services, leads me to believe the numbers just don't add up for them. With regards to angering publishers: I really really do not want to come on their bad side, and I can't see how this project could. It's mission is to help users discover content that they have, help them find the right papers, which are still hosted by publishers. Never will Scholar Ninja be used to circumvent paywalls or share paywalled fulltext papers, this is just not in anyone's interest. Scholar Ninja only indexes pages you read anyway, so it doesn't cause any additional load on servers, and it only contains keyword references to documents, e.g. "cancer": ["10.1371/journal.pmed.0010065", ...], which enables us to do keyword searching.

by yid 12y ago

Blahah 12y ago

Actually PLOS, eLife and PeerJ are all Open Access publishers and explicitly condone this kind of scraping of their sites. They want to promote reuse. ScienceDirect is owned by Elsevier and is a different kettle of fish. One we all hope to boil in the very near future. However, the title, authors, etc. are not copyrightable and are explicitly free for indexing in the ToS. This is not crawling, only scraping from an already rendered page. They really can't complain.

gioele 12y ago
> This is not crawling, only scraping from an already rendered page. They really can't complain. You may want to have a look at http://en.wikipedia.org/wiki/Sui_generis_database_right http://en.wikipedia.org/wiki/Sui_generis_database_right
- juretriglav 12y ago
  Thanks for pointing this out, but IANAL; could you briefly explain what this is about in the context of Scholar Ninja?
  - gioele 12y ago
    The problem is that the phrase "This is not crawling, only scraping" is to be taken with a grain of salt. If you find a page with a lot of information and you "just scrape it", you may still be subject to "sui generis" database rights, i.e. you are probably not allowed to reuse the data you just got. You can say "it is only an alphabetical list of names and titles", but you have to remember that the "sui generis" DB rights have been created to protect phone books, and almost every compilation of data is more complex than a phone book.
    - juretriglav 12y ago
      Thanks for that explanation, makes sense now. I really don't want to upset anyone and would like Scholar Ninja to be perceived as an additional value to all involved parties. For publishers specifically, it surely must be in their interest that people are able to find their papers. I'm hoping that because Scholar Ninja is also framed as an open-source, non-profit initiative, I'll upset people even less. Fingers crossed.

juretriglav 12y ago

The P2P network approach here is important for two reasons, one is that I do not have fulltext access to journals and researchers using this extension do, and second, I do not have the resources to run a centralized search engine, which would be expensive, as you say, because of index/server costs. The fact that it's an unproven concept is exciting to me; that and the fact that it's using very modern technologies to solve an existing problem. If nothing else, maybe this project can server as a cool demo for the underlying tech, i.e. WebRTC. To my knowledge it is the only keyword-base search engine based entirely in the browser. I agree with your remark about Google, I was trying to be witty but often my humor fails me, as my family and friend will be eager to confirm :) Given that Google doesn't have public APIs, even paid ones, for any of their search services, leads me to believe the numbers just don't add up for them. With regards to angering publishers: I really really do not want to come on their bad side, and I can't see how this project could. It's mission is to help users discover content that they have, help them find the right papers, which are still hosted by publishers. Never will Scholar Ninja be used to circumvent paywalls or share paywalled fulltext papers, this is just not in anyone's interest. Scholar Ninja only indexes pages you read anyway, so it doesn't cause any additional load on servers, and it only contains keyword references to documents, e.g. "cancer": ["10.1371/journal.pmed.0010065", ...], which enables us to do keyword searching.