5 ms·

Perhaps a better approach would be building an open source www index or even a full current cache - as an enabler for people to build their own search engines?

Perhaps a better approach would be building an open source www index or even a full current cache - as an enabler for people to build their own search engines? Right now it is extremely difficult to build your own web crawler that would compete with Google. And that is not because of the technology, but because multiple sites will prevent your bot from accessing them if you're not Google or Bing - either through robots.txt, or through directly banning your IP if it's trying to crawl and it's not a confirmed google-bot. Having a non-profit, open source, crawler that keeps an up to date index (or web cache) of the web would help competition spring up.

nix23 4y ago
> Perhaps a better approach would be building an open source www index or even a full current cache - as an enabler for people to build their own search engines? That's a excellent idea! In the spirit of open-data, and people can do with it what they want.
- artificial 4y ago
  I think this is a great idea. How does this work with copyright? Search engines seem to be able to download a reproduce content from scraped pages (and wrap it in ads, and derive content from it) this is called “indexing” when they do it but scraping when everyone else does it.
  - bosie 4y ago
    > and wrap it in ads, and derive content from it i am probably missing something but can you give an example where this happens?
    - Flashtoo 4y ago
      E.g. on Google if you search for "how to tie a tie", a little info box may pop up with step by step instructions. This content is taken from some website, but that website gets no page hits or ad revenue. Instead, Google gets to serve ads on the search engine results page. (I don't know if this happens for this specific example, but Google does this for some searches)
      - jefftk 4y ago
        > that website gets no page hits Part of why sites participate in the infobox program is that in practice you do get quite a lot of hits from it: many people click through to see the answer in context.
      - bosie 4y ago
        Ok, I just tried but don't see that info box but that is exactly what i was asking for. Thank you very much, did not think of that.
    - zo1 4y ago
      I think they're referring to how Google "extracts" answers from your website and shows it on the search results page. Effectively meaning that the user doesn't even need to go to your site to get the answer, because Google extracted it and gave it to them directly.
      - thfuran 4y ago
        It seems to me that what they usually extract is some junk only vaguely related to the query and often cut apart and reassembled in a way that's just wrong.
daoudc 4y ago
The data is available! Not currently documented but check out the code for the API.
- onionisafruit 4y ago
  I'm curious if you think a co-op would be feasible between Mwmbl and other like-minded crawlers who are interested in taking a divide-and-conquer approach to crawling the web.
deleted 4y ago
[deleted]
closedloop129 4y ago
Why is https://commoncrawl.org/ https://commoncrawl.org/ not enough?
- dewey 4y ago
  Isn't that one more data sets for ML other research purposes instead of a highly up to date search index (For example with news from a few minutes ago).
  - jka 4y ago
    Roughly speaking, yep - Common Crawl provides a sizable chunk of web data (420 TiB uncompressed, over 3 billion unique URLs, as of May 2022; historic statistics here[1]), and is updated on monthly basis. Not near-real-time, true, albeit relatively fresh. A question to ask could be: how often do users care about information from a few minutes ago, compared to information that has been available for a longer duration of time? [1] - https://commoncrawl.github.io/cc-crawl-statistics/ https://commoncrawl.github.io/cc-crawl-statistics/
    - flexie 4y ago
      Isn't that more a question of adding to the mix frequent scraping of - a few thousand news-sites (like nyt.com, bbc.co.uk), - a few thousand very popular blogs (based on what influencers people search for), - a handful of social media sites (e.g. Twitter), - a few hundred databases in areas like weather, airlines, sports (like ATP for people who look for Wimbledon results today)?
    - sudodude 4y ago
      I mean, any time someone wants information on current or recent events is your use case right there. If you exclude news entirely, you could maybe disregard recent websites but I imagine that's statistically a pretty large portion of search.
jimnotgym 4y ago
Yes, the article could better read, 'Why the world needs a range of non-profit search engines'
6510 4y ago
> Right now it is extremely difficult to build your own web crawler that would compete with Google. Nah, not a small task but you can break it down into well understood problems that have known solution. The hard part is ranking everything.
- kolinko 4y ago
  Are you speaking from theory or experience? If from experience - how did you get around multiple sites disallowing crawlers other than google or bing?
kordlessagain 4y ago
I built something with an API that uses Selenium to image a site. It works on a large percentage of the sites I feed it. I don't recursively call links found in the pages. I expect the user to give me the URLs to crawl and save. In order to "find" new content, I let the user specify where they want to search for things the engine hasn't "crawled" yet. So, a search for scooters to buy might end up searching Amazon directly, then lets the user "save" the site by passing the Amazon URL for the scooter they like to the system for imaging. I use GPT-3 or other ML models to do some of the heavy lifting for adding labels to the pages or documents the user uploads. This ends up being a "curated" list of documents important to the individual user, not an exhaustive crawl of all things which are important to all users.
gitgud 4y ago
> Perhaps a better approach would be building an open source www index... Wouldn't Google just use this too? Which would give Google in greater dominance over alternatives...

nix23 4y ago

> Perhaps a better approach would be building an open source www index or even a full current cache - as an enabler for people to build their own search engines? That's a excellent idea! In the spirit of open-data, and people can do with it what they want.

artificial 4y ago
I think this is a great idea. How does this work with copyright? Search engines seem to be able to download a reproduce content from scraped pages (and wrap it in ads, and derive content from it) this is called “indexing” when they do it but scraping when everyone else does it.
- bosie 4y ago
  > and wrap it in ads, and derive content from it i am probably missing something but can you give an example where this happens?
  - Flashtoo 4y ago
    E.g. on Google if you search for "how to tie a tie", a little info box may pop up with step by step instructions. This content is taken from some website, but that website gets no page hits or ad revenue. Instead, Google gets to serve ads on the search engine results page. (I don't know if this happens for this specific example, but Google does this for some searches)
    - jefftk 4y ago
      > that website gets no page hits Part of why sites participate in the infobox program is that in practice you do get quite a lot of hits from it: many people click through to see the answer in context.
    - bosie 4y ago
      Ok, I just tried but don't see that info box but that is exactly what i was asking for. Thank you very much, did not think of that.
  - zo1 4y ago
    I think they're referring to how Google "extracts" answers from your website and shows it on the search results page. Effectively meaning that the user doesn't even need to go to your site to get the answer, because Google extracted it and gave it to them directly.
    - thfuran 4y ago
      It seems to me that what they usually extract is some junk only vaguely related to the query and often cut apart and reassembled in a way that's just wrong.

daoudc 4y ago

The data is available! Not currently documented but check out the code for the API.

onionisafruit 4y ago
I'm curious if you think a co-op would be feasible between Mwmbl and other like-minded crawlers who are interested in taking a divide-and-conquer approach to crawling the web.

deleted 4y ago

[deleted]

closedloop129 4y ago

Why is https://commoncrawl.org/ https://commoncrawl.org/ not enough?

dewey 4y ago
Isn't that one more data sets for ML other research purposes instead of a highly up to date search index (For example with news from a few minutes ago).
- jka 4y ago
  Roughly speaking, yep - Common Crawl provides a sizable chunk of web data (420 TiB uncompressed, over 3 billion unique URLs, as of May 2022; historic statistics here[1]), and is updated on monthly basis. Not near-real-time, true, albeit relatively fresh. A question to ask could be: how often do users care about information from a few minutes ago, compared to information that has been available for a longer duration of time? [1] - https://commoncrawl.github.io/cc-crawl-statistics/ https://commoncrawl.github.io/cc-crawl-statistics/
  - flexie 4y ago
    Isn't that more a question of adding to the mix frequent scraping of - a few thousand news-sites (like nyt.com, bbc.co.uk), - a few thousand very popular blogs (based on what influencers people search for), - a handful of social media sites (e.g. Twitter), - a few hundred databases in areas like weather, airlines, sports (like ATP for people who look for Wimbledon results today)?
  - sudodude 4y ago
    I mean, any time someone wants information on current or recent events is your use case right there. If you exclude news entirely, you could maybe disregard recent websites but I imagine that's statistically a pretty large portion of search.

jimnotgym 4y ago

Yes, the article could better read, 'Why the world needs a range of non-profit search engines'

6510 4y ago

> Right now it is extremely difficult to build your own web crawler that would compete with Google. Nah, not a small task but you can break it down into well understood problems that have known solution. The hard part is ranking everything.

kolinko 4y ago
Are you speaking from theory or experience? If from experience - how did you get around multiple sites disallowing crawlers other than google or bing?

kordlessagain 4y ago

I built something with an API that uses Selenium to image a site. It works on a large percentage of the sites I feed it. I don't recursively call links found in the pages. I expect the user to give me the URLs to crawl and save. In order to "find" new content, I let the user specify where they want to search for things the engine hasn't "crawled" yet. So, a search for scooters to buy might end up searching Amazon directly, then lets the user "save" the site by passing the Amazon URL for the scooter they like to the system for imaging. I use GPT-3 or other ML models to do some of the heavy lifting for adding labels to the pages or documents the user uploads. This ends up being a "curated" list of documents important to the individual user, not an exhaustive crawl of all things which are important to all users.

gitgud 4y ago

> Perhaps a better approach would be building an open source www index... Wouldn't Google just use this too? Which would give Google in greater dominance over alternatives...