4 ms·

I don't think that this is the answer to "only google can crawl the web". This is a huge archive suitable for making a web search engine maybe. What if you wan

I don't think that this is the answer to "only google can crawl the web". This is a huge archive suitable for making a web search engine maybe. What if you want to make a simple link previewer? An abstract crawler for scientific articles? Most websites are behind cloudflare which will block/captcha you, but happily whitelist only google & major social sites. Tha answer is measures that bring the web back to basics, not this over-SEOed bot infested ecosystem. FANGS succeeded in sucking out all the information of the web, but they suck at creating protocols that are interoperable (even twitter now needs its own tags!). Incidentally, maybe the next search engine should use a push-system , where websites will ping to it whenever they have updates. If the engine has unique features, it might be actually seen as a measure to reduce the loads from bots.

gillesjacobs 6y ago
This is mainly used for language modeling research. A filtered CC was used in GPT3 and I have personally used data from CC for NLP projects.
- visarga 6y ago
  I recently did a pass through CC it took me 5 days, there were 80,000 1GB warc files. I had to just filter the information I wanted and dump the rest because I can't possibly host that much data on my machines. It's great data but very noisy, like the web is. Lots of repetition and useless junk mixed with useful data.
shadowgovt 6y ago
I don't think we can bring the web back to basics in the sense you're envisioning without kicking most users off of it. Cloudflare's protection is to guard against traffic spikes and automated malicious attacks; Google and social sites are allow-listed because they're trustable entities with well-defined online presences and a vested interest (generally) in not breaking sites. "How do we do away with the need to put up an anti-automated-traffic screen plus whitelist" is in the same category of problem as "How do we change the modern web to address automated malicious attacks?"
- neura 6y ago
  Another way to look at this is if CDNs don't allow Google, nobody is going to want to use them. Their content doesn't get indexed and anybody doing a search is going to get directed to someone else that doesn't put their content behind a CDN with that level of protection. That or someone like google will just solve the problem themselves and be both the CDN and the indexer, bringing them one step closer to complete ownership of finding anything on the web.
- d110af5ccf 6y ago
  > Cloudflare's protection is to guard against traffic spikes and automated malicious attacks This simply isn't true. By my reading, Cloudflare implies any automated traffic that doesn't openly advertise itself as such is bad. They certainly seem to be happy to assist you in blocking it regardless of whether it was actually malicious or actually causing traffic problems. (https://blog.cloudflare.com/super-bot-fight-mode/ https://blog.cloudflare.com/super-bot-fight-mode/)
  - shadowgovt 6y ago
    I'm not sure I understand the distinction you're making. Automated traffic that isn't flagging itself as automated is deceptive, and that's a negative signal for maliciousness right out of the starting gate.
    - ipaddr 6y ago
      Shouldn't automated traffic be allowed privacy? To not do so means we have to take away privacy from humans in order to verify they are not automated. What do we gain for this? Why shouldn't some programmable bot be allowed to see my content but any human no matter their intent are allowed by default?
      - shadowgovt 6y ago
        > What do we gain for this? More reliable services for those willing to de-pssudonymize and declare themselves in an auditable fashion.
th0ma5 6y ago
Actually I was able to search URLs with very little ram across the entire collection as they have a series of indexes you can download. In theory someone could do something similar with terms, or you could first use URLs to filter the text size you download into Elastic or Solr and do your own custom search that way. The indexes are really neat though, I highly recommend playing with them.
- thunderbong 6y ago
  Can you suggest where I can get them from?
  - th0ma5 6y ago
    https://index.commoncrawl.org/ https://index.commoncrawl.org/
breischl 6y ago
>maybe the next search engine should use a push-system So setting up a new search engine that way would require going to every site and convincing them to notify you of changes. Wouldn't that be even more limited than the current Cloudflare whitelist system? At least there's some chance you can get around the whitelist system.
- neura 6y ago
  This... I couldn't get past the irony of the comment. Basically, the problem is that sites only let google and friends index them. The solution? Site should only send index data to google and friends. I mean, I get it, then sites can send to any number of indexers, but let's be honest, like you say, any new search engine has to get sites to push data out to them. That's just not. going. to. happen.

gillesjacobs 6y ago

This is mainly used for language modeling research. A filtered CC was used in GPT3 and I have personally used data from CC for NLP projects.

visarga 6y ago
I recently did a pass through CC it took me 5 days, there were 80,000 1GB warc files. I had to just filter the information I wanted and dump the rest because I can't possibly host that much data on my machines. It's great data but very noisy, like the web is. Lots of repetition and useless junk mixed with useful data.

shadowgovt 6y ago

I don't think we can bring the web back to basics in the sense you're envisioning without kicking most users off of it. Cloudflare's protection is to guard against traffic spikes and automated malicious attacks; Google and social sites are allow-listed because they're trustable entities with well-defined online presences and a vested interest (generally) in not breaking sites. "How do we do away with the need to put up an anti-automated-traffic screen plus whitelist" is in the same category of problem as "How do we change the modern web to address automated malicious attacks?"

neura 6y ago
Another way to look at this is if CDNs don't allow Google, nobody is going to want to use them. Their content doesn't get indexed and anybody doing a search is going to get directed to someone else that doesn't put their content behind a CDN with that level of protection. That or someone like google will just solve the problem themselves and be both the CDN and the indexer, bringing them one step closer to complete ownership of finding anything on the web.
d110af5ccf 6y ago
> Cloudflare's protection is to guard against traffic spikes and automated malicious attacks This simply isn't true. By my reading, Cloudflare implies any automated traffic that doesn't openly advertise itself as such is bad. They certainly seem to be happy to assist you in blocking it regardless of whether it was actually malicious or actually causing traffic problems. (https://blog.cloudflare.com/super-bot-fight-mode/ https://blog.cloudflare.com/super-bot-fight-mode/)
- shadowgovt 6y ago
  I'm not sure I understand the distinction you're making. Automated traffic that isn't flagging itself as automated is deceptive, and that's a negative signal for maliciousness right out of the starting gate.
  - ipaddr 6y ago
    Shouldn't automated traffic be allowed privacy? To not do so means we have to take away privacy from humans in order to verify they are not automated. What do we gain for this? Why shouldn't some programmable bot be allowed to see my content but any human no matter their intent are allowed by default?
    - shadowgovt 6y ago
      > What do we gain for this? More reliable services for those willing to de-pssudonymize and declare themselves in an auditable fashion.

th0ma5 6y ago

Actually I was able to search URLs with very little ram across the entire collection as they have a series of indexes you can download. In theory someone could do something similar with terms, or you could first use URLs to filter the text size you download into Elastic or Solr and do your own custom search that way. The indexes are really neat though, I highly recommend playing with them.

thunderbong 6y ago
Can you suggest where I can get them from?
- th0ma5 6y ago
  https://index.commoncrawl.org/ https://index.commoncrawl.org/

breischl 6y ago

>maybe the next search engine should use a push-system So setting up a new search engine that way would require going to every site and convincing them to notify you of changes. Wouldn't that be even more limited than the current Cloudflare whitelist system? At least there's some chance you can get around the whitelist system.

neura 6y ago
This... I couldn't get past the irony of the comment. Basically, the problem is that sites only let google and friends index them. The solution? Site should only send index data to google and friends. I mean, I get it, then sites can send to any number of indexers, but let's be honest, like you say, any new search engine has to get sites to push data out to them. That's just not. going. to. happen.