2 ms·
A newcomer can index, store, search and display data differently, but first they still have to get that data. Which means dealing with JavaScript, paywalls and
by romaniv 11y ago
A newcomer can index, store, search and display data differently, but first they still have to get that data. Which means dealing with JavaScript, paywalls and other fun stuff like that.
Besides, regardless of what you do, you would need to have tons of storage, bandwidth, CPU power and a high-availability infrastructure.
- grey-area 11y agoThe vast majority of the data on the internet is dross or spam, if someone finds a way to index only quality content, the crawling burden could be radically smaller.