6 ms·
This is a business model I've been thinking about: what if users earned credits for running a crawler on their machine? In other words, as much as I hate crypto
by _xnmw 5y ago
This is a business model I've been thinking about: what if users earned credits for running a crawler on their machine? In other words, as much as I hate crypto scams, a "tokenized" search engine where the "mining" power was put to good use, i.e crawling and indexing.
- thebeastie 5y agoHow would you judge that they had actually done the work? The output needs to be verifiable.
- _xnmw 5y agoThere would have to be some aspect of centralized moderation, I suppose. This is beyond my knowledge: Is there a way to accept output only from signed binaries, so that we assume if X cycles of work were performed by a signed binary, then it is legitimate output?
- born-jre 5y agono not really may be some theoretical way using ZK-snarks or encrypted enclaves (intel sgx) but not practical. also probably does not work cz oracle/enclave also needs input(raw crawl data/ network http bytes) which has to be trusted. one way could be project could make unbreakable mining/crawling chip/box/os/anticheat_layer_with_vm and supply to everyone which has different levels of breakable-lity and complexity to build. One way is we send crawl_task to different to N random nodes and accept one that most similar? another way could be build messy network to solve messy problem. What we do is build reputation bashed graph network and you accept index from nodes you trust. so people will start un following misbehaving nodes. there is not universalroot view of network instead its dynamic and different from prospective of each node. or it could have one root view if we store reputation data in bchain and with some type quadratic voting to modify the chain. ? yeah Bitcoin showed us way to build mathematically secure system without any trusted party but it could do that cz problem it was solving is mathematically provable. Problem like collecting indexing crawl data you have to trust somebody.
- b3kart 5y agoWell they are in a sense: you can just do the task yourself. It’s expensive of course, so you can use methods applied to human labelling for ML: _periodically_ injecting tasks with known results and checking how trustworthy the party is, vending the task to multiple parties and aggregating results, blocking parties that make many “mistakes”, etc.
- hericium 5y agoDepending on who would be providing URIs. If the miner, they would be able to deliver any crap so content deliveries would have to be judged in some way and awarded differently. If the pool provides addresses to crawl, the miner could be given crafted/dedicated URIs from time to time and lack of delivery of Proof Of Crawl could result in a penalty chosen in a way rendering "cheating" unprofitable. But then fresh URIs have to come from somewhere.
- deleted 5y ago[deleted]
- GistNoesis 5y agoYou build a system based on trust but verify. If the output is the result of a known deterministic program on a known input, anyone can verify it. So they just have to sign their work. If later someone find that they have lied and provided a false result, they lose reputation/stacked coins. The second associated problem is how would one prevent them from appropriating the work of others that would just re-sign it. One way would be to allow the worker to introduce a few voluntary errors but have a secret joker that allow him to pass the challenge of a failed verification. One alternative way is based on data malleability. The worker pick a secret one way function and compute is F( data + secretFunction(data,epsilon) ) ~ F(data) and publish the values of the secretFunction(data,epsilon) but not the secretFunction. Only someone with knowledge of the secretFunction can make a claim on the work done. If there is a challenge only the real worker will be able to publish the secret of the secretFunction (Or use some zero knowledge proof to convince you they know it).
- charcircuit 5y ago>If later someone find that they have lied and provided a false result, they lose reputation/stacked coins. A web page isn't an immutable piece of text. It can change on every visit and it can sometimes returns errors.
- GistNoesis 5y agoThat's why you don't index the webpage but a snapshot of it. For example you index the commoncrawl archives, or some content addressable storage like ipfs or torrent file.
- charcircuit 5y agoWhat's the point of crawling the common crawl archive? It's pointless. You can simply download it.
- GistNoesis 5y agoI think crawling and indexing should be treated differently. Indexing is about extracting value from the data, while crawling is about gathering data. Once a reference snapshot has been crawled, the indexing task is more easily verifiable. The crawling task is harder to verify, because external website could lie to the crawler. So the sensible thing to do is have multiple people crawl the same site and compare their results. Every crawler will publish its snapshots (which may contain some errors or not), and then that's the job of the indexer to combine multiple snapshots of various crawler and filter the errors out and do the de-duplication. The crawling task is less necessary now than it was a few years ago, because there is already plenty of available data. Also most of the valuable data is locked in walled garden, and companies like Cloudflare make the crawling difficult for the rest of the fat tail. So it's better to only have data submitted to you, and outsource the crawling.
- zomglings 5y agoOne idea I have been kicking around is the idea of federating the indices, not the crawling. If every contributor maintained their own index, then you could reward contributors based on how many hits their index generated. This would open up the possibility of people maintaining indices for specialized topics that they were experts in, and give the federated search engine a shot at taking on Google.
- bspammer 5y agoYou don’t want to reward people for quantity, but quality. The cost of creating a new webpage is effectively zero, so if you attach an incentive for creating them you are doomed.
- zomglings 5y agoYou and I are talking about completely different types of people. For most people, the cost of creating and maintaining a website is high. This is why products like Wix and Squarespace exist (and are not cheap). I am thinking a simple dashboard where anyone could go and curate a list of content they find useful. They could share this with the world. The interface should be so simple that my parents could use it - and they aren't going to be putting up websites anytime soon.
- DarylZero 5y agoThis is what Yahoo was doing before Google took over. (But of course it wasn't a volunteer public benefit effort like you describe.)
- g105b 5y agoI'm very intrigued by this concept.
- Piezoid 5y agoYaCy is decentralized, but without the credit system. Some tokens, like QBUX, have tried to develop decentralized hosting infrastructure. I also have been wondering how this would play out with some kind of decentralized indexes. The nodes could automatically cluster with other nodes of users sharing the same interests, using some notion of distances between query distributions. The caching and crawling tasks could then be distributed between neighbors.
- _xnmw 5y agoYaCy is too slow for mainstream use. I believe the indices still need to be centralized, only index-building and crawling can be distributed.
- marginalia_nu 5y agoA big part of the problem I see with decentralized search is that you basically need to traverse the index in orthogonal axes to assemble search results. First you need to search word-wise in order to get result candidates, then sort them rank-wise to get relevant results (this also hinges upon an agreed-upon ranking of domains). That's a damn hard nut to crack for a distributed system. Crawling is also not as resource consuming as you might think. Sure you can distribute it, but there isn't a huge benefit to this.
- thebeastie 5y agoActually I have an idea for you: i think you can use cryptography to prove that an SSL session really happened. So you could prove indexing of HTTPS sites.
- thebeastie 5y agoI think the way this works is having the code to execute an ssl session encoded in a zkSnark. One of the zkSnark based blockchains is doing it.
- detaro 5y agoYou can prove that a TLS session happened, but nothing about its contents, so you can't really prove indexing.