4 ms·

How would you judge that they had actually done the work? The output needs to be verifiable.

_xnmw 5y ago
There would have to be some aspect of centralized moderation, I suppose. This is beyond my knowledge: Is there a way to accept output only from signed binaries, so that we assume if X cycles of work were performed by a signed binary, then it is legitimate output?
- born-jre 5y ago
  no not really may be some theoretical way using ZK-snarks or encrypted enclaves (intel sgx) but not practical. also probably does not work cz oracle/enclave also needs input(raw crawl data/ network http bytes) which has to be trusted. one way could be project could make unbreakable mining/crawling chip/box/os/anticheat_layer_with_vm and supply to everyone which has different levels of breakable-lity and complexity to build. One way is we send crawl_task to different to N random nodes and accept one that most similar? another way could be build messy network to solve messy problem. What we do is build reputation bashed graph network and you accept index from nodes you trust. so people will start un following misbehaving nodes. there is not universalroot view of network instead its dynamic and different from prospective of each node. or it could have one root view if we store reputation data in bchain and with some type quadratic voting to modify the chain. ? yeah Bitcoin showed us way to build mathematically secure system without any trusted party but it could do that cz problem it was solving is mathematically provable. Problem like collecting indexing crawl data you have to trust somebody.
b3kart 5y ago
Well they are in a sense: you can just do the task yourself. It’s expensive of course, so you can use methods applied to human labelling for ML: _periodically_ injecting tasks with known results and checking how trustworthy the party is, vending the task to multiple parties and aggregating results, blocking parties that make many “mistakes”, etc.
hericium 5y ago
Depending on who would be providing URIs. If the miner, they would be able to deliver any crap so content deliveries would have to be judged in some way and awarded differently. If the pool provides addresses to crawl, the miner could be given crafted/dedicated URIs from time to time and lack of delivery of Proof Of Crawl could result in a penalty chosen in a way rendering "cheating" unprofitable. But then fresh URIs have to come from somewhere.
deleted 5y ago
[deleted]
GistNoesis 5y ago
You build a system based on trust but verify. If the output is the result of a known deterministic program on a known input, anyone can verify it. So they just have to sign their work. If later someone find that they have lied and provided a false result, they lose reputation/stacked coins. The second associated problem is how would one prevent them from appropriating the work of others that would just re-sign it. One way would be to allow the worker to introduce a few voluntary errors but have a secret joker that allow him to pass the challenge of a failed verification. One alternative way is based on data malleability. The worker pick a secret one way function and compute is F( data + secretFunction(data,epsilon) ) ~ F(data) and publish the values of the secretFunction(data,epsilon) but not the secretFunction. Only someone with knowledge of the secretFunction can make a claim on the work done. If there is a challenge only the real worker will be able to publish the secret of the secretFunction (Or use some zero knowledge proof to convince you they know it).
- charcircuit 5y ago
  >If later someone find that they have lied and provided a false result, they lose reputation/stacked coins. A web page isn't an immutable piece of text. It can change on every visit and it can sometimes returns errors.
  - GistNoesis 5y ago
    That's why you don't index the webpage but a snapshot of it. For example you index the commoncrawl archives, or some content addressable storage like ipfs or torrent file.
    - charcircuit 5y ago
      What's the point of crawling the common crawl archive? It's pointless. You can simply download it.
      - GistNoesis 5y ago
        I think crawling and indexing should be treated differently. Indexing is about extracting value from the data, while crawling is about gathering data. Once a reference snapshot has been crawled, the indexing task is more easily verifiable. The crawling task is harder to verify, because external website could lie to the crawler. So the sensible thing to do is have multiple people crawl the same site and compare their results. Every crawler will publish its snapshots (which may contain some errors or not), and then that's the job of the indexer to combine multiple snapshots of various crawler and filter the errors out and do the de-duplication. The crawling task is less necessary now than it was a few years ago, because there is already plenty of available data. Also most of the valuable data is locked in walled garden, and companies like Cloudflare make the crawling difficult for the rest of the fat tail. So it's better to only have data submitted to you, and outsource the crawling.
zomglings 5y ago
One idea I have been kicking around is the idea of federating the indices, not the crawling. If every contributor maintained their own index, then you could reward contributors based on how many hits their index generated. This would open up the possibility of people maintaining indices for specialized topics that they were experts in, and give the federated search engine a shot at taking on Google.
- bspammer 5y ago
  You don’t want to reward people for quantity, but quality. The cost of creating a new webpage is effectively zero, so if you attach an incentive for creating them you are doomed.
  - zomglings 5y ago
    You and I are talking about completely different types of people. For most people, the cost of creating and maintaining a website is high. This is why products like Wix and Squarespace exist (and are not cheap). I am thinking a simple dashboard where anyone could go and curate a list of content they find useful. They could share this with the world. The interface should be so simple that my parents could use it - and they aren't going to be putting up websites anytime soon.
- DarylZero 5y ago
  This is what Yahoo was doing before Google took over. (But of course it wasn't a volunteer public benefit effort like you describe.)

There would have to be some aspect of centralized moderation, I suppose. This is beyond my knowledge: Is there a way to accept output only from signed binaries, so that we assume if X cycles of work were performed by a signed binary, then it is legitimate output?

born-jre 5y ago
no not really may be some theoretical way using ZK-snarks or encrypted enclaves (intel sgx) but not practical. also probably does not work cz oracle/enclave also needs input(raw crawl data/ network http bytes) which has to be trusted. one way could be project could make unbreakable mining/crawling chip/box/os/anticheat_layer_with_vm and supply to everyone which has different levels of breakable-lity and complexity to build. One way is we send crawl_task to different to N random nodes and accept one that most similar? another way could be build messy network to solve messy problem. What we do is build reputation bashed graph network and you accept index from nodes you trust. so people will start un following misbehaving nodes. there is not universalroot view of network instead its dynamic and different from prospective of each node. or it could have one root view if we store reputation data in bchain and with some type quadratic voting to modify the chain. ? yeah Bitcoin showed us way to build mathematically secure system without any trusted party but it could do that cz problem it was solving is mathematically provable. Problem like collecting indexing crawl data you have to trust somebody.

b3kart 5y ago

Well they are in a sense: you can just do the task yourself. It’s expensive of course, so you can use methods applied to human labelling for ML: _periodically_ injecting tasks with known results and checking how trustworthy the party is, vending the task to multiple parties and aggregating results, blocking parties that make many “mistakes”, etc.

hericium 5y ago

Depending on who would be providing URIs. If the miner, they would be able to deliver any crap so content deliveries would have to be judged in some way and awarded differently. If the pool provides addresses to crawl, the miner could be given crafted/dedicated URIs from time to time and lack of delivery of Proof Of Crawl could result in a penalty chosen in a way rendering "cheating" unprofitable. But then fresh URIs have to come from somewhere.

deleted 5y ago

[deleted]

GistNoesis 5y ago

You build a system based on trust but verify. If the output is the result of a known deterministic program on a known input, anyone can verify it. So they just have to sign their work. If later someone find that they have lied and provided a false result, they lose reputation/stacked coins. The second associated problem is how would one prevent them from appropriating the work of others that would just re-sign it. One way would be to allow the worker to introduce a few voluntary errors but have a secret joker that allow him to pass the challenge of a failed verification. One alternative way is based on data malleability. The worker pick a secret one way function and compute is F( data + secretFunction(data,epsilon) ) ~ F(data) and publish the values of the secretFunction(data,epsilon) but not the secretFunction. Only someone with knowledge of the secretFunction can make a claim on the work done. If there is a challenge only the real worker will be able to publish the secret of the secretFunction (Or use some zero knowledge proof to convince you they know it).

charcircuit 5y ago
>If later someone find that they have lied and provided a false result, they lose reputation/stacked coins. A web page isn't an immutable piece of text. It can change on every visit and it can sometimes returns errors.
- GistNoesis 5y ago
  That's why you don't index the webpage but a snapshot of it. For example you index the commoncrawl archives, or some content addressable storage like ipfs or torrent file.
  - charcircuit 5y ago
    What's the point of crawling the common crawl archive? It's pointless. You can simply download it.
    - GistNoesis 5y ago
      I think crawling and indexing should be treated differently. Indexing is about extracting value from the data, while crawling is about gathering data. Once a reference snapshot has been crawled, the indexing task is more easily verifiable. The crawling task is harder to verify, because external website could lie to the crawler. So the sensible thing to do is have multiple people crawl the same site and compare their results. Every crawler will publish its snapshots (which may contain some errors or not), and then that's the job of the indexer to combine multiple snapshots of various crawler and filter the errors out and do the de-duplication. The crawling task is less necessary now than it was a few years ago, because there is already plenty of available data. Also most of the valuable data is locked in walled garden, and companies like Cloudflare make the crawling difficult for the rest of the fat tail. So it's better to only have data submitted to you, and outsource the crawling.

zomglings 5y ago

One idea I have been kicking around is the idea of federating the indices, not the crawling. If every contributor maintained their own index, then you could reward contributors based on how many hits their index generated. This would open up the possibility of people maintaining indices for specialized topics that they were experts in, and give the federated search engine a shot at taking on Google.

bspammer 5y ago
You don’t want to reward people for quantity, but quality. The cost of creating a new webpage is effectively zero, so if you attach an incentive for creating them you are doomed.
- zomglings 5y ago
  You and I are talking about completely different types of people. For most people, the cost of creating and maintaining a website is high. This is why products like Wix and Squarespace exist (and are not cheap). I am thinking a simple dashboard where anyone could go and curate a list of content they find useful. They could share this with the world. The interface should be so simple that my parents could use it - and they aren't going to be putting up websites anytime soon.
DarylZero 5y ago
This is what Yahoo was doing before Google took over. (But of course it wasn't a volunteer public benefit effort like you describe.)