7 ms·

Long time ago I worked for a startup called Wowd which built distributed search engine. It was acquihired by Facebook. On of the biggest issues was how to enti

by ssijak 3y ago

Long time ago I worked for a startup called Wowd which built distributed search engine. It was acquihired by Facebook. On of the biggest issues was how to entice people to download and run the client/node. I half wondered afterwards if slapping some crypto on top of it which would be mined by running the node and providing resources would help. My gut says easy yes, but my mind grimace at the abomination.

zoklet-enjoyer 3y ago
We have proof of stake now. The nodes could be run by the chain validators and they get a cut of the staking rewards. Look up how proof of stake works on the Cosmos chain. You could totally do this and I bet it would take off, at least in that section of the Internet that's into Cosmos/Tendermint chains. I'd use it
- ssijak 3y ago
  I was definitely thinking of some kind of proof of stake, not proof of work.
- zoklet-enjoyer 3y ago
  Hahaha one downvote. I love to see it
worksonmine 3y ago
> but my mind grimace at the abomination Why would that be an abomination? It's a perfect use-case. Like you noticed people need incentives to volunteer their hardware. If you hate crypto because it's crypto you can just use fiat instead.
- komali2 3y ago
  > Like you noticed people need incentives to volunteer their hardware. I wonder if this is because "volunteer your hardware" projects sometimes involve someone making else money, and if someone else is making money but not you, why should you donate your hardware? For the truly libre "hardware donation" projects, they seem to be doing ok without financial incentivization. What immediately comes to mind is the petabytes of data flying around on peer to peer systems through torrenting. I know people that spend thousands of dollars a year on upkeep and upgrades for what are essentially super seedbox homelabs (I'm one of them too :P ) There's also communities like soulseek where people keep TBs of music up, often seeking out rare tracks to make available to the community for free. There's folding@home and seti@home, and I'm sure other similar projects I haven't heard of, where people donate cycles just for the common good. folding@home is a great example because we can directly compare the people that are "incentivized" to participate with bananocoin, a cryptocurrency rewarded based on work cycles in folding@home. You can see all bananocoin miners here under the banano.cc team: https://stats.foldingathome.org/ https://stats.foldingathome.org/ That team is in first place for work completed, however are only just surpassing the linus tech tips team, and not to mention compared to a bunch of other teams (and private "donors") they're a very small % of work completed for folding@home So therefore I disagree that people "need" incentives, there just needs to be no, erm, disincentives, if that's a word.
  - shinryuu 3y ago
    > I know people that spend thousands of dollars a year on upkeep and upgrades for what are essentially super seedbox homelabs And then you end with "there just needs to be no disincentives". If anything spending thousands of dollars a year on upkeep should be a disincentive for most people. You are not most people though, since you do it voluntarily.
    - komali2 3y ago
      I'm a maniac though. I used to run my stack just fine off a raspberri pi with a USB harddrive plugged in. Actually, before that, I used to run it off an old macbook. Do we need it to be where everyone hosts a node? I just had this conversation with a friend yesterday actually. We were in disagreement about the accessibility of self hosting and federation. He was of the opinion that we should push LLMs to where anyone can type "I want to host a video hosting platform" and chatgpt.exe will find and install jellyfin on their computer and set up a cloudflare tunnel, or whatever. I'm more of the opinion that we should increase the quality of documentation until the one person just weird and nerdy enough out of a group of 20 will be able to deploy things on leftover hardware, and share with their friends. What do you think?
      - shinryuu 3y ago
        In terms of accessibility I don't think it would be bad per se if chatgpt.exe would be able to help you with that. Though both of us know that there is maintenance involved and once something catch fire (which will happen at some point), you are kind of helpless. Something like pikapods.com certainly helps with accessibility, even if it isn't self-hosting per se. But all of that doesn't have little to do with incentives or disincentives. Even with very high accessibility there are disincentives to self-host. It will cost time and money in some way. For some people the intrinsic motivation will override those disincentives. But I think for the majority of people there will still not be enough motivation to do it. There are more important things to do for them.
        komali2 3y ago
- bawolff 3y ago
  I mean, how do you verify nodes are being honest and not just sending fake data for the free crypto (like what happened with seti@home and there wasn't even money involved) Not to mention, where is the value of this coin going to come from? Will people pay to use this search engine? That seems unlikely. It doesn't sound like the perfect use case to me.
  - px43 3y ago
    By ignoring cryptocurrencies, you've missed out on over 10 years of progress in this space. We have things like zero knowledge notaries and data availability sampling proofs. Actively Validated Services are also a thing. Service providers stake some asset, and interested parties can challenge them at certain intervals to ensure that they are properly performing their duties. Through the magic of Merkel Trees, and soon Verkel Trees (basically Merkel tress, but using vector commitments for super fast proofs) challengers can demand that that service providers generate a proof that some data they hold matches some criteria. The nice thing about it is that because it's a zero knowledge proof, the challenger doesn't even need to know what that data is, and what they get back is a succinct proof that they can check very quickly, basically like checking an md5 some for execution correctness. It's cool shit, you should really look into it.
  - worksonmine 3y ago
    That's exactly why blockchain is a good choice. You verify that whatever X sends matches what Y and Z would send before any reward is received. Based on the shared index every query should return the same results, kids stuff really. The monetization is a nut to crack yes, but Kagi works as a paywalled search engine. Otherwise just serve ads like all the rest already do? Tried and proven model, and in this solution they could be very transparent as there's no corporation behind trying to dupe the users for clicks to maximize profits. I even see the possibility for a hybrid model, don't like ads? Pay for the compute with your own coins. The value comes from the network, trust and use-case. It doesn't have to be a new coin.
  - numpad0 3y ago
    Agreed; feels to me that people here is underestimating malice on the Internet. Simple crypto-based search credit system will be overtaken with fake queries and fake data. I'm not entirely confident that crypto-like reward mechanisms for distributed search is fundamentally flawed and unusable, but both the problem and solution needs to be refined a bit more.
lifty 3y ago
Not sure why it would be an abomination. This is the exact use case which is a fit for cryptocurrency networks.
- rakoo 3y ago
  You have to look beyond the surface. Cryptocurrencies work specifically to address a system where no node can trust any other node. If I cannot trust any other node, why would I fetch anyone else's index, or ask them for the results of a query, or even talk to them ? Unless there can be a way to trivially verify what others tell you, crypto currencies are a dead end
  - mhluongo 3y ago
    You have that issue without cryptocurrencies as well, you'd just be relying on the kindness of users rather than crypto incentives. You always need a way to hold nodes accountable in a system like this, or it'll be rife with manipulation — because there's already a strong, innate incentive to manipulate results. Today, we call that industry "SEO".
    - rakoo 3y ago
      What you don't understand is that "I don't trust others" is not a terminal statee I'd rather build trust again, create human connections, or rather, put them in front because there are always connections; nothing works if you trust noone. Building a societal system where you know you can rely on your peers, you build together, is a more joyful, more resilient, more ecological and also more realistic way of building a thriving society than distrust-by-default that cryptocurrencies live for.
      - idiotsecant 3y ago
        Your current fiat currency is not based on love and trust. Its Proof-of-World-Hegemony which puts crypto based consensus mechanisms to shame in terms of how not based on love and trust it is.
6510 3y ago
After that the issue becomes ranking. Should say became since LLM's could both rank pages and generate them on "demand" to fit the query. YaCy has so many buttons I'm not even sure if it lacks it but playing around with it it is very cool to crawl large amounts of pages and serve requests until you want to do other things with the computer and the background process is to bloated. Something like a turtle mode like torrent clients have would be useful. Long ago there was a Chinese p2p client with a rootkit that would seed at 1 kb. I haven't used it but was told it worked remarkably well.
duckman1 3y ago
[dead]
mdaniel 3y ago
Nothing new under the sun, as they say: https://www.presearch.io/engine https://www.presearch.io/engine and just as you said I was unwilling to run a closed-source node binary
colinsane 3y ago
if the situation is really "nobody will run this software unless i pay them to", then you're doomed regardless. there's nothing wrong with the classic route: package your software for the stores/distros you're familiar with, make your software as easy to package as humanly possible for anyone else who'll come around, document the hell out of it, submit it to the handful of top-level news feeds from which it'll percolate, and then wait. maybe you don't like waiting?

by ssijak 3y ago

zoklet-enjoyer 3y ago

We have proof of stake now. The nodes could be run by the chain validators and they get a cut of the staking rewards. Look up how proof of stake works on the Cosmos chain. You could totally do this and I bet it would take off, at least in that section of the Internet that's into Cosmos/Tendermint chains. I'd use it

ssijak 3y ago
I was definitely thinking of some kind of proof of stake, not proof of work.
zoklet-enjoyer 3y ago
Hahaha one downvote. I love to see it

worksonmine 3y ago

> but my mind grimace at the abomination Why would that be an abomination? It's a perfect use-case. Like you noticed people need incentives to volunteer their hardware. If you hate crypto because it's crypto you can just use fiat instead.

komali2 3y ago
> Like you noticed people need incentives to volunteer their hardware. I wonder if this is because "volunteer your hardware" projects sometimes involve someone making else money, and if someone else is making money but not you, why should you donate your hardware? For the truly libre "hardware donation" projects, they seem to be doing ok without financial incentivization. What immediately comes to mind is the petabytes of data flying around on peer to peer systems through torrenting. I know people that spend thousands of dollars a year on upkeep and upgrades for what are essentially super seedbox homelabs (I'm one of them too :P ) There's also communities like soulseek where people keep TBs of music up, often seeking out rare tracks to make available to the community for free. There's folding@home and seti@home, and I'm sure other similar projects I haven't heard of, where people donate cycles just for the common good. folding@home is a great example because we can directly compare the people that are "incentivized" to participate with bananocoin, a cryptocurrency rewarded based on work cycles in folding@home. You can see all bananocoin miners here under the banano.cc team: https://stats.foldingathome.org/ https://stats.foldingathome.org/ That team is in first place for work completed, however are only just surpassing the linus tech tips team, and not to mention compared to a bunch of other teams (and private "donors") they're a very small % of work completed for folding@home So therefore I disagree that people "need" incentives, there just needs to be no, erm, disincentives, if that's a word.
- shinryuu 3y ago
  > I know people that spend thousands of dollars a year on upkeep and upgrades for what are essentially super seedbox homelabs And then you end with "there just needs to be no disincentives". If anything spending thousands of dollars a year on upkeep should be a disincentive for most people. You are not most people though, since you do it voluntarily.
  - komali2 3y ago
    I'm a maniac though. I used to run my stack just fine off a raspberri pi with a USB harddrive plugged in. Actually, before that, I used to run it off an old macbook. Do we need it to be where everyone hosts a node? I just had this conversation with a friend yesterday actually. We were in disagreement about the accessibility of self hosting and federation. He was of the opinion that we should push LLMs to where anyone can type "I want to host a video hosting platform" and chatgpt.exe will find and install jellyfin on their computer and set up a cloudflare tunnel, or whatever. I'm more of the opinion that we should increase the quality of documentation until the one person just weird and nerdy enough out of a group of 20 will be able to deploy things on leftover hardware, and share with their friends. What do you think?
    - shinryuu 3y ago
      In terms of accessibility I don't think it would be bad per se if chatgpt.exe would be able to help you with that. Though both of us know that there is maintenance involved and once something catch fire (which will happen at some point), you are kind of helpless. Something like pikapods.com certainly helps with accessibility, even if it isn't self-hosting per se. But all of that doesn't have little to do with incentives or disincentives. Even with very high accessibility there are disincentives to self-host. It will cost time and money in some way. For some people the intrinsic motivation will override those disincentives. But I think for the majority of people there will still not be enough motivation to do it. There are more important things to do for them.
      - komali2 3y ago
bawolff 3y ago
I mean, how do you verify nodes are being honest and not just sending fake data for the free crypto (like what happened with seti@home and there wasn't even money involved) Not to mention, where is the value of this coin going to come from? Will people pay to use this search engine? That seems unlikely. It doesn't sound like the perfect use case to me.
- px43 3y ago
  By ignoring cryptocurrencies, you've missed out on over 10 years of progress in this space. We have things like zero knowledge notaries and data availability sampling proofs. Actively Validated Services are also a thing. Service providers stake some asset, and interested parties can challenge them at certain intervals to ensure that they are properly performing their duties. Through the magic of Merkel Trees, and soon Verkel Trees (basically Merkel tress, but using vector commitments for super fast proofs) challengers can demand that that service providers generate a proof that some data they hold matches some criteria. The nice thing about it is that because it's a zero knowledge proof, the challenger doesn't even need to know what that data is, and what they get back is a succinct proof that they can check very quickly, basically like checking an md5 some for execution correctness. It's cool shit, you should really look into it.
- worksonmine 3y ago
  That's exactly why blockchain is a good choice. You verify that whatever X sends matches what Y and Z would send before any reward is received. Based on the shared index every query should return the same results, kids stuff really. The monetization is a nut to crack yes, but Kagi works as a paywalled search engine. Otherwise just serve ads like all the rest already do? Tried and proven model, and in this solution they could be very transparent as there's no corporation behind trying to dupe the users for clicks to maximize profits. I even see the possibility for a hybrid model, don't like ads? Pay for the compute with your own coins. The value comes from the network, trust and use-case. It doesn't have to be a new coin.
- numpad0 3y ago
  Agreed; feels to me that people here is underestimating malice on the Internet. Simple crypto-based search credit system will be overtaken with fake queries and fake data. I'm not entirely confident that crypto-like reward mechanisms for distributed search is fundamentally flawed and unusable, but both the problem and solution needs to be refined a bit more.

lifty 3y ago

Not sure why it would be an abomination. This is the exact use case which is a fit for cryptocurrency networks.

rakoo 3y ago
You have to look beyond the surface. Cryptocurrencies work specifically to address a system where no node can trust any other node. If I cannot trust any other node, why would I fetch anyone else's index, or ask them for the results of a query, or even talk to them ? Unless there can be a way to trivially verify what others tell you, crypto currencies are a dead end
- mhluongo 3y ago
  You have that issue without cryptocurrencies as well, you'd just be relying on the kindness of users rather than crypto incentives. You always need a way to hold nodes accountable in a system like this, or it'll be rife with manipulation — because there's already a strong, innate incentive to manipulate results. Today, we call that industry "SEO".
  - rakoo 3y ago
    What you don't understand is that "I don't trust others" is not a terminal statee I'd rather build trust again, create human connections, or rather, put them in front because there are always connections; nothing works if you trust noone. Building a societal system where you know you can rely on your peers, you build together, is a more joyful, more resilient, more ecological and also more realistic way of building a thriving society than distrust-by-default that cryptocurrencies live for.
    - idiotsecant 3y ago
      Your current fiat currency is not based on love and trust. Its Proof-of-World-Hegemony which puts crypto based consensus mechanisms to shame in terms of how not based on love and trust it is.

6510 3y ago

After that the issue becomes ranking. Should say became since LLM's could both rank pages and generate them on "demand" to fit the query. YaCy has so many buttons I'm not even sure if it lacks it but playing around with it it is very cool to crawl large amounts of pages and serve requests until you want to do other things with the computer and the background process is to bloated. Something like a turtle mode like torrent clients have would be useful. Long ago there was a Chinese p2p client with a rootkit that would seed at 1 kb. I haven't used it but was told it worked remarkably well.

duckman1 3y ago

[dead]

mdaniel 3y ago

Nothing new under the sun, as they say: https://www.presearch.io/engine https://www.presearch.io/engine and just as you said I was unwilling to run a closed-source node binary

colinsane 3y ago

if the situation is really "nobody will run this software unless i pay them to", then you're doomed regardless. there's nothing wrong with the classic route: package your software for the stores/distros you're familiar with, make your software as easy to package as humanly possible for anyone else who'll come around, document the hell out of it, submit it to the handful of top-level news feeds from which it'll percolate, and then wait. maybe you don't like waiting?