6 ms·

This is my current favourite use case (from the blog https://blog.cloudflare.com/e2e-integrity/ https://blog.cloudflare.com/e2e-integrity/) I wanted to provide

This is my current favourite use case (from the blog https://blog.cloudflare.com/e2e-integrity/ https://blog.cloudflare.com/e2e-integrity/) I wanted to provide an example of the kinds of secure, performant applications that are possible with IPFS, and this made building a search engine seem like a prime candidate. Rather than steal Protocol Labs' idea of 'Wikipedia on IPFS', we decided to take the Kiwix archives of all the different StackExchange websites and build a distributed search engine on top of that. You can play with the finished product here: https://ipfs-sec.stackexchange.cloudflare-ipfs.com https://ipfs-sec.stackexchange.cloudflare-ipfs.com

nathcd 8y ago
Neat! Looks like this generally works by building a static index and publishing it to IPFS [1], then having some client-side JS do a lookup in the index, and getting a metadata file for each result [2]? I made something kinda similar a while ago [3], where I gathered a bunch of metadata about files on IPFS at the time by scraping a Searx instance, with a query like "site:ipfs.io/ipfs" or something like that. It was a quick hack job but was fun, and it's cool to see something similar on a bigger scale! [1] like https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index/tokens https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index... [2] like https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index/d/186 https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index... [3] https://ipfs.io/ipfs/QmYo5ZWqNW4ib1Ck4zdm6EKteX3zZWw1j4CVfKtnAzNdvu/ https://ipfs.io/ipfs/QmYo5ZWqNW4ib1Ck4zdm6EKteX3zZWw1j4CVfKt..., https://github.com/doesntgolf/ipfs-search https://github.com/doesntgolf/ipfs-search, https://github.com/doesntgolf/ipfs-searx-scraper https://github.com/doesntgolf/ipfs-searx-scraper
- lgierth 8y ago
  Yeah! Every question of "how would X work on IPFS?" boils down to data structures. An example of slightly different approach (a sharded index) is the wikipedia-on-ipfs experiment from last year: https://github.com/magik6k/distributed-wiki-search https://github.com/magik6k/distributed-wiki-search && https://github.com/ipfs/distributed-wikipedia-mirror https://github.com/ipfs/distributed-wikipedia-mirror
- hoschicz 8y ago
  Wow, I never thought there is something like Lunr.js! In my free time, I have been working on https://ipfsearch.xyz https://ipfsearch.xyz, thinking about pivoting into a direction of search in JS, but never getting the time to do it. Now I'm pretty sad that something like ipfsearch (most of my work was rebuilding Lunr.js) already exists. I guess I should do more research before embarking on a side project :( Anyway, I definitely learned a lot about how search engines work... P.S.: now, I'm noticing that Lunr can't fetch the index over the network. OK, so not all time has been lost!
  - nathcd 8y ago
    Yeah, and I think it actually uses a pretty old (0.x) version of lunr. You could definitely grab data over the network and then feed it into lunr's indexer, you'd just have to do it manually. I'd prefer to do it more like Cloudflare's stackexchange search (build the index locally, and publish the index to IPFS), rather than having the client have to fetch the data and build the index itself. Your tool looks cool! If I find the time, maybe I'll try to dig in a bit and see if I can help with anything!
  - cbluth 8y ago
    I'm selling ipfs.works domain, if you're interested
    - namanyayg 8y ago
      Would they really be interested if they just found out that most of their work was rebuilding an already existing tool..?
      - cbluth 8y ago
        Are you interested?
        Operyl 8y ago
        Starting bid, 99 cents! In reality, why are you soliciting the sale of your domain name that doesn't seem entirely relevant to the project this person was doing?
aw3c2 8y ago
Considering I want to use this as it is meant to be used, distributed, how do I pin all of say https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ham/ https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ham/ (including the search) to my local IPFS node so I can use it offline?
- diggan 8y ago
  Normally you would go about it like this: dig +short txt _dnslink.ipfs-sec.stackexchange.cloudflare-ipfs.com ^ will get you the root hash. Then for finding the hash of that particular directory: ipfs resolve /ipfs/QmVsEiBRsdiHMvmDVJTNtrXNqSKYdtoepcoZxUrQ8rU7qR/ham But unfortunately it errors, probably because it's a HAMT sharded directory and the `resolve` call hasn't been changed to accept them yet. There is a open issue about it here: https://github.com/ipfs/go-ipfs/issues/5270 https://github.com/ipfs/go-ipfs/issues/5270 Edit: Actually found a work-around. Loading up the hash via our IPLD Explorer reveals the (re-)name of the directory, which happens to be `05ham` and we can call `resolve` on that: "ipfs resolve /ipfs/QmVsEiBRsdiHMvmDVJTNtrXNqSKYdtoepcoZxUrQ8rU7qR/05ham" Then we get the `/ipfs/zdj7WZpLoZYr4AoDUdpCNBVHLi1S8y6muFoCgwUa9kmxUpYCr` hash which you can use for pinning.
  - wongarsu 8y ago
    Now we just need a browser extension that does all that. If pinning was a button press away it would be much more attractive than making a dns query, resolve the result and pinning that.
    - diggan 8y ago
      Indeed, that's a good idea! I took the freedom to open a issue about this idea on the repository for the IPFS-Companion extension here: https://github.com/ipfs-shipyard/ipfs-companion/issues/581 https://github.com/ipfs-shipyard/ipfs-companion/issues/581
  - aw3c2 8y ago
    So I've been trying to get "ipfs pin add --progress /ipfs/zdj7WZpLoZYr4AoDUdpCNBVHLi1S8y6muFoCgwUa9kmxUpYCr" to complete but it just takes hours and hours. Is this normal? What might be the issue?
wumms 8y ago
Stack Exchange for Kiwix, yes! I can't find Stack Overflow though - has anybody tried scraping it with https://github.com/openzim/sotoki https://github.com/openzim/sotoki ?
- bren2013 8y ago
  Kiwix has an archive of StackOverflow but its gigantic. :( We barely got the Math StackExchange uploaded in time.
  - wumms 8y ago
    Great Scott! 122GB. A weekly update of the local rpi would seem quite irresponsible on a regular basis - how would I use rsync with IPFS? (I could not find Wayne Davison's user id on HN, he seems to maintain rsync) - Are zim files optimized for rsync?
nikkwong 8y ago
How does IFPS index new content that's added to the stackexchange network daily? And/or how long does it take to add new data?
- X-Cubed 8y ago
  The site is a snapshot of StackExchange data. If a new snapshot gets published, a new index would be generated and uploaded with it by the uploader.
  - nikkwong 8y ago
    Ah. So it happens manually? Do automated backups occur? Or is that not the/a possible goal of IFPS? Just trying to understand.
    - lgierth 8y ago
      The automation is something you can do relatively easily with the IPFS libraries, API, or CLI. Ideally your approach to pulling the data from its source into IPFS is somewhat deterministic, so that you get de-duplication and generally more efficient updates.

nathcd 8y ago

Neat! Looks like this generally works by building a static index and publishing it to IPFS [1], then having some client-side JS do a lookup in the index, and getting a metadata file for each result [2]? I made something kinda similar a while ago [3], where I gathered a bunch of metadata about files on IPFS at the time by scraping a Searx instance, with a query like "site:ipfs.io/ipfs" or something like that. It was a quick hack job but was fun, and it's cool to see something similar on a bigger scale! [1] like https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index/tokens https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index... [2] like https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index/d/186 https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index... [3] https://ipfs.io/ipfs/QmYo5ZWqNW4ib1Ck4zdm6EKteX3zZWw1j4CVfKtnAzNdvu/ https://ipfs.io/ipfs/QmYo5ZWqNW4ib1Ck4zdm6EKteX3zZWw1j4CVfKt..., https://github.com/doesntgolf/ipfs-search https://github.com/doesntgolf/ipfs-search, https://github.com/doesntgolf/ipfs-searx-scraper https://github.com/doesntgolf/ipfs-searx-scraper

lgierth 8y ago
Yeah! Every question of "how would X work on IPFS?" boils down to data structures. An example of slightly different approach (a sharded index) is the wikipedia-on-ipfs experiment from last year: https://github.com/magik6k/distributed-wiki-search https://github.com/magik6k/distributed-wiki-search && https://github.com/ipfs/distributed-wikipedia-mirror https://github.com/ipfs/distributed-wikipedia-mirror
hoschicz 8y ago
Wow, I never thought there is something like Lunr.js! In my free time, I have been working on https://ipfsearch.xyz https://ipfsearch.xyz, thinking about pivoting into a direction of search in JS, but never getting the time to do it. Now I'm pretty sad that something like ipfsearch (most of my work was rebuilding Lunr.js) already exists. I guess I should do more research before embarking on a side project :( Anyway, I definitely learned a lot about how search engines work... P.S.: now, I'm noticing that Lunr can't fetch the index over the network. OK, so not all time has been lost!
- nathcd 8y ago
  Yeah, and I think it actually uses a pretty old (0.x) version of lunr. You could definitely grab data over the network and then feed it into lunr's indexer, you'd just have to do it manually. I'd prefer to do it more like Cloudflare's stackexchange search (build the index locally, and publish the index to IPFS), rather than having the client have to fetch the data and build the index itself. Your tool looks cool! If I find the time, maybe I'll try to dig in a bit and see if I can help with anything!
- cbluth 8y ago
  I'm selling ipfs.works domain, if you're interested
  - namanyayg 8y ago
    Would they really be interested if they just found out that most of their work was rebuilding an already existing tool..?
    - cbluth 8y ago
      Are you interested?
      - Operyl 8y ago
        Starting bid, 99 cents! In reality, why are you soliciting the sale of your domain name that doesn't seem entirely relevant to the project this person was doing?

aw3c2 8y ago

Considering I want to use this as it is meant to be used, distributed, how do I pin all of say https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ham/ https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ham/ (including the search) to my local IPFS node so I can use it offline?

diggan 8y ago
Normally you would go about it like this: dig +short txt _dnslink.ipfs-sec.stackexchange.cloudflare-ipfs.com ^ will get you the root hash. Then for finding the hash of that particular directory: ipfs resolve /ipfs/QmVsEiBRsdiHMvmDVJTNtrXNqSKYdtoepcoZxUrQ8rU7qR/ham But unfortunately it errors, probably because it's a HAMT sharded directory and the `resolve` call hasn't been changed to accept them yet. There is a open issue about it here: https://github.com/ipfs/go-ipfs/issues/5270 https://github.com/ipfs/go-ipfs/issues/5270 Edit: Actually found a work-around. Loading up the hash via our IPLD Explorer reveals the (re-)name of the directory, which happens to be `05ham` and we can call `resolve` on that: "ipfs resolve /ipfs/QmVsEiBRsdiHMvmDVJTNtrXNqSKYdtoepcoZxUrQ8rU7qR/05ham" Then we get the `/ipfs/zdj7WZpLoZYr4AoDUdpCNBVHLi1S8y6muFoCgwUa9kmxUpYCr` hash which you can use for pinning.
- wongarsu 8y ago
  Now we just need a browser extension that does all that. If pinning was a button press away it would be much more attractive than making a dns query, resolve the result and pinning that.
  - diggan 8y ago
    Indeed, that's a good idea! I took the freedom to open a issue about this idea on the repository for the IPFS-Companion extension here: https://github.com/ipfs-shipyard/ipfs-companion/issues/581 https://github.com/ipfs-shipyard/ipfs-companion/issues/581
- aw3c2 8y ago
  So I've been trying to get "ipfs pin add --progress /ipfs/zdj7WZpLoZYr4AoDUdpCNBVHLi1S8y6muFoCgwUa9kmxUpYCr" to complete but it just takes hours and hours. Is this normal? What might be the issue?

wumms 8y ago

Stack Exchange for Kiwix, yes! I can't find Stack Overflow though - has anybody tried scraping it with https://github.com/openzim/sotoki https://github.com/openzim/sotoki ?

bren2013 8y ago
Kiwix has an archive of StackOverflow but its gigantic. :( We barely got the Math StackExchange uploaded in time.
- wumms 8y ago
  Great Scott! 122GB. A weekly update of the local rpi would seem quite irresponsible on a regular basis - how would I use rsync with IPFS? (I could not find Wayne Davison's user id on HN, he seems to maintain rsync) - Are zim files optimized for rsync?

nikkwong 8y ago

How does IFPS index new content that's added to the stackexchange network daily? And/or how long does it take to add new data?

X-Cubed 8y ago
The site is a snapshot of StackExchange data. If a new snapshot gets published, a new index would be generated and uploaded with it by the uploader.
- nikkwong 8y ago
  Ah. So it happens manually? Do automated backups occur? Or is that not the/a possible goal of IFPS? Just trying to understand.
  - lgierth 8y ago
    The automation is something you can do relatively easily with the IPFS libraries, API, or CLI. Ideally your approach to pulling the data from its source into IPFS is somewhat deterministic, so that you get de-duplication and generally more efficient updates.