3 ms·

Neat! Looks like this generally works by building a static index and publishing it to IPFS [1], then having some client-side JS do a lookup in the index, and ge

by nathcd 8y ago

Neat! Looks like this generally works by building a static index and publishing it to IPFS [1], then having some client-side JS do a lookup in the index, and getting a metadata file for each result [2]? I made something kinda similar a while ago [3], where I gathered a bunch of metadata about files on IPFS at the time by scraping a Searx instance, with a query like "site:ipfs.io/ipfs" or something like that. It was a quick hack job but was fun, and it's cool to see something similar on a bigger scale! [1] like https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index/tokens https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index... [2] like https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index/d/186 https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index... [3] https://ipfs.io/ipfs/QmYo5ZWqNW4ib1Ck4zdm6EKteX3zZWw1j4CVfKtnAzNdvu/ https://ipfs.io/ipfs/QmYo5ZWqNW4ib1Ck4zdm6EKteX3zZWw1j4CVfKt..., https://github.com/doesntgolf/ipfs-search https://github.com/doesntgolf/ipfs-search, https://github.com/doesntgolf/ipfs-searx-scraper https://github.com/doesntgolf/ipfs-searx-scraper

lgierth 8y ago
Yeah! Every question of "how would X work on IPFS?" boils down to data structures. An example of slightly different approach (a sharded index) is the wikipedia-on-ipfs experiment from last year: https://github.com/magik6k/distributed-wiki-search https://github.com/magik6k/distributed-wiki-search && https://github.com/ipfs/distributed-wikipedia-mirror https://github.com/ipfs/distributed-wikipedia-mirror
hoschicz 8y ago
Wow, I never thought there is something like Lunr.js! In my free time, I have been working on https://ipfsearch.xyz https://ipfsearch.xyz, thinking about pivoting into a direction of search in JS, but never getting the time to do it. Now I'm pretty sad that something like ipfsearch (most of my work was rebuilding Lunr.js) already exists. I guess I should do more research before embarking on a side project :( Anyway, I definitely learned a lot about how search engines work... P.S.: now, I'm noticing that Lunr can't fetch the index over the network. OK, so not all time has been lost!
- nathcd 8y ago
  Yeah, and I think it actually uses a pretty old (0.x) version of lunr. You could definitely grab data over the network and then feed it into lunr's indexer, you'd just have to do it manually. I'd prefer to do it more like Cloudflare's stackexchange search (build the index locally, and publish the index to IPFS), rather than having the client have to fetch the data and build the index itself. Your tool looks cool! If I find the time, maybe I'll try to dig in a bit and see if I can help with anything!
- cbluth 8y ago
  I'm selling ipfs.works domain, if you're interested
  - namanyayg 8y ago
    Would they really be interested if they just found out that most of their work was rebuilding an already existing tool..?
    - cbluth 8y ago
      Are you interested?
      - Operyl 8y ago
        Starting bid, 99 cents! In reality, why are you soliciting the sale of your domain name that doesn't seem entirely relevant to the project this person was doing?
        cbluth 8y ago
        Sold for 99¢. Its about to expire, and I would rather someone else have it, perhaps someone with a little more ambition than I.

by nathcd 8y ago

lgierth 8y ago

Yeah! Every question of "how would X work on IPFS?" boils down to data structures. An example of slightly different approach (a sharded index) is the wikipedia-on-ipfs experiment from last year: https://github.com/magik6k/distributed-wiki-search https://github.com/magik6k/distributed-wiki-search && https://github.com/ipfs/distributed-wikipedia-mirror https://github.com/ipfs/distributed-wikipedia-mirror

hoschicz 8y ago

Wow, I never thought there is something like Lunr.js! In my free time, I have been working on https://ipfsearch.xyz https://ipfsearch.xyz, thinking about pivoting into a direction of search in JS, but never getting the time to do it. Now I'm pretty sad that something like ipfsearch (most of my work was rebuilding Lunr.js) already exists. I guess I should do more research before embarking on a side project :( Anyway, I definitely learned a lot about how search engines work... P.S.: now, I'm noticing that Lunr can't fetch the index over the network. OK, so not all time has been lost!

nathcd 8y ago
Yeah, and I think it actually uses a pretty old (0.x) version of lunr. You could definitely grab data over the network and then feed it into lunr's indexer, you'd just have to do it manually. I'd prefer to do it more like Cloudflare's stackexchange search (build the index locally, and publish the index to IPFS), rather than having the client have to fetch the data and build the index itself. Your tool looks cool! If I find the time, maybe I'll try to dig in a bit and see if I can help with anything!
cbluth 8y ago
I'm selling ipfs.works domain, if you're interested
- namanyayg 8y ago
  Would they really be interested if they just found out that most of their work was rebuilding an already existing tool..?
  - cbluth 8y ago
    Are you interested?
    - Operyl 8y ago
      Starting bid, 99 cents! In reality, why are you soliciting the sale of your domain name that doesn't seem entirely relevant to the project this person was doing?
      - cbluth 8y ago
        Sold for 99¢. Its about to expire, and I would rather someone else have it, perhaps someone with a little more ambition than I.