6 ms·

I built an IPFS crawler with libp2p

116 points by adlrocha 6y ago

carapace 6y ago
So this is crawling the nodes not the content?
- mirimir 6y ago
  Seems so. There is https://ipfs-search.com/ https://ipfs-search.com/ [https://github.com/ipfs-search/ipfs-search https://github.com/ipfs-search/ipfs-search].
kickscondor 6y ago
Was hoping there was a link to the results - either a search box or some statistics. But hey - free code. Not complaining. Also, if you haven't checked out Beaker Browser now that it's nearing 1.0 - great things are happening. This hasn't been broadcast much - but Beaker now has a built-in editor pane and forking mechanism. It's the best shot I've seen at a 'view source' 2.0. There's been some healthy competition lately forking and editing the default twitter/mastodon clone. (Which is here: hyper://a8e9bd0f4df60ed5246a1b1f53d51a1feaeb1315266f769ac218436f12fda830/ But forks of it are here: hyper://0e0d1685c3575cf1df94191e14b4ecf068c43bcefee787ffbbed3d1f4666cab4/ and here: hyper://b798b86cc282ae065697f36a2a07072bfa28e5fdfa1d9a8c6fac0f07ff7d6676/.) If you want to learn Javascript, this seems like an amazing way. (Kind of like jsfiddle but for actual apps.)
- Malp 6y ago
  hyper://a8e9bd0f4df60ed5246a1b1f53d51a1feaeb1315266f769ac21 has been giving me a 500
  - kickscondor 6y ago
    Somehow your link is truncated. The complete link is: hyper://a8e9bd0f4df60ed5246a1b1f53d51a1feaeb1315266f769ac218436f12fda830/ (with a '30' at the end)
    - Malp 6y ago
      My bad
- koeng 6y ago
  Is beaker going to support IPFS?
  - anchpop 6y ago
    They used to, but they removed support. It makes sense to focus on just one protocol I think since they're such a lean operation. Also the guy running Beaker seems to be sort of anti-blockchain (describing himself as an "Unblockchain consultant") which might make them biased against it. Also, IPFS urls change when the content they're pointing to changes, which makes them bad for decentralized web stuff. They are working on IPNS to handle this but I don't know what the status is.
    - p4bl0 6y ago
      IPNS has been working for a few years already. Both the dnslink and actual IPNS. However an instance of the IPFS daemon can only serve a single IPNS address. The real problem with IPFS is that hosting a node requires a lot of bandwidth. I regularly tried to self-host an IPFSite but it makes make internet access literally unusable. So I abandoned that idea for good and now I use a free external service for that.
    - momack2 6y ago
      Actually, content-based addresses is really critical for decentralized web stuff - since you can't trust a single source to name data, and instead want many different parties to participate in the same global address space in a secure and trustless manner. Check out the dweb primer for why working with a base-layer built on content addressing is so powerful: https://flyingzumwalt.gitbooks.io/decentralized-web-primer/avenues-for-access/lessons/power-of-content-addressing.html https://flyingzumwalt.gitbooks.io/decentralized-web-primer/a... IPNS is used by many groups to create a mutable layer over IPFS (it recently got much faster in our 0.5.0 release in April), but you can also use tools like Textile, OrbitDB, or other mutable databases built on IPFS.
      - anchpop 6y ago
        I read that document and it seems to me like everything in there applies to Dat also, which isn't really content-addressed. But if IPNS is fast now it seems like it could be a good replacement.
wrnr 6y ago
How is the adoption of IPFS going, any example application that are using it and are themselves being used? I'm thinking of integrading it into a product, it has the best blockchain like properties without the mining madness.
- momack2 6y ago
  Going well! Check out all the awesome applications using IPFS in our editable ecosystem diagram: https://www.figma.com/file/2bVNnjoOmxh4TtID9kzqEG/IPFS-Applications-Diagram https://www.figma.com/file/2bVNnjoOmxh4TtID9kzqEG/IPFS-Appli... A few highlights are Fleek, Textile, Audius, 3box, Anytype, Qri, Berty, and MagicLeap! Specifically in the blockchain space, there's Peepeth, Augur, Uniswap, Civic, Arbore, and Aragon.
momack2 6y ago
For anyone looking at https://arxiv.org/pdf/2002.07747.pdf https://arxiv.org/pdf/2002.07747.pdf to get an estimate on the actual # of IPFS nodes, note that these tools are trying to crawl the number of nodes acting as "DHT Servers" in the Public IPFS DHT. Nodes behind home firewalls or NATs don't become DHT routers because they aren't dialable, and other large users intentionally set many of their nodes into a lower-bandwidth client mode. Also, many applications run their own private IPFS networks (like OpenBazaar for instance) and don't participate in the Public IPFS DHT, so wouldn't be reached through a crawler here.

116 points by adlrocha 6y ago

carapace 6y ago

So this is crawling the nodes not the content?

mirimir 6y ago
Seems so. There is https://ipfs-search.com/ https://ipfs-search.com/ [https://github.com/ipfs-search/ipfs-search https://github.com/ipfs-search/ipfs-search].

kickscondor 6y ago

Was hoping there was a link to the results - either a search box or some statistics. But hey - free code. Not complaining. Also, if you haven't checked out Beaker Browser now that it's nearing 1.0 - great things are happening. This hasn't been broadcast much - but Beaker now has a built-in editor pane and forking mechanism. It's the best shot I've seen at a 'view source' 2.0. There's been some healthy competition lately forking and editing the default twitter/mastodon clone. (Which is here: hyper://a8e9bd0f4df60ed5246a1b1f53d51a1feaeb1315266f769ac218436f12fda830/ But forks of it are here: hyper://0e0d1685c3575cf1df94191e14b4ecf068c43bcefee787ffbbed3d1f4666cab4/ and here: hyper://b798b86cc282ae065697f36a2a07072bfa28e5fdfa1d9a8c6fac0f07ff7d6676/.) If you want to learn Javascript, this seems like an amazing way. (Kind of like jsfiddle but for actual apps.)

Malp 6y ago
hyper://a8e9bd0f4df60ed5246a1b1f53d51a1feaeb1315266f769ac21 has been giving me a 500
- kickscondor 6y ago
  Somehow your link is truncated. The complete link is: hyper://a8e9bd0f4df60ed5246a1b1f53d51a1feaeb1315266f769ac218436f12fda830/ (with a '30' at the end)
  - Malp 6y ago
    My bad
koeng 6y ago
Is beaker going to support IPFS?
- anchpop 6y ago
  They used to, but they removed support. It makes sense to focus on just one protocol I think since they're such a lean operation. Also the guy running Beaker seems to be sort of anti-blockchain (describing himself as an "Unblockchain consultant") which might make them biased against it. Also, IPFS urls change when the content they're pointing to changes, which makes them bad for decentralized web stuff. They are working on IPNS to handle this but I don't know what the status is.
  - p4bl0 6y ago
    IPNS has been working for a few years already. Both the dnslink and actual IPNS. However an instance of the IPFS daemon can only serve a single IPNS address. The real problem with IPFS is that hosting a node requires a lot of bandwidth. I regularly tried to self-host an IPFSite but it makes make internet access literally unusable. So I abandoned that idea for good and now I use a free external service for that.
  - momack2 6y ago
    Actually, content-based addresses is really critical for decentralized web stuff - since you can't trust a single source to name data, and instead want many different parties to participate in the same global address space in a secure and trustless manner. Check out the dweb primer for why working with a base-layer built on content addressing is so powerful: https://flyingzumwalt.gitbooks.io/decentralized-web-primer/avenues-for-access/lessons/power-of-content-addressing.html https://flyingzumwalt.gitbooks.io/decentralized-web-primer/a... IPNS is used by many groups to create a mutable layer over IPFS (it recently got much faster in our 0.5.0 release in April), but you can also use tools like Textile, OrbitDB, or other mutable databases built on IPFS.
    - anchpop 6y ago
      I read that document and it seems to me like everything in there applies to Dat also, which isn't really content-addressed. But if IPNS is fast now it seems like it could be a good replacement.

wrnr 6y ago

How is the adoption of IPFS going, any example application that are using it and are themselves being used? I'm thinking of integrading it into a product, it has the best blockchain like properties without the mining madness.

momack2 6y ago
Going well! Check out all the awesome applications using IPFS in our editable ecosystem diagram: https://www.figma.com/file/2bVNnjoOmxh4TtID9kzqEG/IPFS-Applications-Diagram https://www.figma.com/file/2bVNnjoOmxh4TtID9kzqEG/IPFS-Appli... A few highlights are Fleek, Textile, Audius, 3box, Anytype, Qri, Berty, and MagicLeap! Specifically in the blockchain space, there's Peepeth, Augur, Uniswap, Civic, Arbore, and Aragon.

momack2 6y ago

For anyone looking at https://arxiv.org/pdf/2002.07747.pdf https://arxiv.org/pdf/2002.07747.pdf to get an estimate on the actual # of IPFS nodes, note that these tools are trying to crawl the number of nodes acting as "DHT Servers" in the Public IPFS DHT. Nodes behind home firewalls or NATs don't become DHT routers because they aren't dialable, and other large users intentionally set many of their nodes into a lower-bandwidth client mode. Also, many applications run their own private IPFS networks (like OpenBazaar for instance) and don't participate in the Public IPFS DHT, so wouldn't be reached through a crawler here.