4 ms·

IPFS has its own challenges with large, living collections like this, which results in their not being especially navigation- and updating-friendly without an e

by ashark 9y ago

IPFS has its own challenges with large, living collections like this, which results in their not being especially navigation- and updating-friendly without an external content index and some custom tooling on top of IPFS. In practice it ends up being about as cumbersome as torrents. At least AFAIK. I'd love to know if that's not true, since it's one of the things keeping me from adopting IPFS for certain use cases.

jrowley 9y ago
Ahh the immutable nature could be a limitation. Good point.
- ashark 9y ago
  Specific issues include: 1) you can only provide useful names (not simply hashes) for things that are in directories, so a large number of bare files (easiest to update in IPFS) absolutely requires an external index linking the hash to a friendly name. I haven't found a suitable tool for automating any of this. 2) you can't modify an existing IPFS directory in-place and just get a new hash—you need the whole thing outside of IPFS to perform a CRUD operation. Maybe you can perform some trick with links or something, IDK, but it's definitely a pain. This makes One Giant Root Directory to Rule Them All very cumbersome. 3) IPNS is kind-of a solution, but centralizes things, and you can only host one of them per node-daemon (!) and last I checked these were still prone to availability problems and general quirky behavior. 4) Most software that doesn't speak IPFS natively (so, nearly everything) expects stuff to be in ordinary directories. Nothing expects to receive as config giant list of files that's changing all the time. They all expect one directory, or maybe a few directories, often with some reasonable structure. Translating between easiest-for-IPFS (a whole bunch of bare files, or smallish sets of closely related files each in their own root directory) and easiest-for-literally-all-other-software-that-uses-the-filesystem practically requires some kind of linking—I am not aware of any existing solution for this. [EDIT] to be clear, (4) is no less a problem if you're using an IPFS FUSE mount. In fact you'd have to for linking to be viable at all, though I'm not 100% sure it works even then.
  - lgierth 9y ago
    Hey, IPFS dev here, happy to help :) I promise we'll improve our documentation around storing and providing large datasets. 1) You can have individual files automatically be wrapped in a directory: `ipfs add -w dog.jpg` will result in `/ipfs/QmSomeHash/dog.jpg` 2) There's commands for modifying objects: e.g. `ipfs object patch add-link QmDirHash the-link-name QmLinkHash` 3) You can host multiple IPNS names from the same node since 0.4.9 or so -- see `ipfs key` and `ipfs name publish -k`. You can also publish updates for the same IPNS name from multiple nodes, although admittedly that's a bit hacky right now, since it involves an additional tool: https://github.com/whyrusleeping/ipns-pub https://github.com/whyrusleeping/ipns-pub 4) What kind of directory structure are you looking for here? We've successfully and repeatedly used IPFS with large datasets of different kinds so I'm curious how we can improve this.
    - ashark 9y ago
      Oh man, I could have sworn I saw commands like in (2) a while back but then just could not find them again the other day when I went looking, so I thought I'd imagined it. Great news. So the workflow there is to add the new file or directory, then link the hash you get back inside the collection you want to modify with the `object patch add-link` business? Not exactly ideal for direct use, but simple enough to be abstracted away in a not-insane script, looks like. Nice. On (3), cool, I'd checked out a bit on following IPFS while waiting for private networks and filesystem-store to leave experimental status (here's where you tell me that's happened, too!) so I must have missed that. Good to know. (4) was mostly my explaining why just having all the files individually in IPFS is really, really useless if you need to apply any tools to them that expect to operate on one or a small number of (possibly structured) directories, which describes an awful lot of tools that operate on large numbers of files, but that's moot given collection-modifying commands are available. Thanks for the response.
      - lgierth 9y ago
        And thanks for the feedback! :) We're actually thinking through the documentation overhaul right now (today and tomorrow) and large datasets are a top priority use case, so your post is really valuable. I'd say at this point private networks and the filestore are unlikely to change significantly, but we're not 100% satisfied with the respective test suites (and the docs, oh my) so it'll be a bit longer until they'll be elevated out of experimental status. About the object patch command, you can also call it through the HTTP API if that makes it easier. The CLI is essentially a 1:1 mapping of the HTTP API to a console command.

by ashark 9y ago

jrowley 9y ago

Ahh the immutable nature could be a limitation. Good point.

ashark 9y ago
Specific issues include: 1) you can only provide useful names (not simply hashes) for things that are in directories, so a large number of bare files (easiest to update in IPFS) absolutely requires an external index linking the hash to a friendly name. I haven't found a suitable tool for automating any of this. 2) you can't modify an existing IPFS directory in-place and just get a new hash—you need the whole thing outside of IPFS to perform a CRUD operation. Maybe you can perform some trick with links or something, IDK, but it's definitely a pain. This makes One Giant Root Directory to Rule Them All very cumbersome. 3) IPNS is kind-of a solution, but centralizes things, and you can only host one of them per node-daemon (!) and last I checked these were still prone to availability problems and general quirky behavior. 4) Most software that doesn't speak IPFS natively (so, nearly everything) expects stuff to be in ordinary directories. Nothing expects to receive as config giant list of files that's changing all the time. They all expect one directory, or maybe a few directories, often with some reasonable structure. Translating between easiest-for-IPFS (a whole bunch of bare files, or smallish sets of closely related files each in their own root directory) and easiest-for-literally-all-other-software-that-uses-the-filesystem practically requires some kind of linking—I am not aware of any existing solution for this. [EDIT] to be clear, (4) is no less a problem if you're using an IPFS FUSE mount. In fact you'd have to for linking to be viable at all, though I'm not 100% sure it works even then.
- lgierth 9y ago
  Hey, IPFS dev here, happy to help :) I promise we'll improve our documentation around storing and providing large datasets. 1) You can have individual files automatically be wrapped in a directory: `ipfs add -w dog.jpg` will result in `/ipfs/QmSomeHash/dog.jpg` 2) There's commands for modifying objects: e.g. `ipfs object patch add-link QmDirHash the-link-name QmLinkHash` 3) You can host multiple IPNS names from the same node since 0.4.9 or so -- see `ipfs key` and `ipfs name publish -k`. You can also publish updates for the same IPNS name from multiple nodes, although admittedly that's a bit hacky right now, since it involves an additional tool: https://github.com/whyrusleeping/ipns-pub https://github.com/whyrusleeping/ipns-pub 4) What kind of directory structure are you looking for here? We've successfully and repeatedly used IPFS with large datasets of different kinds so I'm curious how we can improve this.
  - ashark 9y ago
    Oh man, I could have sworn I saw commands like in (2) a while back but then just could not find them again the other day when I went looking, so I thought I'd imagined it. Great news. So the workflow there is to add the new file or directory, then link the hash you get back inside the collection you want to modify with the `object patch add-link` business? Not exactly ideal for direct use, but simple enough to be abstracted away in a not-insane script, looks like. Nice. On (3), cool, I'd checked out a bit on following IPFS while waiting for private networks and filesystem-store to leave experimental status (here's where you tell me that's happened, too!) so I must have missed that. Good to know. (4) was mostly my explaining why just having all the files individually in IPFS is really, really useless if you need to apply any tools to them that expect to operate on one or a small number of (possibly structured) directories, which describes an awful lot of tools that operate on large numbers of files, but that's moot given collection-modifying commands are available. Thanks for the response.
    - lgierth 9y ago
      And thanks for the feedback! :) We're actually thinking through the documentation overhaul right now (today and tomorrow) and large datasets are a top priority use case, so your post is really valuable. I'd say at this point private networks and the filestore are unlikely to change significantly, but we're not 100% satisfied with the respective test suites (and the docs, oh my) so it'll be a bit longer until they'll be elevated out of experimental status. About the object patch command, you can also call it through the HTTP API if that makes it easier. The CLI is essentially a 1:1 mapping of the HTTP API to a console command.