4 ms·

Hey, IPFS dev here, happy to help :) I promise we'll improve our documentation around storing and providing large datasets. 1) You can have individual files au

Hey, IPFS dev here, happy to help :) I promise we'll improve our documentation around storing and providing large datasets. 1) You can have individual files automatically be wrapped in a directory: `ipfs add -w dog.jpg` will result in `/ipfs/QmSomeHash/dog.jpg` 2) There's commands for modifying objects: e.g. `ipfs object patch add-link QmDirHash the-link-name QmLinkHash` 3) You can host multiple IPNS names from the same node since 0.4.9 or so -- see `ipfs key` and `ipfs name publish -k`. You can also publish updates for the same IPNS name from multiple nodes, although admittedly that's a bit hacky right now, since it involves an additional tool: https://github.com/whyrusleeping/ipns-pub https://github.com/whyrusleeping/ipns-pub 4) What kind of directory structure are you looking for here? We've successfully and repeatedly used IPFS with large datasets of different kinds so I'm curious how we can improve this.

ashark 9y ago
Oh man, I could have sworn I saw commands like in (2) a while back but then just could not find them again the other day when I went looking, so I thought I'd imagined it. Great news. So the workflow there is to add the new file or directory, then link the hash you get back inside the collection you want to modify with the `object patch add-link` business? Not exactly ideal for direct use, but simple enough to be abstracted away in a not-insane script, looks like. Nice. On (3), cool, I'd checked out a bit on following IPFS while waiting for private networks and filesystem-store to leave experimental status (here's where you tell me that's happened, too!) so I must have missed that. Good to know. (4) was mostly my explaining why just having all the files individually in IPFS is really, really useless if you need to apply any tools to them that expect to operate on one or a small number of (possibly structured) directories, which describes an awful lot of tools that operate on large numbers of files, but that's moot given collection-modifying commands are available. Thanks for the response.
- lgierth 9y ago
  And thanks for the feedback! :) We're actually thinking through the documentation overhaul right now (today and tomorrow) and large datasets are a top priority use case, so your post is really valuable. I'd say at this point private networks and the filestore are unlikely to change significantly, but we're not 100% satisfied with the respective test suites (and the docs, oh my) so it'll be a bit longer until they'll be elevated out of experimental status. About the object patch command, you can also call it through the HTTP API if that makes it easier. The CLI is essentially a 1:1 mapping of the HTTP API to a console command.
jancsika 9y ago
Now that we're talking about IPFS, here's a slightly more complicated question: Suppose a maintainer has a "legacy web" application consisting of two parts: 1) the app proper with a publicly accessible interface, and 2) regular backups of the app database that also have a publicly accessible interface. Let's assume for the moment that 3rd parties have automated the process of pulling from the backups into well-seeded torrents to prevent data loss. The app proper functions to add entries to the database. What does IPFS bring to the table for such a legacy web app as the one I've described?
- lgierth 9y ago
  You're not giving me much to work with :) It's all about the data, how you write/read it, and how you structure it. IPFS gives you a networked database for immutable graphs, IPNS for mutable links, and PubSub plus CRDTs for multi-writer scenarios.
  - jancsika 9y ago
    To be specific-- the legacy web app maintainer has to make sure that a) snapshots of the data are available and backed up, and b) the app proper is available to users who will add entries to the database. For the legacy web, once the maintainer makes the snapshots available they get scooped up and replicated by the general public. So while IPFS' immutable graph might be a design improvement on whatever set of scripts our hypothetical maintainer is using, it's an incremental improvement and not a paradigm shift. For the app proper-- what is the benefit of content-addressability over location-addressability? Let's assume both the identity and location of the maintainer is known. I think I know a vague answer to that, but I'd like to read what a dev has to say about it.

ashark 9y ago

Oh man, I could have sworn I saw commands like in (2) a while back but then just could not find them again the other day when I went looking, so I thought I'd imagined it. Great news. So the workflow there is to add the new file or directory, then link the hash you get back inside the collection you want to modify with the `object patch add-link` business? Not exactly ideal for direct use, but simple enough to be abstracted away in a not-insane script, looks like. Nice. On (3), cool, I'd checked out a bit on following IPFS while waiting for private networks and filesystem-store to leave experimental status (here's where you tell me that's happened, too!) so I must have missed that. Good to know. (4) was mostly my explaining why just having all the files individually in IPFS is really, really useless if you need to apply any tools to them that expect to operate on one or a small number of (possibly structured) directories, which describes an awful lot of tools that operate on large numbers of files, but that's moot given collection-modifying commands are available. Thanks for the response.

lgierth 9y ago
And thanks for the feedback! :) We're actually thinking through the documentation overhaul right now (today and tomorrow) and large datasets are a top priority use case, so your post is really valuable. I'd say at this point private networks and the filestore are unlikely to change significantly, but we're not 100% satisfied with the respective test suites (and the docs, oh my) so it'll be a bit longer until they'll be elevated out of experimental status. About the object patch command, you can also call it through the HTTP API if that makes it easier. The CLI is essentially a 1:1 mapping of the HTTP API to a console command.

jancsika 9y ago

Now that we're talking about IPFS, here's a slightly more complicated question: Suppose a maintainer has a "legacy web" application consisting of two parts: 1) the app proper with a publicly accessible interface, and 2) regular backups of the app database that also have a publicly accessible interface. Let's assume for the moment that 3rd parties have automated the process of pulling from the backups into well-seeded torrents to prevent data loss. The app proper functions to add entries to the database. What does IPFS bring to the table for such a legacy web app as the one I've described?

lgierth 9y ago
You're not giving me much to work with :) It's all about the data, how you write/read it, and how you structure it. IPFS gives you a networked database for immutable graphs, IPNS for mutable links, and PubSub plus CRDTs for multi-writer scenarios.
- jancsika 9y ago
  To be specific-- the legacy web app maintainer has to make sure that a) snapshots of the data are available and backed up, and b) the app proper is available to users who will add entries to the database. For the legacy web, once the maintainer makes the snapshots available they get scooped up and replicated by the general public. So while IPFS' immutable graph might be a design improvement on whatever set of scripts our hypothetical maintainer is using, it's an incremental improvement and not a paradigm shift. For the app proper-- what is the benefit of content-addressability over location-addressability? Let's assume both the identity and location of the maintainer is known. I think I know a vague answer to that, but I'd like to read what a dev has to say about it.