3 ms·
Hi @Arathorn, any work on using IPFS as the media store? It seems like this would be also a really good project for Matrix and doubly so for p2p matrix...
by rglullis 6y ago
Hi @Arathorn, any work on using IPFS as the media store? It seems like this would be also a really good project for Matrix and doubly so for p2p matrix...
- vertex-four 6y agoIPFS doesn't help - you still need a store-and-forward architecture for messaging, so you might as well use that for images too. IPFS is not a store-and-forward architecture; a file does not get automatically stored by the system, so when the sender disconnects, the file is no longer accessible. There's "solutions" to this, like Filecoin, but nobody wants folks to be paying to upload images. Or, in other words - the problem IPFS solves is peer discovery for a given hash, not storage. In Matrix, you already know who your peer is - it's whoever sent you the message referencing the image.
- rglullis 6y agoMy idea was more in that every p2p node of Matrix can also be a IPFS server, and be used to pin all of your own media + a small cache of your contacts/group media. You wouldn't need to save all of the media your homeserver is dealing with, but you still could be reasonably sure that your node would be able to find all media you are supposed to access. Given that the post talks about libp2p and how libp2p's main driver is IPFS, it seemed reasonable to think that if you are adding Dentrite to your browser you might embed an IPFS server as well.
- vertex-four 6y agoBut if you're doing that then those nodes could also implement the store-and-forward aspect of p2p matrix, which comes with its own non-IPFS discovery and thus solves the discovery problem. libp2p is what IPFS is built on, not the other way round.
- rglullis 6y agoI think I am failing to get my idea through. I am thinking more in the sense of having IPFS as a way to increase redundancy of nodes that might be able to serve some media. Yes, discovery is not an issue for matrix. But p2p matrix on its own is not able to say "oh, your contact is sending you a message that contains a 10GB file, let me see if I can find other nodes that have this file already and get it from the swarm" 10GB is an exaggeration, but picture a group sharing hundreds of silly videos a day and you already can distribute the load quite a bit, no?
- Groxx 6y agoAnime-over-IRC can hit 10GB files pretty easily. I wouldn't label it an exaggeration so much as "uncommon". I do like the idea of automatically deduplicating every shared gif tho, which would happen automatically since it's content-addressed.
- vertex-four 6y agoYes, possibly this could be an optimisation; although then you get to worry about leaking information about your group's contents to the world - anything you send, other clients in your group will ask random servers on the internet for, by a globally identifiable hash. In the best case, this just leaks my friends' meme sharing habits - in the worst case, this announces to the world "I am in a group sharing materials that my Government thinks are dangerous". If someone posts an original file to a public group and a private group, then anyone who downloads that file who isn't in the public group obviously knows that person well enough to be in a private group with them, leaking group membership metadata. And so on. Random identifiers for each instance of sharing a file to a group would thus be needed, reducing the usefulness of this optimisation quite a bit, where e.g. using the results of the group's p2p matrix discovery mechanism as your peer list to download blocks from has much the same results.
- rglullis 6y agoYou are right about the problem of leaking metadata - just by asking for the file reveals to the world that you know about it. However, this is something that can be fixed on IPFS itself, and I believe it is even in their roadmap: the idea of adding "friend peers". Your IPFS node could ask for sensitive files to this privileged list of nodes only. So, your client can have a simple logic: if you have e2e encrypted message with IPFS-linked media, ask only the group member peers about the contents of the file. Messages that are not e2e (or marked as non-sensitive) can be asked to the global IPFS swarm.
- viraptor 6y ago> + a small cache of your contacts/group media This idea always sounds interesting and dangerous to me. There's a lot of content you really don't want to be a provider for for legal and other reasons. Caching it for yourself is one thing - public distribution is another.
- rglullis 6y agoYeah, I stopped development of one project due to this: https://news.ycombinator.com/item?id=22246797 https://news.ycombinator.com/item?id=22246797
- Arathorn 6y agohey - for whatever reason, nobody's hooked up IPFS (as far as I know) as a filestore. It might be because you still need to pin the content somewhere, as vertex-four says, and so even in a P2P network the fact that the nodes keep coming and going means it might not help much.
- tulir 6y agomatrix-media-repo has an early implementation for IPFS support: https://github.com/turt2live/matrix-media-repo/issues/115#issuecomment-593772429 https://github.com/turt2live/matrix-media-repo/issues/115#is...
- rglullis 6y agoThanks for sharing this. I am working on matrix hosting and a multi-homed media service will be very useful.
- karatestomp 6y agoI thought part of the point of how IPFS is designed is that you don't need to "implement" it, you just ask for a hash (file) in the IPFS mount and it goes and grabs it if it's not already there? Should just be a matter of handling IPFS URLs correctly. Which, I guess that's something to implement, kinda, though if you already have a handler for 'ipfs://' and Matrix passes off a click on such a URL to your OS to handle as it will (as it might to a browser for 'http://' http://') then there really isn't anything to do
- rglullis 6y agoI am talking about the media store, the subsystem of the Matrix homeserver that stores all media that is uploaded.
- karatestomp 6y agoAh, yeah, that's less straightforward. checks the docs Looks like you could point it at IPFS "MFS" mount (Mutable File System) instead of whatever directory it's using now for its normal local storage media store (which I assume exists?), and that'd do the trick. Files stored in IPFS (so mirrorable through that, or whatever it is that you've got in mind) but appearing as ordinary files with normal names, so the normal file storage media store might manage to Just Work. [EDIT] Incidentally MFS is exactly what I wished existed back when was kinda halfway considering using IPFS for my personal files. Been added since the last time I looked in on the project. Was too easy to "lose" files in IPFS otherwise. That sort-of fixes the problem, at least, provided the file tree can also be replicated pretty easily.
- WhatIsDukkha 6y agoHave you looked at git-annex recently? https://git-annex.branchable.com/special_remotes/ipfs/ https://git-annex.branchable.com/special_remotes/ipfs/ It takes a little bit to work out why you want it and why its so much better in that unixy way many of us insist on vs git-lfs (supporting ipfs should be a clue there). *as a warning some of the defaults in last years git-annex 7 versions were aggressively annexing files in the git archives but fixed now.