7 ms·
How we put IPFS in Brave (2021)
- ChrisClark 4y agoThe actual title is "How we put IPFS in Brave" and that is a lot more interesting.
- aftbit 4y agoI've had extremely bad luck running go-ipfs at any scale. GC is braindead (literally deletes the whole cache on a timer), networking is slow and drops tons of packets (apparently too much UDP?), and by default it stores each object 2 or 3 times. I'm sure it'll work fine for people using http://dweb.app http://dweb.app, and probably go-ipfs will work okay for super casual browsing, but as soon as someone tries to download any substantial IPFS dataset, expect lots of resource limits.
- kevincox 4y agoYup, I had a (tiny, spinning rust) home server that was slowed to nearly a halt. SSH logins would take minutes even when limiting IPFS to 2GiB of the 16GiB of RAM. Stopped go-ipfs and it was instantly snappy again. My impression of the IPFS project is that the goals are excellent, the core protocol is quite good however they like rewriting the higher level layers far too frequently (for example they have deprecated UnixFS which seems to be the most used format and they keep switching between JSON, Protocol Buffers and CBOR) and go-ipfs seems to be a pretty garbage codebase.
- hecturchi 4y agoAny chance that, by limiting RAM usage, you forced your application to heavily swap, clogging the disk and making your machine slow? I have run a public gateway on 2GB of RAM. Later 4GB because it was subject to very heavy abuse, but it was perfectly possible. Perhaps it is a matter of knowing how to configure things and how to not inflict self-pain with wrong settings.
- kevincox 4y agoYes, there is definitely a chance but it was wayyyy worse when I gave it more or unlimited ram. At least this way the machine was operational most of the time. I don't think it was swapping but since I think the limit I applied also affected the page cache it was likely reading it's data from disk a lot more often than it would of if it could own the whole page cache. But this is basically the same effect as swapping. Maybe there is a Goldilocks value I could find, but I didn't really need IPFS running that much so I just removed it.
- thomashop 4y agoSame experience here. It's a real shame. I have the feeling IPFS is trying to do too much and became a bit of a bloated mess. I love the idea of decentralized content-addressed storage and wish there were a more lightweight way to get there.
- sph 4y agoIt was slow and buggy when it first released; understandable so I waited a few years, tried again recently now that it's popular and it still is an unpractical proof of concept.
- stavros 4y agoYep, I made an IPFS pinning service (the second one to exist, IIRC, and the first usable one), and I wish I hadn't. It's a bit of a trash fire.
- ushakov 4y agoIPFS is especially heavy on bandwith if you plan to host IPFS at home and meanwhile do things on the internet then IPFS isn’t for you although it’d be a good excuse to upgrade your home network
- salmonlogs 4y agoThat seems like an insane usability trade off that would limit adoption quite heavily
- tamrix 4y agoThere's different profiles you can select from. You might have the server profile enabled. Powers save probably consumes the least abs you can opt out of sharing altogether. But the entire point is p2p.
- musicale 4y ago> if you plan to host IPFS at home and meanwhile do things on the internet then IPFS isn’t for you Is it possible to limit the bandwidth and queue depth for IPFS? I bet you could also lower its limit dynamically whenever web traffic is seem.
- ushakov 4y agonope, the only real possibility (to lower bandwith usage) is to disable peering without p2p IPFS is nearly useless
- musicale 4y agoSo you're saying it's somehow impossible to deploy QoS management in your network to limit IPFS the same way you would limit anything else? Presumably one could also run IPFS (or Brave) in its own VM, container, or hardware server and to rate limit traffic in and out of it.
- ushakov 4y agoIPFS content loads slow enough without rate limiting rate-limiting will only make the matters worse and again turn IPFS nearly useless they haven't managed to solve these issues for 6 years now
- zcw100 4y agoIt does not delete the whole cache on a timer. It will delete blocks that are not pinned periodically or when it reaches a high-water mark. It does not store each object 2 or 3 times. First, it doesn't refer to anything as an object but rather blocks and a block is only stored once. It will only be replicated if you're running a cluster in which case replication is the point.
- resonious 4y agoI don't really know anything about the Golang GC, but I would not be surprised if the process of scanning for unpinned blocks results in a lot of memory accesses. If too many useful things get evicted from the cache during that process, then I can see why GP is saying it deletes the whole cache.
- RcrdBrt 4y agoGolang has only 1 tunable GC parameter by design so it could be not opaque enough for certain loads but I learned that putting a ballast in ram fixes the too frequent GC sweeps https://eng.uber.com/how-we-saved-70k-cores-across-30-mission-critical-services/ https://eng.uber.com/how-we-saved-70k-cores-across-30-missio...
- zcw100 4y agoWhy would you ever start anything with, "I don't really know anything about the Golang GC but..." IPFS GC is separate from the Golang GC. IPFS GC deletes blocks on the local node that are not pinned. I'm not sure what you mean by "too many useful things". If it's not pinned it's evicted.
- deleted 4y ago[deleted]
- resonious 4y agoHahaha oh wow. How embarrassing. I thought the original comment was talking about Golang's GC, as they did specifically mention go-ipfs. I suppose that's your answer! Simple misunderstanding.
- TeeMassive 4y agoPlug and play implementations like this is really how decentralized tech should be done and promoted.
- jeroenhd 4y agoIt's a little sad that Firefox isn't the first mobile browserto receive and experiment with new tech like IPFS. I do wonder if they have solved the privacy issues with IPFS before they put it into Brave. IPFS is probably the best contender for Web3 right now and I hope it'll see more use before the crypto bros take over the term completely
- skrowl 4y agoFirefox has worked with IPFS since early 2019 https://addons.mozilla.org/en-US/firefox/addon/ipfs-companion/ https://addons.mozilla.org/en-US/firefox/addon/ipfs-companio... Brave just took the step of forcing you to always have the extension installed instead of making it optional, basically I've tried IPFS a time or two and always found it to be INCREDIBLY slow (even worse than tor) with ZERO content discoverability.
- dleslie 4y agoThis still requires external software to operate, and isn't available on mobile. It's effectively dead in the water by not being available to use without additional configuration, by default. I'd argue this is worse than doing nothing. This gave Firefox the ability to say they care, and yet not deliver something meaningful.
- spicybright 4y agoI agree. It just leads to "Oh, IPFS? I tried that years ago, it was terrible. I don't recommend trying it." What do you think firefox could have done to improve things? And, as someone not well versed, is there any "killer demo" that uses IPFS currently?
- TheaomBen 4y agoI'm working on a collaborative photogrammetry solution (think async/distributed 3d mapping from overlapping pictures) that shares data via IPFS. Flattering myself heavily, I believe this sort of public-data consuming application fits like nothing else.
- trhr 4y ago
- deleted 4y ago[deleted]
- baggiponte 4y agoI don’t understand what this implies/enables O.o could somebody be kind enough to explain? Thanks!
- k__ 4y agoYou can use ipfs protocol instead of http. It will start a local ipfs node in the background. Which is basically a local webserver.
- eminence32 4y agoSpecifically, it's a local process that participates in the global IPFS p2p network, and also exposes content via a local web server
- igorkraw 4y agoSounds like fun in jurisdictions where seeding a torrent can get you a copyright lawsuit like my native Germany
- zcw100 4y agoThis is a common misperception of IPFS. IPFS does not push any content onto your node or force you to host random content. It will only host content you have added or requested to be added to your node.
- SpelingBeeChamp 4y agoWhy should I (or anyone else) care about IPFS? It's not like I have trouble storing and retrieving data currently. Moreover, every anecdote in this topic seems to be from people who found IPFS unusable in reality.
- k__ 4y agoIPFS uses the content's hash as address. This means, there are no needless duplicates and if an IPFS node in your local network holds the data, you don't have to go to the source.
- eminence32 4y ago(2021)
- deleted 4y ago[deleted]
- EGreg 4y agoWhat I wonder is, how to handle cross-domain security in browsers when you have IPFS? Is there any standard for this yet?
- hinkley 4y agoI've been wondering this as well. In IPFS isn't -everything- cross-domain by definition? Each publication of a document would have a new identity and I'm not sure how you'd shoehorn that into something like CORS. You'd probably need something interactive, akin to OSCP vs CRLs.
- anonymousnotme 4y agoI used to be very gung ho on IPFS, until I learned that the content ID does not depend solely on the content of the file. When one puts a file into the system one can choose different hashing algorithms, which will cause the content ID to be different obviously. However, even when using the same algorithm the content ID will change depending on how it is chunked. I would expect any sane system to consistently produce the same hash/content ID a file. I can see if the system is moving from using SHA2 to SHA3 that it could be stored twice. Don't know whether they have changed things so that a consistent content ID will be produced or not.
- siwatanejo 4y agoWow good to know!
- cle 4y agoHow would you produce the same hash for different encodings of data?
- anonymousnotme 4y agonot sure that I follow what you are asking. I would expect if sha2-256 is used then the content ID would be the same. However, depending on how the content is chunked, the content ID will change. Two disadvantages that I see: 1. if new packages are produced for a release of open source, could I see if there is a copy available via IPFS? No, because one can't predict how it would be chunked. So, one would have to download and then derive a content ID and one can only tell if it is available if the same chunking algorithm is available. 2. if I want to push a package or other binary, can I figure out if it is already available via IPFS? No, one can't.
- hecturchi 4y agoThe content ID is not the hash of the content, it is the hash of the root of the Merkle DAG that carries the content. Doing it like that has many advantages, like being able to verify hashes as small blocks are downloaded and not after downloading a huge file. Being able to de-duplicate data, being able to represent files, folders and any type of linked content-addressed data structure. As long as your content is under 4MiB you can opt out of all this and have a content ID that is exactly the hash of the content.
- im3w1l 4y agoSo are there any interesting ipfs websites?
- samsquire 4y agoDoes anyone understand merkle clock CRDTs? How do you handle conflicts where two concurrent events occur at the same time? Who wins? I know timestamps are not reliable but I want last write wins behaviour and seamless merge. The paper leaves data layer conflict resolution to the reader. It does suggest sorting by CID. I added a timestamp field for conflict resolution. After reading Merkle-DAGs meet CRDTs whitepaper I took a go to implement a MerkleClock. It's incomplete. I need to maintain the partial order of "occurs before". https://github.com/samsquire/merkle-crdt https://github.com/samsquire/merkle-crdt I also implemented part of the YATA algorithm yesterday. So I think I could merge the plain text merging functionality of that with the Merkle CRDT. https://github.com/samsquire/yata https://github.com/samsquire/yata
- hecturchi 4y agoIn https://github.com/ipfs/go-ds-crdt https://github.com/ipfs/go-ds-crdt, every node in the Merkle DAG has a "Priority" field. When adding a new head, this is set to (maximum of the priorities of the children)+1. Thus, this priority represents the current depth (or height) of the DAG at each node. It is sort of a timestamp and you could use a timestamp, or whatever helps you sort. In the case of concurrent writes, the write with highest priority wins. If we have concurrent writes of same priority, then things are sorted by CID. The idea here is that in general, a node that is lagging behind or not syncing would have a dag with less depth, therefore its writes would have less priority when they conflict with writes from others that have built deeper DAGs. But this is after all an implementation choice, and the fact that a DAG is deeper does not mean that the last write on a key happened "later".
- samsquire 4y agoThank you for answering this. I thought of using user indexes by order of connection or user last active time but if you're not worried of the security implications of wall clock and time skew. Hyperhyperspace project has a previous field on the CRDT operations and can issue undo events to reverse operations. I suspect you could have a time validator service that is a node that issues revocations if times are in the future. It wouldn't be on the read or write path and it's more to validate that times aren't in the future.