8 ms·
LF: Fully Decentralized Fully Replicated Key/Value Store
- beamatronic 7y agoIn your readme, could you add a mention of memcached and how you differ?
- thdxr 7y agoThis is decentralized as in peer-to-peer, not in the same category as memcached
- marknadal 7y agoAfter just giving scathing remarks to another database system, I'm pleasantly surprised by LF! LF looks like a well-reasoned project with a solid direction that is unique and applicable. I do disagree with some of its design choices (DAGs aren't very scalable, full replication versus partial/sharded, etc., and GPL instead of MIT - which my decentralized DB, http://gun.eco http://gun.eco is MIT/Apache2), but for the problem it is trying to solve, it seem like you've made the right choices for LF and its use cases. Congrats and keep it up! Excited to hear about this over time. :)
- heyoni 7y agoOh hi Mark!
- twokei 7y agoInteresting - would love your thoughts, why is it that DAGs aren't very scalable?
- marknadal 7y agoDAGs aren't very scalable, because like with git, you have to store the whole history. In some scenarios, you can rebase/snapshot to clean up history, but these usually require a type of centralization or consensus, which defeats the point of using something like git. As a result, DAGs can only be used to replace a subset of apps/tools out there. The most important/used apps, though, are indexed lists. Things like: - Google rankings - Reddit homepages - AirBnB listings - Ubers nearby If you're updating a geo-index 100s of times a second, like in the case of cars' GPS locations, then you're just wasting resources with a DAG that you'll never use and wind up bottlenecking the system, preventing scaling beyond a certain threshold. I've dealt with this in practice, and it was no fun. We switched off DAGs, and our biggest production deployment now has been with 15 million monthly users. This is far far far beyond the scale of any of the previous systems.
- tasubotadas 7y agoI think that as it currently stands, the GPL3 licence makes the project unusable. There aren't any externally packaged drivers with a different licence, and you can't include their native code in your commercial projects without having to share the code due to the inclusion of the GPL3 licenced code.
- api 7y agoWe are considering a license more like that adopted by CockroachDB for all our projects in the future. Haven't decided on the particulars yet.
- olah_1 7y agoReally strange that this reply is greyed out. I specifically came to the comments hoping that someone compared this project to Gun. Who better than Mark?
- godzillabrennus 7y agoDistributed computing technologies are fascinating and evolving quickly. Caught a demo of a blockchain product performing at over 6,000 tps on a single shard spread over about a dozen geographic regions on a test net last week. Expected to be more performant than anything out there today when at scale.
- RL_Quine 7y agoWhenever "blockchain" is mentioned there's always exactly these sort of comments popping up. They are always demos because verifying even 6000 signatures in a second on a normal CPU is not a walk in the park, let alone any other computation, network latency. The concept is as implausible as its real world applicability. # openssl speed ecdsa Doing 256 bits sign ecdsa's for 10s: 494208 256 bits ECDSA signs in 10.00s Doing 256 bits verify ecdsa's for 10s: 161277 256 bits ECDSA verify in 10.00s
- rolltiide 7y agoI think it is more interesting that enthusiasts still fall for the "testnet tps" sales pitch
- DINKDINK 7y ago>Proof of work is not foolproof, especially in LF where there is no intrinsic economic mechanism to incentivize the runaway growth of PoW "mining" investment. As a result a well financed or determined attacker willing to throw a lot of compute power at the problem could overcome the PoW "weight" of the records beneath a target record in the LF DAG and replace it. This issue -- of creating equivalent proofs of work -- is what I was most interested to see if there was any novel art (which it appears there isn't). Bitcoin's difficulty adjustment is the only well-functioning algo that deals with the economic coordination issue around how to -- without any third party coordinator -- equate the value of a proof of work made by computer's of differing computational capacities (a phone, an ASIC). All PoW consumed by Bitcoin has marginally positive contribution to the system. Sacrificial PoW (as in LF, hashcash) only has value if reorgs happen. It also results in a system with tenuous game stability properties ("When will it be uneconomical for an attacker to reorged these records out of the DaG?" "Can't say, blocks' PoW requirement is arbitrary") vs bitcoin ("When will this 50 BTC txn be uneconomical for the most adversarial miner to reorg out?" "well there's about 14 BTC of block reward per block so about 4 blocks ~40-60 minutes"
- synctext 7y agoFascinating project. It is the first time I've seen proof-of-work combined with DAG and real world engineering details such as PulseToken, oracle nodes, varient on S/KEY one-time password scheme, etc. > adamierymenko 478 commits 403,913 ++ 148,611 -- Impressive productivity.. Worried about the Sybil attack. The documentation uses the "foolproof" wording, but the overlay might not be attack-resilient and creating a million fake identities would be sufficient to dominate the system. Even a loose connection to the main DAG might be sufficient to hurt LF. > Oracle nodes document low reputation records in commentary records. When a node sees a commentary record it extracts this information and caches it internally in an indexed table. That could be an attack vector. Broadcasting negative information can often be exploited in my experience. Slandering attack with a 100 Mbps VPS might cripple the network. Could this be altered into positive re-enforcement loop? > Casual users can choose to trust nodes that are generally trusted. How will central server "lf.zerotier.com" handle copyright claims by Hollywood (people will try to upload also large files)? (disclaimer, creator of Tribler, we're designing systems like this within our university lab for 14 years and 8 months)
- mattupstate 7y agoZeroTier tech is so promising, but it appears Adam is trying to do too much at once. For example, I never received the ZeroTier Edge box, or a refund, from the IndieGoGo campaign. It's frustrating, but I hope the ZeroTier succeeds in the long run.
- jbverschoor 7y agoYup.. no focus. The product is really good. Could use a better interface. But he should be working on getting it into people's hands, instead of creating new tech. Seeing this leads me to believe that zerotier wants to do this whole global network thing. This is not what I want from zt. This probably came from questions like "how can we trust you as a network controller". And the answer is to create some tech. But in that case, why not simply offer an AMI on aws for a controller? All assumptions of course. Upon further checking: From https://news.ycombinator.com/item?id=20329060 https://news.ycombinator.com/item?id=20329060 and https://www.zerotier.com/lf-announcement/ https://www.zerotier.com/lf-announcement/ : > We are going to use it to fully decentralize ZeroTier and for some specific enterprise customer projects, but others can use it for other stuff. > Our interest in decentralizing the roots wasn’t just philosophical. Over the years we had numerous requests by policy-bound or very (justifiably) paranoid customers to run their own infrastructure that could be logically separated from ours as much as possible I dunno. I'll have to see what it turns into. But it surely explains the silence and lack of updates on zt/zt-one
- api 7y agoLF is part of a return to focus: global scale network coordination and decentralization. Making a hardware box was the big defocusing project and took more time (with less results) than this. Lesson learned: avoid hardware if you are not a hardware company in your DNA, and hardware will always take 10-20X longer to ship than to prototype. So yeah, defocus is a legit criticism but not because of this project. This was also a two birds with one stone project in that it is also useful on networks with arbitrary connectivity graphs and unreliable connectivity. An enterprise customer of ours funded some of this R&D for that purpose and we combined their objectives with our root server decentralization and network robustness improvement goals. I wish I could talk about the enterprise customer's project. It's quite cool but we're under NDA. All I can say is that it's industrial and involves mobile ad-hoc networks. There is admittedly quite a lot behind this that is not apparent from the README, and from the comments here its clear that people have trouble understanding this because it is not yet another cryptocurrency and isn't solving exactly the same problems. (There is some overlap but this is different.) We built this to solve a problem nothing else solves, but I figure it will take the appearance of use cases for people to get what that problem is. The people bringing up etcd and consul are closer than the people bringing up coins.
- viraptor 7y agoI'm having an issue seeing why would I use this. I mean, given the high resource and bandwidth usage, why would I opt for this rather than a zerotier'd network of consul/etcd/zookeeper/something distributed on a number of tiny instances between many regions? It also looks like it's write-once which makes usage non trivial. Also it gives you the usual blockchain fun of: now you're hosting whatever you're given, have fun defending yourself if you ever get sued. So... What's the realistic use case for this? (In both public and private network variation) Zerotier's network seems like one of the very few projects where this makes sense. (From the tech point of view, this looks really cool though!)
- api 7y agoThis is for systems that cross trust boundaries. Etcd or consul would be a better fit for systems controlled by one entity. The only exception might be where the network is unreliable or transient. Think of it (as the readme says) as consul for decentralized open systems.
- joatmon-snoo 7y agoWhat's the mitigation for OOMs triggered by someone dumping records into the system? Not someone trying to overwrite existing records, just constantly inserting new records.
- api 7y agoThis is mitigated via a proof of work "hashcash" mechanism, or for signed owners the ability of the CA to revoke the cert. BTW "high resource use" is in reference to things like tiny embedded devices. You won't be putting this in a fridge magnet or a smart light bulb. This will run fine on a $10/month VPS or a Raspberry Pi 3 or 4, though if the data set gets huge you might need to attach some storage.
- Proven 7y agoI can't see any mainstream use cases for this, - the same problem as Bitcoin. At least the network component has some use as personal or P2P VPN but even that is a niche use case.
- panarky 7y agoCould LF be used to replace the SKS keyserver network that's hopelessly borked? https://gist.github.com/rjhansen/67ab921ffb4084c865b3618d6955275f https://gist.github.com/rjhansen/67ab921ffb4084c865b3618d695...
- api 7y agoYes
- deleted 7y ago[deleted]
- conradludgate 7y ago"Doing proof of work for all those is possible but costly, so we stuffed a certificate into Sol's configuration to let us cheat and be special. Think of it as our "fee" for creating and maintaining LF and donating it as open source to the community." Not sure how I feel about this, and I'm sure someone's going to figure out how to remove that 'feature' in a fork... Does kinda seem against the nature or p2p and blockchain
- api 7y agoYou can create your own database/network easily. There is even a command for it. This is just the default.
- winrid 7y agoBriefly reading the code, it seems like a distributed wrapper around sqlite.
- silasdavis 7y agoSee also: https://github.com/davebryson/bftdb https://github.com/davebryson/bftdb
- misterdata 7y agoSee https://github.com/pixelspark/catena https://github.com/pixelspark/catena for a SQLite-on-blockchain implementation
- api 7y agoWe found stuff like this while researching before creating LF. I remember finding this exact project and a few others like it. This is in the ballpark but has numerous issues in a federated or fully decentralized use case, such as who sets database schema and what happens when you need to change it. It also lacks any privacy protections. All keys and values are visible to everyone all the time. That opens up a ton of attack surface that has to be specifically mitigated on a case-by-case basis. Better to have data be blind. LF isn't a wrapper around SQLite at all. It just uses SQLite for meta-data and index information because it happens to be a very capable database and because the ability to do relational queries saves a ton of time vs. hand-rolling such queries on top of something like LevelDB. The last issue is something I've never understood about cryptocurrency code bases. Why do they insist on implementing a slow ad-hoc relational model on top of a simple KV store when they could just use SQLite? I assume it's because they assumed SQLite would be slow, but it's really not unless you are doing massive numbers of writes.
- mirekrusin 7y agoIt would be great if the author (or some good soul) explicitly stated that this project doesn't violate any hashgraph patents as at first sight it's not clear.
- sschueller 7y agoWouldn't that fall under software patent which is not something that can be patented Europe?
- mirekrusin 7y agoI don't know, it's one of those things that you read more and know less, ie. first google result for "software patents in europe" says "yes, but no": >> "The European Patent Convention states that software is not patentable. But laws are always interpreted by courts, and in this case interpretations of the law differ. So the European Patents Office (EPO) grants software patents by declaring them as "computer implemented inventions"." ...but it's interesting, ie. what would it mean exactly if bitcoin was built on something patented? It's completely distributed so nobody to sue. Would the repo be banned on github? Let's say it would and the development would move to linux kernel like development, then what? US based exchanges would probably have some premium fee and that's it?
- viach 7y agoGood marketing move - building a distributed ledger not on blockchain and without tokens, positioning it as a key-value store. Kudos.
- api 7y agoIt wasn't built to be a cryptocurrency but to serve as a data store for a decentralized network backplane. You could theoretically build a token on top of it I suppose.
- viach 7y agoWe built a similar thing using Tendermint. Not open source yet but already have several clients. Good to see other people appreciate similar move - a tokenless DLT.
- pcr910303 7y agoWow, I was waiting for a global ledger/consensus system based on DAGs/non-blockchains forever... Blockchains have an inevitable scaling problem. I was only aware of iota & Radixdlt but I’m pleasantly surprised that there’s lf now! Kudos for the great work!
- eigenvalue 7y agoThese CS sayings (linked to on the github page for LF) gave me a chuckle: From https://www.martinfowler.com/bliki/TwoHardThings.html https://www.martinfowler.com/bliki/TwoHardThings.html : "There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton Long a favorite saying of mine, one for which I couldn't find a satisfactory URL. Like many good phrase, it's had a host of riffs on it. A couple of them I feel are worth adding to the page There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors. -- Leon Bambrick There are only two hard problems in distributed systems: 2. Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery -- Mathias Verraes there's two hard problems in computer science: we only have one joke and it's not funny. -- Phillip Scott Bowden"
- draw_down 7y agoBoy, that last quote couldn’t possibly sum up my feelings any better.
- e12e 7y ago"Unlike web SSL servers there is no need to send a certificate back to the owner. LF is a global shared data store and therefore makes use of itself to publish certificates." Nice! Thinking a bit about this - this sounds like it has real potential vis-a-vis traditional CA. With a (many) globally trusted db - it sounds quite feasible for eg Mozilla or Debian to simply store (or "bless"/trust) any number of root certs - and record that in lf. And the same could be done for signing binaries (whether via x509 or some other scheme - "all" that is needed is some way to "discover" trust of the signing key after all - be that a gnupg key or a x509 cert etc).