10 ms·

Databases we have today aren't designed for the kind of consumer applications we use today. They are designed for banks, Amazons and their smaller counterparts

• beagle3 6y ago
  FB, LinkedIn and Slack are exceptionally happy that it is centralized; what do you mean “shouldn’t be centralized”? Who cares about what’s better for the user? He’s not the customer anyway. Sadly.
  • jeswin 6y ago
    > Who cares about what’s better for the user? All of these apps chose technology that was available to them when they started building. So the focus could be to enable building modern usecases on decentralized data platforms. If one of them becomes the next FaceBook, personal ownership could become a baseline expectation for similar apps. Once data is owned by users, apps become a layer on top of data which can be substituted as needed. Edit: simplified.
    • lozenge 6y ago
      Decentralization has its own issues - spam filtering, lack of moderation (people often do actually want it), anonymity.
      • jeswin 6y ago
        Sure. But there are good examples of decentralization making spam difficult or impossible. One such is Secure ScuttleButt (SSB). You sync with friends and their friends (and their friends maybe, based on configuration). Now people aren't generally friends with spammers so you don't see their feeds. And if they are, a sufficient number of your friends would be blocking them - once a person is blocked by a certain number of people, stop syncing with them. I am not convinced that SSB is a solution for all types of apps, but for Social Networks it works darn well. Oh, and zero ads too because you'd never sync with ad networks.
        • amelius 6y ago
          Great idea. Perhaps we could apply it in different areas as well, such as consumer reviews?
        • aflag 6y ago
          I don't know. Facebook can spot fake profiles because of the centralisation, not sure how that'd be avoided in that system. Also, if users are really anonymous, the system will end up turning into a cesspool of disgusting and illegal stuff. At least that was the fate of all peer to peer social networks I'm aware of.
          • curryst 6y ago
            That would be handled client-side, like email. You could build in a standard set of rules, and then have some kind of ML process profiles you manually mark as spam. Just have it filter profiles in API requests through the fake profile detector. > At least that was the fate of all peer to peer social networks I'm aware of. That doesn't seem true of Mastodon. The few servers I've been on were fairly tame. I think the worst I saw was someone posted a sexualized anime-style image, but there wasn't any nudity. I didn't immediately spot anything weird on Diaspora either, but I only spent like 45 minutes on there. > Also, if users are really anonymous, the system will end up turning into a cesspool of disgusting and illegal stuff. Anonymity can be okay, but only in the presence of heavy moderation. HN is basically anonymous if you want it to be, but it stays on target because of the moderation. You can see it on Reddit too. AskHistorians is a wonderful place to be; the subreddits without even basic moderation do indeed turn into cesspools. Federation has historically made this difficult because you can only moderate your own instance, which is often a minority of the content you consume. There are potential solutions for that, though. You could allow users to "flag" posts, and add those flags to the data you federate. Then, like the fake profile system, you could filter them based on user parameters. I.e. hide posts that have at least 1 "racism" flag. If necessary to prevent brigades, you could also filter the flags that you accept based on the user that flagged the post. Maybe weight them based on similarities in accounts you follow.
    • beagle3 6y ago
      I think it’s a naïve position. Mail was decentralized. Now it’s mostly federated among 4-5 big providers. Historically, people value low cost and convenience over any other feature. Unless there is some GDPR like directive requiring decentralization and self hosting, it will likely stay centralized and become even more so.
      • wolco 6y ago
        There are so many more email providers being used worldwide. Who are the 4 or 5 big providers everyone is using? gmail, yahoo, hotmail, fastmail and protonmail? Don't forget outlook and zoho Many people still use aol or their email given by ips or work email domain. Worldwide yandex.mail is popular. Colleges still give email addresses. People like myself run their own Vanity domains are popular. Mail is still decentralized and news of its death have been greatly exaggerated
        • beagle3 6y ago
          Google, Fastmail, Microsoft(Outlook/Office365/Hotmail/Live) already cover something like 90% of one side of emails, and 50% of both sides (that is, 50% of emails originate AND terminate among these three, and 90% originate OR terminate among them). I haven't seen any recent stats, but in 2017, 50% of emails already originated or terminated with Google, one provider. The significance of "originate OR terminate" is that the provider has the ability to meddle or outright ban, either going out or going in. > work email domain. > Vanity domains are popular. > Colleges still give email addresses. Most universities, workplaces and vanity domains are running within the Google+Fastmail+Microsoft sphere, regardless of the domain name. Every university I've worked with in the last 10 years (e.g. NYU) was using a G-Suite or Microsoft hosted solution, as was every single workplace. > People like myself run their own There is a very long tail for the last 1%, but likely 99% of the users are served by 10 companies worldwide. I call that federated, not distributed. I used to run my own server as well, for ~15 years. Until one day, google stopped delivering it to gmail or g-suite addresses (sent it to spam). There was no one to talk to to figure out why. Which turned out to be ~70% of email I sent -- despite having SPF, DKIM, DMARC and everything else properly set up. Eventually, I switched to fastmail hosting because I have better things to do with my time than figure out why Google is blacklisting me. It's federated, not distributed, and we're all at the mercy of the big few.
• viraptor 6y ago
  > where a central entity (the business or corp) needs full control over the data. Or where we want some central control. Until we solve decentralised spam, I really don't want to share databases directly with other users. If you can generate a new identity yourself and then inject your own content into other databases, that's a recipe for pain. Federation kind of works, but I can't imagine a fully distributed slack for example. I'm on matrix and get the "elon musk wants to share bitcoin" images every day from telegram bridge. Even in email, spam is still not really solved. I still get both false positives and false negatives. > This also improves data security in general; there is no central place to hack. I see your point here, but would also counter with: degraded security because we need N databases to patch every issue after the issue is public, rather than 1 database (potentially) before the issue is public.
  • jeswin 6y ago
    > Even in email, spam is still not really solved. I still get both false positives and false negatives. See my comment about Secure Scuttlebutt below. Email is a little different, in that complete strangers (and apps) should be able to reach you. However, that's not the case with FaceBook, Slack, Instagram etc. If you sync exclusively with people you know (or your organization) and their connections, spam won't exist by design. What you gotta do is to model online communities similar to the physical world. In meat space, you choose the people who you are accessible to. That'll work here as well. > I see your point here, but would also counter with: degraded security because we need N databases to patch every issue after the issue is public, rather than 1 database (potentially) before the issue is public. WASM-based db engine could be a good option, if the app is browser based. We could treat it as part of the app, and the standard security considerations apply.
    • sitkack 6y ago
      SQLite running in wasi with a sync protocol, maybe like couch, or a CRDT depending on multiple writers. When I use an application on the web, it should save data into my database not the app providers.
• bergstromm466 6y ago
  > The most popular consumer apps (FB, LinkedIn, blogs, photo sharing, goal trackers, slack, basecamp etc) shouldn't be centralized in this fashion. Data(bases) should be owned by users, and synced with friends and colleagues. So instead of a single centralized database, we need a million tiny little databases which are synced with peers. > Tiny databases fit well into today's hardware capabilities landscape. We have plenty of fast storage, and powerful CPUs on computers and phones. It's time to stop building as if the server is the only place where queries can run efficiently. Truly serverless, that is. This also improves data security in general; there is no central place to hack. Sounds like you're describing the MetaCurrency Project's Holographic chain pattern that is currently being developed: https://medium.com/holochain/holochain-reinventing-applications-d2ac1e4f25ef https://medium.com/holochain/holochain-reinventing-applicati... Edit: I looked at Gitsta, have you heard of Junto? they are launching their app built on Holochain (which is itself built using Rust) very soon: https://github.com/juntofoundation/Junto https://github.com/juntofoundation/Junto + https://www.kickstarter.com/projects/1958309948/junto-a-new-breed-of-social-media https://www.kickstarter.com/projects/1958309948/junto-a-new-...
  • jeswin 6y ago
    Gitsta is work in progress; nothing to show yet, sorry. I had looked at holochain earlier, but to be honest I found their literature a bit difficult to grasp (with a quick read). But I was aware though that they're trying to solve the same problem. My view is that once you are able to reduce the amount of data an app has to deal with, you might even be able to rely on plain text files, yaml collections etc. for data storage/exchange; and fed into a db engine for faster querying. I'll look up Junto, thanks for the link.
    • bergstromm466 6y ago
      > I had looked at holochain earlier, but to be honest I found their literature a bit difficult to grasp (with a quick read). But I was aware though that they're trying to solve the same problem. I posted your feedback into the Holochain forum, and used it as an entryway into asking some questions I am grappling with: https://forum.holochain.org/t/technical-holochain-101-for-semi-noobies-like-me/4039 https://forum.holochain.org/t/technical-holochain-101-for-se... Do I have your permission to link it to our comments here? > My view is that once you are able to reduce the amount of data an app has to deal with, you might even be able to rely on plain text files, yaml collections etc. for data storage/exchange; and fed into a db engine for faster querying. If I understand you correctly then I think this is something Holochain's developers are tackling too. I think for the purpose you describe, they put a big focus on relationships and p2p actions that are 'gossip-ed' (spread) through a content addressable distributed hash table (if me mentioning this strategy seems unrelated, it shows the limitations of my technical competences). - If you're interested to engage, I'd love to know what things you found difficult to grasp in your quick read? This is a big subject with a lot of entry points, so my question is a broad question, on purpose. I'm interested in which functional use cases you are seeking/developing, and whether your critique of the current centralized architectures is similar to Arthur and Eric's critiques (the two Holochain originators), versus whether we have reached - and agree on - the same conclusions. The first one (critique) being more important to me: do we see the same challenges/shortcomings? I hope that make sense. From my understanding Holochain's main thesis is that centralized software obscures 1) an app's inner logic/workings, and 2) the relationships between peers and the data they create (as well as the shape and size of the data they generate). Through my own my research the overarching patterns I found is that corporations monopolize blueprints. Through the use of centralized black box software, they can also act as middlemen, extracting parasitic rents. They can control and manipulate us by tracking our behavior and steering us into the direction it's algorithms (and often the profit motive, e.g. advertising) have decided for us. Society's innermost services (like money, law, identity, intellectual property databases, healthcare, land registry) are often invisible or semi-invisible architectures (they require a high clearance level, or the information is stuck in old proprietary formats that make them expensive and difficult to maintain). Because these services are still centralized, they evolve very slowly, meaning that their use remains limited. A big part of my interest in Holochain comes from the work of the Mikorizal.org team (Bob and Lynn, and Pospi) being done on Open Value Network economic protocols. Using Holochain they are building a Resource Event Agent (REA) acounting model that is completely P2P. Their projects aims to provide an alternative to the closed corporate Enterprise Resouce Planning (ERP) software that is the norm today. I think this new paradigm can help us to see more clearly the actual scarcity of natural resources, as well as use it as a tool to cooperatively fund and develop open source hardware (and software) in the open. Edit: updated for clarity.
• amelius 6y ago
  Well, some projects use Software Transactional Memory, which shows that some database concepts are already used in normal applications.
  • faichai 6y ago
    This is a way to do concurrency without explicit locks. It doesn’t help directly with a decentralised database.
• arethuza 6y ago
  "tiny little databases which are synced with peers" Wasn't that the idea behind Lotus Notes?
  • josephg 6y ago
    Yes! There’s a lot of good ideas and features in lotus notes that I want, and despite it being 2020 I can’t seem to get for any price. There’s reasons why Notes failed, but the data model was way ahead of its time and I think it’s past time we brought some of that innovation back.
    • pessimizer 6y ago
      Isn't that what CouchDB was doing?
• willvarfar 6y ago
  If you are syncing your databases with friends and colleagues, then there are multiple places to hack.
• Quarrelsome 6y ago
  so I spill hot coffee on my laptop and brick it I lose my entire Facebook history? Not seeing it really. Centralised _is_ what consumers want, without an generalised and public IT infrastructure for users to handle backups, which we don't have.
  • jeswin 6y ago
    But you don't lose all your code if you spill coffee on it. Similar mechanisms could work here as well. > generalised and public IT infrastructure You don't need public infrastructure for this. There could be free and paid options (similar to Dropbox) to hold your data securely in encrypted form.
    • Quarrelsome 6y ago
      I mean public in so far as it just happens without you thinking about it, that could be commercial too, its just more pervasive as public. When such infrastructure becomes a _guaranteed_ aspect of a given operating system. But until then people implicitly _want_ centralised.
      • josephg 6y ago
        One healthy middle ground here is the situation with git. Git is fundamentally decentralized, but I can use it in a centralised way if I want to via services like github, gitlab, bitbucket, etc. Those services allow me to collaborate with other users and back up my projects. And because it’s decentralised, I can trivially move / clone my repository to other providers (or to my own servers) without losing any data. (... Well, except for github issues; which arguably should be stored in repository itself.) I can imagine a similar situation for a small database too. Live clone to Dropbox or something similar. The user gets all the benefits of a centralised system but stays in control, and they can switch providers whenever they want.
        • Quarrelsome 6y ago
          ye but you know in practice that means at least 95% of users are on the centralised version. We continue to build up this 5% of technocracy that have different rules applied to them because they know enough to care.
• deepstack 6y ago
  mnesia seems like what you are looking for.
  • lioeters 6y ago
    Fascinating! To save others a search: Mnesia - "A heavy duty real-time distributed database" (written in Erlang) Git repo - https://github.com/erlang-labs/mnesia https://github.com/erlang-labs/mnesia Documentation - http://erlang.org/doc/apps/mnesia/Mnesia_overview.html http://erlang.org/doc/apps/mnesia/Mnesia_overview.html
    • deepstack 6y ago
      Yeah, it is being around forever. Erisson or some Nordic company use it for phone switches. It is design for concurrency. Personally I think everything pretty much has being invented, it is all about people looking for it and bring into fashion.
      • lioeters 6y ago
        I see, Erlang makes perfect sense for the purpose. I think I did read about Mnesia once before, but I must have forgotten about it. ;)
• edwintorok 6y ago
  I think it is a common misconception that banks use database transactions the old fashioned way. They seem to be more like eventual consistency. I initiate a transfer and there is some check about available balance (although with various overdrafts, etc. it is unclear how strict), then it is marked pending and eventually the transaction succeeds or fails days later (an international transaction).
  • Hallucinaut 6y ago
    This is conflating things. At a database level it is absolutely ACID compliant, full transactions are used for situations like you mention within one bank. The fact there may be interplay between custodian banks and various triggered events has nothing to do with the type of database consistency features of an RDBMS.
• viach 6y ago
  > The most popular consumer apps (FB, LinkedIn, blogs, photo sharing, goal trackers, slack, basecamp etc) shouldn't be centralized in this fashion The evil CAP theorem enters the town here. Consumers want things to work fast, it doesn't fit the decentralized approach. > Databases we have today aren't designed for the kind of consumer applications we use today Consumers don't really care what kind of database do you use. They want to open the page and click the button, quick, now.
  • pdimitar 6y ago
    You are correct but part of this can be alleviated by aggressive caching by the client devices and delayed (optimistic) updates -- to a large extent though. Doesn't apply to all problems in this space.
  • jeswin 6y ago
    > The evil CAP theorem enters the town here. Consumers want things to work fast, it doesn't fit the decentralized approach. I think we have to distinguish 'decentralized' from 'distributed' in this case. In a decentralized approach we reduce (by many orders of magnitude) the size of data we're dealing with. Locally installed/browser-based applications only have to deal with data that's created by, or directed at that user (and locally stored). Add: For a number of applications, hard consistency guarantees are overkill. You could totally live with eventual consistency (some don't even need that). In fact, other than e-com and banking I can't think of any of my daily-use apps needing such guarantees. And indeed, decentralized data storage is a bad fit for banking and commerce.
    • eternalban 6y ago
      Network hops cost time regardless of payload size. Distributed consensus (which is what is required for a “decentralized” system) requires interaction between distinct nodes. You should note most consensus algorithm designs start with sharing a single ‘register’ (or ‘value’). Most banks were doing “decentralized” relaxed consensus from the beginning of “banking”. Local office accepts your deposit of your check. “Funds available next day”. Because by “next day” they have confirmation from counter party for the funds.
• macspoofing 6y ago
  >Databases we have today aren't designed for the kind of consumer applications we use today. Yes they are? Relational databases are very good for consumer applications with relational data. NoSQL databases are good for everything else. >Data(bases) should be owned by users, and synced with friends and colleagues. Why? You use words like should - why SHOULD it be that way? Users are able to create / buy / rent infrastructure to manage their own data - and yet they don't ... why? >Shouldn't be centralized in this fashion. Data(bases) should be owned by users, and synced with friends and colleagues. So instead of a single centralized database, we need a million tiny little databases which are synced with peers. Nice in theory (maybe). The problem is that centralization provides enormous quality and performance metrics. Distributed systems are resilient, but slow, and require a lot of care to maintain consistency and integrity of the network. Take email as an example. It took us 20 years to get email spam somewhat under control and a major part of that was creating a centralized infrastructure (from using centralized email providers to third party whitelists and blacklists). It's gotten to the point where hosting your own mail server is a huge hassle and pretty much impossible for regular people. I spent about a week trying to figure out why our Office365-originating mail wasn't being received by one of our customers. It turned out, their mail provider matched a particular phrase in our automatically-inserted disclaimer and completely rejected the mail. Decentralized systems also make controlling your data all but impossible. If I want to remove my data from Facebook, that's the only entity that I need to deal with and it's relatively easy for regulators to set sound policies in place. In your distributed system, once your data leaves your node, you've lost all control all of it forever.
  • hansvm 6y ago
    > Decentralized systems also make controlling your data all but impossible. If I want to remove my data from Facebook, that's the only entity that I need to deal with and it's relatively easy for regulators to set sound policies in place. In your distributed system, once your data leaves your node, you've lost all control all of it forever. That's already the case though. When I ask Facebook to delete my data they delete their copies of it. However, anyone who has downloaded photos, any advertisers who have scraped that data, any archival attempts, any search results, and a whole slew of other partial copies of that data still exist. It's almost better for data to be explicitly distributed to make it more obvious that it can't be easily removed.
• ccleve 6y ago
  Most of the responses to your post are critical, but I couldn't agree with you more. Having a local database would solve all kinds of performance, connectivity and privacy problems. Transparent background synchronization would be beautiful. Unfortunately, this is a really hard problem to solve. Sync is a bear of a problem in the face of conflicts. An immutable datastore with a log file of all mutations would help a lot, but you'll still have to manage the conflicts. This is not only a back end problem. What kind of user interface do you present that actually makes sense to an untrained user? "Bill changed Mary's name to Marybeth, but Fred changed it to Mary Beth. Which is correct?" Most users will say "Whaaaa? Why are you bothering me?"
  • deleted 6y ago
    [deleted]
• jhardy54 6y ago
  Are you familiar with Scuttlebutt? It's exactly what you're describing.
• RandoHolmes 6y ago
  I actually feel like Fossil could be adapted to do this fairly easily. It uses Sqlite (it's made by the author of sqlite) so it has the same downsides described in the article, but if you could solve those problems I think it does pretty much exactly what you and the author are describing. https://fossil-scm.org/home/doc/trunk/www/index.wiki https://fossil-scm.org/home/doc/trunk/www/index.wiki
• AtlasBarfed 6y ago
  Sync between friends? That will basically require standards hell, and the Standards+1 problem XKCD made famous.
• dandanua 6y ago
  Look at the hyper-stack https://hypercore-protocol.org/ https://hypercore-protocol.org/
• aabbcc1241 6y ago
  The approach adopted by zeronet maybe what you're looking for that allows peers to synchronize their data in master-less manner.