14 ms·

Scuttlebot: Peer-to-peer database, identity provider, and messaging system

361 points by pcr910303 7y ago

SupriseAnxiety 7y ago
THE SCUTTLEBUG HELPED MARIO BEAT FOX
z0mbie42 7y ago
I recently discovered SSB and was really intrigued. I would love to hear an experience report from actual users like what is working great, and what is bad.
- jekrb 7y ago
  It's been a while, but I used to be an active SSB user. I hosted SSB pubs and used to post on patchwork semi-regularly. I thought it worked pretty well as a social network. I discovered new and interesting ideas from folks that I don't see much on mainstream social media. I haven't followed the space much recently, and I'm curious about how it has evolved over the last year or so. My favorite memories on SSB: Someone promoted a book that they had written, and we arranged for a sales transaction by talking purely over the network. I sent them some amount of Bitcoin, and they sent me the PDF of their book. It felt very personal to work with the author directly, and side-step payment processors. I loved taking my laptop out on the train or to a coffee shop, and replying to threads and publishing a post to SSB while offline. Something about reading other peoples ideas while disconnected, and then writing my thoughts, and having them automatically sync to the network when I got back on my WiFi at home, gave me a different perspective on ways to use technology.
  - z0mbie42 7y ago
    Thank you! The lack of multi-device support wasn't too much cumbersome?
    - jekrb 7y ago
      The lack of multi-device support was a constraint that gave me a different perspective of interactions on the web. My ssb keypair was on a work laptop, so when I changed jobs and had to give my laptop back, I lost my keypair. Now, I could have exported the keypair and continued to use my "account" on my new laptop. The network would have synced on my new device, and I'd get all my posts and pictures back. But decided to embrace the constraint instead. When I rejoin the network, I'll have a new keypair, and no post history. I think this can have an interesting effect on how we view our attachment to data.
      - rakoo 7y ago
        Creating a new account from scratch also means rotating your keys, which is a good practice. As the last few discussions on PGP have shown, the model of having a long-term identity key is more dangerous than it seems, because a single mistake (by you or by the application developer) means so much content can now be leaked. It's probably easier to let the natural connection between people be the vector of long-term trust, which ironically SSB emphasizes on
  - zozbot234 7y ago
    > I loved taking my laptop out on the train or to a coffee shop, and replying to threads and publishing a post to SSB while offline. Something about reading other peoples ideas while disconnected, and then writing my thoughts, and having them automatically sync to the network when I got back on my WiFi at home, gave me a different perspective on ways to use technology. You can do this with Usenet and most BBS's. Most native IM apps will also do this for you, plus of course there's email.
    - capableweb 7y ago
      One cool feature of Scuttlebot is that if you and your friend are already following each other, you only need a connection to each other P2P to be able to send messages to each other. So if you're on a train with ad-hoc WiFi connected to each other, you can still proceed as usual and sync stuff. I don't think this feature exists in Usenet and BBS's where there is a central server who masterminds the sync that everyone is doing. Same with email, requires a server (local or remote) to send/receive stuff while in SSB both local and remote are usually the same machine.
      - tlavoie 7y ago
        For BBSs, you're correct. Usenet (and email) used UUCP, which is actually much closer in concept I think here. UUCP is a store-and-forward mechanism, not dependent on a real-time connection to a particular server. I used to run a node, connected to a guy I'd met who worked for an ISP. He had, gasp, a full-time network connection via ISDN; pretty magical in these days of dial up. So, Usenet feeds were configured on my own little system, essentially subscribing to the newsgroups I wanted. Periodically, it would dial out to the other gent, upload any new posts from me, and download anything new on those newsgroups. My email came and went the same way. Naturally, what I got was a subset of what he had accessible. While I never used this functionality, I could have had others call up to me, and I would just be an intermediate link in the chain. RFC 976 (https://tools.ietf.org/html/rfc976 https://tools.ietf.org/html/rfc976) describes how this works for email, including SMTP over UUCP.
        7y ago
- oosh0Jai 7y ago
  I've been using ssb for about a year and a half, and love it. The ideas behind the dweb do not get the credit they deserve, IMO : p2p data sharing where your data is sent directly to your friends rather than centralized on a know-it-all server is what the world needs, for anyone concerned about privacy. If I had to say what I don't like about ssb, it would probably be that it's not easy to write an application for it. After trying for a while, I turned to Dat and the Beaker browser, which let me write frontend applications the usual way, with just an api to manage the p2p archives. Both dat and ssb are javascript projects, so if you want to write your dweb applications using an other language, you're out of luck. I heard there is a rust implementation of dat being worked on that will allow to access an ABI, and thus write bindings for other languages, if it gets to conclusion. Also, for all dweb stuff, the main current problem IMO is privacy. You can make private discussions on ssb, you can add an encryption layer on top of dat/beaker (I made myself a library using libsodium, that's good enough), but the main focus is about publishing things for the world, like you publish blog posts. All in all, on my free time, I'm now more interested to work on the dweb than on the web, so I can only encourage people to toy with it.
  - z0mbie42 7y ago
    Thank you! The lack of multi-device support isn't too much cumbersome? Also, you do not fear, due to the gossip protocol, that your private messages may be stored forever by peers and one day, your private key leak, and all your conversations are publicly exposed?
    - oosh0Jai 7y ago
      You're welcome :) > The lack of multi-device support isn't too much cumbersome? For ssb, it's not a problem for me, because I only use it on my laptop. There's a mobile app that exists, Manyverse, but you have to make a separate account for it, so usually ssb users will have a "john_doe" account and a "john_doe_mobile" one. I guess that's good enough. For dat, yes, it's been a major problem for me for a while, because I mainly use it to make my own "p2p cloud", so I want my data on mobile as well. There is the Bunsen browser on android, quite experimental while able to load dat urls. Sadly for me, localStorage doesn't work in it, and it's what I use to store my encryption keys. I thought it was hopeless for a while, until I started using Termux (which basically provided a POSIX environment on android). From there, I start dat processes to replicate my data, and I wrote a small server to serve them on 127.0.0.1, which then allow me to use the app in any browser on mobile. Completely hackish, and I can't recommend any sane person to do that, of course:) > Also, you do not fear, due to the gossip protocol, that your private messages may be stored forever by peers and one day, your private key leak, and all your conversations are publicly exposed? Yes indeed, this is a real risk for any p2p data. I _think_ it's still better than having them unencrypted on big database known for snooping, but we'll have to deal with that at some point. I guess the best would be to have some sort of encryption capable of autodestructing past a given age, I guess? That's a challenge for cryptographers, especially given it needs to not allow to just set the clock in the past to bypass it. Well, I hope the world will surprise me once again :) On the other hand, when I thought about that, I also considered it may actually be a good thing, depending on how many years it takes to break the encryption or find the keys. If I'm long dead, I'm fine with my data being decrypted, because otherwise we'll make the work of future historians impossible, if data is sparse and severely encrypted so they can't access it.
- vector_spaces 7y ago
  I love SSB, in principle. The protocol itself is very well documented[1]. The community tends to center lofty ideals around accessibility, anti-authoritarianism, and social responsibility[2][3] which I'm all about. Unfortunately, I've found the software implementations maintained by the SSBC to be "barely working" at best, with pretty scant and out of date documentation (to the point where code in "Getting Started" sections doesn't actually work) for most libraries/tools, PRs and issues lounge for many months without a response, and I've noticed a disappointing tendency among the SSB community and maintainers to be a bit condescending to newcomers and less technical users (not to mention cliqueish) in a way that seems in tension with some of the ideals that they pay lip service to. That said, I'm aware that we're all human, and my experience here is as more of an observer and tinkerer than active participant, so should be taken with a grain of salt [1] https://ssbc.github.io/scuttlebutt-protocol-guide/ https://ssbc.github.io/scuttlebutt-protocol-guide/ [2] https://www.theatlantic.com/technology/archive/2017/05/meet-the-counterantidisintermediationists/527553/ https://www.theatlantic.com/technology/archive/2017/05/meet-... [3] https://www.zdnet.com/article/manyverse-and-scuttlebutt-a-human-centric-technology-stack-for-social-applications/ https://www.zdnet.com/article/manyverse-and-scuttlebutt-a-hu...
- kseistrup 7y ago
  I've been on SSB for some 3 years (with some breaks when I had enough of npm). Once you're onboarded it works like a charm: exchange of data between peers works swiftly and efficiently, to the extend that you can even use it for realtime chat the way IRC works. The community is colourful and friendly, and the signal to noise ratio is high. I've learned a lot about fermentation and growing mushrooms and living off-grid while reading posts on SSB. My biggest frustration has been that all usable clients are written in nodejs. I recently took a seven months break to cool off from rage over npm (and yarn, for that matter), but now I'm back again. Onboarding can be tricky because there are no central servers — it's 100% p2p — but I guess it's easier these days than it was in the beginning. And if you know somebody who is already onboard it shouldn't pose a problem at all. A better site to start with is probably https://scuttlebutt.nz/ https://scuttlebutt.nz/
  - z0mbie42 7y ago
    Thank you! The lack of multi-device support isn't too much cumbersome? Also, do you not fear, due to the gossip protocol, that your private messages may be stored forever by peers and one day, your private key leaks, and all your conversations are publicly exposed?
    - kseistrup 7y ago
      It would be sweet if we had multi-device support, but it doesn't bother me too much: All my IDs are mutually following each other, and I have my “hops” set to 3, so I see virtually the same timeline on each device. What can be frustrating, though, is that notifications and private messages to a given ID are only visible from the device with that ID. I am not overly concerned about leaks of my private key, although the risk is certainly there. One think many people have to get used to, though, is that the log is append-only: once you've published a message and is no way to delete or undo the message — is it there for “all eternity”. The positive side is that you're more conscious about what you post and why, because there is no way of taking it back.
    - dduane 7y ago
      Not as much as I fear this from Facebook
magwa101 7y ago
Are these new systems created just to get a fun name out there?
- kseistrup 7y ago
  Scuttlebot is a pun on scuttlebutt, which is the cask used to serve water on a ship, and the protocol was invented by a guy who lives on a boat. Think of it as “water cooler”: a place where people can meet and smalltalk.
matlin 7y ago
This is really cool. I dig the API and mechanisms to exchange data. I've been working on a DBaaS (like Firebase) that works via message passing between users and now I'm thinking Scuttlebot could be great tool to integrate with it. I wish there was more info on how the replication worked...
- seanb 7y ago
  SSB Protocol Guide: https://ssbc.github.io/scuttlebutt-protocol-guide/ https://ssbc.github.io/scuttlebutt-protocol-guide/
milansuk 7y ago
So the homepage of this project is basically documentation. Where others introduce use-cases, you actually teach us how it's done. I love it.
- jariel 7y ago
  Except for those of us who have no idea really what it is, how it would be used, or what the features really do.
  - SyrupThinker 7y ago
    Then you probably want https://scuttlebutt.nz/ https://scuttlebutt.nz/ instead, which is more of a user introduction.
audiodude 7y ago
I've used patchwork before, but can someone comment on the other applications of the scuttlebut network and where this scuttlebot fits in?
- TimJRobinson 7y ago
  Scuttlebot is the core server and plugins for Scuttlebutt. The Github repo is now called ssb-server. Patchwork uses it for replication and adds a gui and new plugins on top of it.
- fabianhjr 7y ago
  Yes, there is a decentralized git: https://git.scuttlebot.io/%25n92DiQh7ietE%2BR%2BX%2FI403LQoyf2DtR3WQfCkDKlheQU%3D.sha256 https://git.scuttlebot.io/%25n92DiQh7ietE%2BR%2BX%2FI403LQoy... A decentralized book review system and a decentralized chess.
dang 7y ago
A thread from 2017: https://news.ycombinator.com/item?id=14409187 https://news.ycombinator.com/item?id=14409187 Points to https://scuttlebutt.nz/ https://scuttlebutt.nz/ which is related
- masukomi 7y ago
  This HN post is linking to the protocol docs. Scuttlebutt.nz is about the social network built on top of them. More importantly it's the most significant thing that uses the protocol. The "apps" that use the protocol are, in practice, just plugins that work in one particular app (somewhat geeky app) and that particular app is primarily a client for the social network.
asdkhadsj 7y ago
Warning: A bit of a ramble about my experience trying to use SSB for my "application". I really enjoy SSB, conceptually. In reality I've had a fair amount of trouble getting started writing an application in it. Notably, I have difficulty knowing what patterns are good or bad in SSB. As far as I can tell SSB utilizes effectively a commit ledger for user data. This is fine, but some of the specifics of my application mean that I feel the need to create many sub-identities so that revisions to a piece of content don't bloat the parent identity when syncing to mobile/etc. Strangely, I've got a fair bit of experience in these types of designs - as I've written my own distributed storage for the application in question in many different forms (various ledgers, CRDTs, etc). Yet my issue is that I'm unsure what is considered "bad form" for the SSB crowd. I don't want to write an application for SSB but written in some form of SSB anti-pattern. Combine this with SSB not having well established[1] implementations in my language of choice (Rust), and all around as much as I want to use SSB as the foundation for my app the protocol was just a bit of a heavy burden currently. I care what the SSB community thinks because I don't want to just "use" the SSB protocol, I want my application to join the SSB hubs/networks and share data. I've looked a lot into this, but I still struggle to conceive exactly how my application can fit into the SSB ecosystem, if at all. This isn't a complaint post, just sharing some personal thoughts and my confusing history with SSB. Love the project, and I especially love the design of gossip, person to person based applications in this day and age. [1]: Rust does have a handful of SSB libs, but I think I'd be looking for a full, batteries included library. Everything I've seen is quite low level SSB.
- moojd 7y ago
  I've been tinkering with SSB recently as well and I'm also finding it difficult to figure out how to interface with the wider SSB network in the 'correct' way. The communication protocol is well documented and easy to implement if you have libsodium available. Building applications on it should be straight forward considering it's just an eventually consistent event store. I am having a few issues though. The 'api' that patchwork and other parts of the network use on top of the communication protocol seems to not be documented at all outside of the 'post' message. I'm not sure if it's not documented or if I just haven't discovered it yet. For example, I'm having to dive in to the code to see how a 'vote' message is formatted. I'm not sure how all of the various clients are staying compatible. None of the message formats seem to versioned in anyway either. I'm also not sure what are the 'correct' ways to introduce my own message types. Should I use a vendor prefix in the type or something? Is it bad form to pollute the network with your own message types that other clients can't understand? I've just started diving in to all this so maybe I just haven't gotten deep enough yet. I wrote a little toy SSB implementation last weekend to learn the protocol. I've really enjoyed playing around with it so far but I'm still not sure if it's the right fit for the applications I want to build yet.
TimJRobinson 7y ago
Huge fan of Scuttlebutt and think it could (should) be the future of the social internet. I recently quit my job to write about and work on decentralized tech full time with scuttlebutt being my primary focus. I've written about why I think it's so important here: https://adecentralizedworld.com/2020/03/what-is-scuttlebutt/ https://adecentralizedworld.com/2020/03/what-is-scuttlebutt/
- alephnan 7y ago
  > Decentralized social networks have been tried before, the two most well known are Diaspora and Mastadon. With these services there [is still moderation]... Scuttlebutt is how I believe the social web should function in the future. Even the Hackernews community which lean towards decentralization more so than the general public would still argue there is value in moderation.
  - TimJRobinson 7y ago
    For sure, any network without any moderation is going to quickly fall into chaos. I'm working on a post on decentralized personalized moderation at the moment which should be out in the next day or two :)
    - moojd 7y ago
      I think this right here is a killer app for decentralized social networks and is something I've been thinking about a lot. I'm looking forward to your post!
  - moojd 7y ago
    It's possible (and important) that moderation can be implemented in a decentralized community. Aether[0] has a really interesting take on it where it is essentially democratic. [0] https://getaether.net/ https://getaether.net/
  - sneak 7y ago
    Moderation feeds can be published into decentralized networks just like the content messages themselves. Then readers can choose their set of preferred moderators (author/item killfile-stream publishers) to apply clientside.
seemslegit 7y ago
Uh, for something touting e2e encryption and security it would be better if the site did not serve over plain http by default
- progre 7y ago
  https adds nothing to a page when the only trafic is server -> client
  - seemslegit 7y ago
    Of course it does, HTTP is never only server -> client - from preventing a passive eavesdropper from seeing what pages are being browsed on the server, cookies, UA fingerprinting etc. to active content modification in transit.
    - progre 7y ago
      Pages are still visable in the tls handshake, no coockies on this page (that would be client-> server traffic). But yeah, good point about the fingerprinting and content modification
      - seemslegit 7y ago
        No, they aren't - just the hostname (domain name).
cachestash 7y ago
npm install sigh...
- camdenlock 7y ago
  Yeah, this is where I lost interest too.
  - crispyporkbites 7y ago
    I don’t understand why npm install is so bad here? It’s just a package manager. It’s not hard to write your implementation without npm if you prefer.
    - fabianhjr 7y ago
      it is, I tried with haskell and gave up when I noticed I couldn't easily get json serialization to be in the same order as the javascript implementation and therefor couldn't verify nor sign messages with interoperability with the current clients.
      - codygman 7y ago
        > I tried with haskell and gave up when I noticed I couldn't easily get json serialization to be in the same order as the javascript implementation and therefor couldn't verify nor sign messages with interoperability with the current clients. This gets you most of the way there: > With the Aeson library, the order of keys in objects is undefined due to objects being implemented as HashMaps. To allow user-specified key orders in the pretty-printed JSON, encodePretty' can be configured with a comparison function. - https://hackage.haskell.org/package/aeson-pretty-0.8.7/docs/Data-Aeson-Encode-Pretty.html#g:3 https://hackage.haskell.org/package/aeson-pretty-0.8.7/docs/... I'm not sure that messes with spacing or not though or you require specific space preserving requirements to interoperate with current clients. Even if not you can probably look at the source of encodePretty' and work out how to not mess with spacing or keep it all on one line. I've wanted a version of Aeson for a while that preserves order and all formatting for things like linters that are very unobtrusive. It seems your use case would have benefitted from that as well.
deleted 7y ago
[deleted]
Vendan 7y ago
Scuttlebutt is a neat concept, burdened by a bad protocol. Signing a message involves serializing a json object, signing it, adding the signature as a field on that json object, and then serializing it again. To verify, you deserialize the message into an object, remove the signature field, and then reserialize it, and verify the signature against that new serialization. This means that all the clients have to have a json serialization that 100% matches the node.js serialization that the majority (totality?) of current clients use. Any other implementation (I wrote one in Go) becomes a nightmare of chasing rarer and rarer differences that prevent verification, and if you miss one before you serialize your own message with it, suddenly you've poisoned your own journal... All in all, it's something to study and learn from, but I strongly recommend not becoming involved unless you are 100% happy with being tied into node.js.
- alexh1 7y ago
  This is how Cosmos (https://cosmos.network/ https://cosmos.network/) deals with signing transactions, caused _major_ headaches for me trying to figure out why my signatures were invalid when sending them from Node.js
  - goliatone 7y ago
    I did something similar for a project that had clients in go and node. I solved this by flattening the object key paths to strings and sorting the keys, basically. You need to build that into the client serialization/deserialization, it feels clunky but I had 0 issues and has been working smoothly for a long while now.
- sneak 7y ago
  When I was faced with this (signing a structure), I serialized the json into base64, then put that base64 string as a value (along with the MAC) into a new json document. It of course increases deserialization overhead (json, verify, unbase64, inner json) but sidesteps this issue. I thought about sorting keys and other things like that, and the dozen edge cases and potential malleability issues dissuaded me for the compatibility issues mentioned above. How have others solved it?
Boss123123 7y ago
Thanks for this post really great to read same here I wrote at https://www.triveditech.com/ https://www.triveditech.com/
thulecitizen 7y ago
Awesome similar DWeb projects for those interested/inspired: Ceptr/Holochain: http://developer.holochain.org/ http://developer.holochain.org/ DAT protocol: https://datprotocol.github.io/how-dat-works/ https://datprotocol.github.io/how-dat-works/
jariel 7y ago
Looks very cool - but to the author, you'll have to write some introductory 'what is this documentation' because the single sentence on the landing page isn't quite enough. "It's a magofinisticks peer-to-peer storage functionomatic" is how it might read to some people. So what it is, what it does, the value of it, some general overview of the features and how they work would go a long way. Looks cool though.
miguelmota 7y ago
Here's a getting started with scuttlebutt tutorial I wrote last year for beginners https://miguelmota.com/blog/getting-started-with-secure-scuttlebutt/ https://miguelmota.com/blog/getting-started-with-secure-scut...
- myself248 7y ago
  There's an unstated assumption in all these things, that you already know some people in the network. If not, you can see all sorts of interesting things, but nobody knows you're replying to them, and it's a very lonely experience. If there's a way around that, I haven't figured it out, and all these tutorials aimed at people who've never heard of the network seem misplaced -- if you don't already have a personal invite, the network is useless to you, but if you already have a personal invite, the article is useless to you.
arj 7y ago
Can run directly (https://github.com/arj03/ssb-browser-demo https://github.com/arj03/ssb-browser-demo) in the browser as well.
nanomonkey 7y ago
The protocol guide is quite beautiful: https://ssbc.github.io/scuttlebutt-protocol-guide/ https://ssbc.github.io/scuttlebutt-protocol-guide/
aabbcc1241 7y ago
In addition, Manyverse is a social network built on SSB. Each user has a chain, and they sync and relay their friends chain.
dpc_pw 7y ago
SSB is great conceptually, but is plagued with bad design decisions, and with JS. I gave up on it, after trying it couple of times. If someone took it, and did it right, it might be a huge hit.
freelancercv 7y ago
Surprised, no one mentioned GUNDB, which does very similar thing and supports real time operation. Gundb: https://gun.eco https://gun.eco
snicky 7y ago
Why does the users' data have to be stored in a form of a linked list (or "blockchain")? Couldn't every peer in the network just hold his own database full of messages and blobs and sign them with their private key when requested by an invited follower? Edit: typo
- capableweb 7y ago
  Basically Scuttlebot is a collection of linked lists (or blockchains if you will) because you only have to discover one message to be able to fetch the rest of the related content, since it's all linked as a DAG.
  - snicky 7y ago
    Yeah, so this is the design decision that I don't really understand. Each message on the blockchain contains the author's public key representing his identity. If I already know the identity of the author, I could just go ahead and befriend him and later ask him or other peers who are his friends to share his "feed" with me. If the feed lives in a relational DB, I could then use SQL to ask for particular messages like: "select * from posts order by created_at desc" or: "select * from posts order by created_at asc where created_at > <timestamp>" or: "select * from blobs where type = 'photo'" etc. In other words, I could query the feeds the way I want to. Also, this allows the author to delete the content, although this would also require to send some DELETE requests to peers mirroring the author's feed and those requests would have to be respected, but that's another story.
    - capableweb 7y ago
      The idea behind this is that it's streams all the way through, so there could be something more efficient to use for these "rolling" database that basically are streams of data. So instead of SQL, it's map-filter-reduce, a query engine that works over streams. https://github.com/ssbc/ssb-query/blob/master/README.md https://github.com/ssbc/ssb-query/blob/master/README.md - https://github.com/dominictarr/map-filter-reduce https://github.com/dominictarr/map-filter-reduce It's probably a result of going with pull-streams and basically architecture all scuttlebot tools/libs around pull-streams, so you want something that works over streams. And yeah, Scuttlebot is not built with SQL as it's query language, but you could probably easily write something that understands both SQL queries and the file format scuttlebot stores the messages in. Thing is, you don't want to send these queries to other peers (they are all local), because the system is setup to be offline-first and only reach out to the network when it has to, not just for listing latest messages. In general, you want all the action of fetching/syncing of data to happen exactly when you want it to happen, and then when you're done it doesn't continue until next "sync". So everything you do is local. But then, if "query the feeds the way I want to" is the reason you don't like it, then so be it. But if it's just that you don't understand it, then reading through and digging into the different repositories will give you a better viewpoint. Start at the ssb-query and look what's using it and what it's using.
soapdog 7y ago
Hey, this page is outdated. It is basically a list of APIs from ssb-server and some other related plugins, but there are entries there that are old and haven't been kept in sync with the actual modules. https://scuttlebutt.nz https://scuttlebutt.nz (nee https://ssb.nz https://ssb.nz) is a better link for all things SSB. The protocol guide at https://ssbc.github.io/scuttlebutt-protocol-guide/ https://ssbc.github.io/scuttlebutt-protocol-guide/ is your friend in understanding our little garden. Be aware that SSB grew much like a garden. It is not a protocol and ecosystem designed by committee with a cold and effective process. It grew from simple stuff into more complex stuff, and yes we all understand some of the challenges pilled upon all of us due to bad decisions in the past. There is a lot to love in SSB. Instead of going "npm install, meh", you should try it out. You don't need npm or nodejs to try SSB out, you can just pick any of the clients listed in the first page I linked. I develop one of those clients, patchfox, but it is not a full client so you'll need to bring your own ssb-server.

361 points by pcr910303 7y ago

THE SCUTTLEBUG HELPED MARIO BEAT FOX

I recently discovered SSB and was really intrigued. I would love to hear an experience report from actual users like what is working great, and what is bad.

jekrb 7y ago
It's been a while, but I used to be an active SSB user. I hosted SSB pubs and used to post on patchwork semi-regularly. I thought it worked pretty well as a social network. I discovered new and interesting ideas from folks that I don't see much on mainstream social media. I haven't followed the space much recently, and I'm curious about how it has evolved over the last year or so. My favorite memories on SSB: Someone promoted a book that they had written, and we arranged for a sales transaction by talking purely over the network. I sent them some amount of Bitcoin, and they sent me the PDF of their book. It felt very personal to work with the author directly, and side-step payment processors. I loved taking my laptop out on the train or to a coffee shop, and replying to threads and publishing a post to SSB while offline. Something about reading other peoples ideas while disconnected, and then writing my thoughts, and having them automatically sync to the network when I got back on my WiFi at home, gave me a different perspective on ways to use technology.
- z0mbie42 7y ago
  Thank you! The lack of multi-device support wasn't too much cumbersome?
  - jekrb 7y ago
    The lack of multi-device support was a constraint that gave me a different perspective of interactions on the web. My ssb keypair was on a work laptop, so when I changed jobs and had to give my laptop back, I lost my keypair. Now, I could have exported the keypair and continued to use my "account" on my new laptop. The network would have synced on my new device, and I'd get all my posts and pictures back. But decided to embrace the constraint instead. When I rejoin the network, I'll have a new keypair, and no post history. I think this can have an interesting effect on how we view our attachment to data.
    - rakoo 7y ago
      Creating a new account from scratch also means rotating your keys, which is a good practice. As the last few discussions on PGP have shown, the model of having a long-term identity key is more dangerous than it seems, because a single mistake (by you or by the application developer) means so much content can now be leaked. It's probably easier to let the natural connection between people be the vector of long-term trust, which ironically SSB emphasizes on
- zozbot234 7y ago
  > I loved taking my laptop out on the train or to a coffee shop, and replying to threads and publishing a post to SSB while offline. Something about reading other peoples ideas while disconnected, and then writing my thoughts, and having them automatically sync to the network when I got back on my WiFi at home, gave me a different perspective on ways to use technology. You can do this with Usenet and most BBS's. Most native IM apps will also do this for you, plus of course there's email.
  - capableweb 7y ago
    One cool feature of Scuttlebot is that if you and your friend are already following each other, you only need a connection to each other P2P to be able to send messages to each other. So if you're on a train with ad-hoc WiFi connected to each other, you can still proceed as usual and sync stuff. I don't think this feature exists in Usenet and BBS's where there is a central server who masterminds the sync that everyone is doing. Same with email, requires a server (local or remote) to send/receive stuff while in SSB both local and remote are usually the same machine.
    - tlavoie 7y ago
      For BBSs, you're correct. Usenet (and email) used UUCP, which is actually much closer in concept I think here. UUCP is a store-and-forward mechanism, not dependent on a real-time connection to a particular server. I used to run a node, connected to a guy I'd met who worked for an ISP. He had, gasp, a full-time network connection via ISDN; pretty magical in these days of dial up. So, Usenet feeds were configured on my own little system, essentially subscribing to the newsgroups I wanted. Periodically, it would dial out to the other gent, upload any new posts from me, and download anything new on those newsgroups. My email came and went the same way. Naturally, what I got was a subset of what he had accessible. While I never used this functionality, I could have had others call up to me, and I would just be an intermediate link in the chain. RFC 976 (https://tools.ietf.org/html/rfc976 https://tools.ietf.org/html/rfc976) describes how this works for email, including SMTP over UUCP.
      - 7y ago
oosh0Jai 7y ago
I've been using ssb for about a year and a half, and love it. The ideas behind the dweb do not get the credit they deserve, IMO : p2p data sharing where your data is sent directly to your friends rather than centralized on a know-it-all server is what the world needs, for anyone concerned about privacy. If I had to say what I don't like about ssb, it would probably be that it's not easy to write an application for it. After trying for a while, I turned to Dat and the Beaker browser, which let me write frontend applications the usual way, with just an api to manage the p2p archives. Both dat and ssb are javascript projects, so if you want to write your dweb applications using an other language, you're out of luck. I heard there is a rust implementation of dat being worked on that will allow to access an ABI, and thus write bindings for other languages, if it gets to conclusion. Also, for all dweb stuff, the main current problem IMO is privacy. You can make private discussions on ssb, you can add an encryption layer on top of dat/beaker (I made myself a library using libsodium, that's good enough), but the main focus is about publishing things for the world, like you publish blog posts. All in all, on my free time, I'm now more interested to work on the dweb than on the web, so I can only encourage people to toy with it.
- z0mbie42 7y ago
  Thank you! The lack of multi-device support isn't too much cumbersome? Also, you do not fear, due to the gossip protocol, that your private messages may be stored forever by peers and one day, your private key leak, and all your conversations are publicly exposed?
  - oosh0Jai 7y ago
    You're welcome :) > The lack of multi-device support isn't too much cumbersome? For ssb, it's not a problem for me, because I only use it on my laptop. There's a mobile app that exists, Manyverse, but you have to make a separate account for it, so usually ssb users will have a "john_doe" account and a "john_doe_mobile" one. I guess that's good enough. For dat, yes, it's been a major problem for me for a while, because I mainly use it to make my own "p2p cloud", so I want my data on mobile as well. There is the Bunsen browser on android, quite experimental while able to load dat urls. Sadly for me, localStorage doesn't work in it, and it's what I use to store my encryption keys. I thought it was hopeless for a while, until I started using Termux (which basically provided a POSIX environment on android). From there, I start dat processes to replicate my data, and I wrote a small server to serve them on 127.0.0.1, which then allow me to use the app in any browser on mobile. Completely hackish, and I can't recommend any sane person to do that, of course:) > Also, you do not fear, due to the gossip protocol, that your private messages may be stored forever by peers and one day, your private key leak, and all your conversations are publicly exposed? Yes indeed, this is a real risk for any p2p data. I _think_ it's still better than having them unencrypted on big database known for snooping, but we'll have to deal with that at some point. I guess the best would be to have some sort of encryption capable of autodestructing past a given age, I guess? That's a challenge for cryptographers, especially given it needs to not allow to just set the clock in the past to bypass it. Well, I hope the world will surprise me once again :) On the other hand, when I thought about that, I also considered it may actually be a good thing, depending on how many years it takes to break the encryption or find the keys. If I'm long dead, I'm fine with my data being decrypted, because otherwise we'll make the work of future historians impossible, if data is sparse and severely encrypted so they can't access it.
vector_spaces 7y ago
I love SSB, in principle. The protocol itself is very well documented[1]. The community tends to center lofty ideals around accessibility, anti-authoritarianism, and social responsibility[2][3] which I'm all about. Unfortunately, I've found the software implementations maintained by the SSBC to be "barely working" at best, with pretty scant and out of date documentation (to the point where code in "Getting Started" sections doesn't actually work) for most libraries/tools, PRs and issues lounge for many months without a response, and I've noticed a disappointing tendency among the SSB community and maintainers to be a bit condescending to newcomers and less technical users (not to mention cliqueish) in a way that seems in tension with some of the ideals that they pay lip service to. That said, I'm aware that we're all human, and my experience here is as more of an observer and tinkerer than active participant, so should be taken with a grain of salt [1] https://ssbc.github.io/scuttlebutt-protocol-guide/ https://ssbc.github.io/scuttlebutt-protocol-guide/ [2] https://www.theatlantic.com/technology/archive/2017/05/meet-the-counterantidisintermediationists/527553/ https://www.theatlantic.com/technology/archive/2017/05/meet-... [3] https://www.zdnet.com/article/manyverse-and-scuttlebutt-a-human-centric-technology-stack-for-social-applications/ https://www.zdnet.com/article/manyverse-and-scuttlebutt-a-hu...
kseistrup 7y ago
I've been on SSB for some 3 years (with some breaks when I had enough of npm). Once you're onboarded it works like a charm: exchange of data between peers works swiftly and efficiently, to the extend that you can even use it for realtime chat the way IRC works. The community is colourful and friendly, and the signal to noise ratio is high. I've learned a lot about fermentation and growing mushrooms and living off-grid while reading posts on SSB. My biggest frustration has been that all usable clients are written in nodejs. I recently took a seven months break to cool off from rage over npm (and yarn, for that matter), but now I'm back again. Onboarding can be tricky because there are no central servers — it's 100% p2p — but I guess it's easier these days than it was in the beginning. And if you know somebody who is already onboard it shouldn't pose a problem at all. A better site to start with is probably https://scuttlebutt.nz/ https://scuttlebutt.nz/
- z0mbie42 7y ago
  Thank you! The lack of multi-device support isn't too much cumbersome? Also, do you not fear, due to the gossip protocol, that your private messages may be stored forever by peers and one day, your private key leaks, and all your conversations are publicly exposed?
  - kseistrup 7y ago
    It would be sweet if we had multi-device support, but it doesn't bother me too much: All my IDs are mutually following each other, and I have my “hops” set to 3, so I see virtually the same timeline on each device. What can be frustrating, though, is that notifications and private messages to a given ID are only visible from the device with that ID. I am not overly concerned about leaks of my private key, although the risk is certainly there. One think many people have to get used to, though, is that the log is append-only: once you've published a message and is no way to delete or undo the message — is it there for “all eternity”. The positive side is that you're more conscious about what you post and why, because there is no way of taking it back.
  - dduane 7y ago
    Not as much as I fear this from Facebook

magwa101 7y ago

Are these new systems created just to get a fun name out there?

kseistrup 7y ago
Scuttlebot is a pun on scuttlebutt, which is the cask used to serve water on a ship, and the protocol was invented by a guy who lives on a boat. Think of it as “water cooler”: a place where people can meet and smalltalk.

matlin 7y ago

This is really cool. I dig the API and mechanisms to exchange data. I've been working on a DBaaS (like Firebase) that works via message passing between users and now I'm thinking Scuttlebot could be great tool to integrate with it. I wish there was more info on how the replication worked...

seanb 7y ago
SSB Protocol Guide: https://ssbc.github.io/scuttlebutt-protocol-guide/ https://ssbc.github.io/scuttlebutt-protocol-guide/

milansuk 7y ago

So the homepage of this project is basically documentation. Where others introduce use-cases, you actually teach us how it's done. I love it.

jariel 7y ago
Except for those of us who have no idea really what it is, how it would be used, or what the features really do.
- SyrupThinker 7y ago
  Then you probably want https://scuttlebutt.nz/ https://scuttlebutt.nz/ instead, which is more of a user introduction.

audiodude 7y ago

I've used patchwork before, but can someone comment on the other applications of the scuttlebut network and where this scuttlebot fits in?

TimJRobinson 7y ago
Scuttlebot is the core server and plugins for Scuttlebutt. The Github repo is now called ssb-server. Patchwork uses it for replication and adds a gui and new plugins on top of it.
fabianhjr 7y ago
Yes, there is a decentralized git: https://git.scuttlebot.io/%25n92DiQh7ietE%2BR%2BX%2FI403LQoyf2DtR3WQfCkDKlheQU%3D.sha256 https://git.scuttlebot.io/%25n92DiQh7ietE%2BR%2BX%2FI403LQoy... A decentralized book review system and a decentralized chess.

dang 7y ago

A thread from 2017: https://news.ycombinator.com/item?id=14409187 https://news.ycombinator.com/item?id=14409187 Points to https://scuttlebutt.nz/ https://scuttlebutt.nz/ which is related

masukomi 7y ago
This HN post is linking to the protocol docs. Scuttlebutt.nz is about the social network built on top of them. More importantly it's the most significant thing that uses the protocol. The "apps" that use the protocol are, in practice, just plugins that work in one particular app (somewhat geeky app) and that particular app is primarily a client for the social network.

asdkhadsj 7y ago

Warning: A bit of a ramble about my experience trying to use SSB for my "application". I really enjoy SSB, conceptually. In reality I've had a fair amount of trouble getting started writing an application in it. Notably, I have difficulty knowing what patterns are good or bad in SSB. As far as I can tell SSB utilizes effectively a commit ledger for user data. This is fine, but some of the specifics of my application mean that I feel the need to create many sub-identities so that revisions to a piece of content don't bloat the parent identity when syncing to mobile/etc. Strangely, I've got a fair bit of experience in these types of designs - as I've written my own distributed storage for the application in question in many different forms (various ledgers, CRDTs, etc). Yet my issue is that I'm unsure what is considered "bad form" for the SSB crowd. I don't want to write an application for SSB but written in some form of SSB anti-pattern. Combine this with SSB not having well established[1] implementations in my language of choice (Rust), and all around as much as I want to use SSB as the foundation for my app the protocol was just a bit of a heavy burden currently. I care what the SSB community thinks because I don't want to just "use" the SSB protocol, I want my application to join the SSB hubs/networks and share data. I've looked a lot into this, but I still struggle to conceive exactly how my application can fit into the SSB ecosystem, if at all. This isn't a complaint post, just sharing some personal thoughts and my confusing history with SSB. Love the project, and I especially love the design of gossip, person to person based applications in this day and age. [1]: Rust does have a handful of SSB libs, but I think I'd be looking for a full, batteries included library. Everything I've seen is quite low level SSB.

moojd 7y ago
I've been tinkering with SSB recently as well and I'm also finding it difficult to figure out how to interface with the wider SSB network in the 'correct' way. The communication protocol is well documented and easy to implement if you have libsodium available. Building applications on it should be straight forward considering it's just an eventually consistent event store. I am having a few issues though. The 'api' that patchwork and other parts of the network use on top of the communication protocol seems to not be documented at all outside of the 'post' message. I'm not sure if it's not documented or if I just haven't discovered it yet. For example, I'm having to dive in to the code to see how a 'vote' message is formatted. I'm not sure how all of the various clients are staying compatible. None of the message formats seem to versioned in anyway either. I'm also not sure what are the 'correct' ways to introduce my own message types. Should I use a vendor prefix in the type or something? Is it bad form to pollute the network with your own message types that other clients can't understand? I've just started diving in to all this so maybe I just haven't gotten deep enough yet. I wrote a little toy SSB implementation last weekend to learn the protocol. I've really enjoyed playing around with it so far but I'm still not sure if it's the right fit for the applications I want to build yet.

TimJRobinson 7y ago

Huge fan of Scuttlebutt and think it could (should) be the future of the social internet. I recently quit my job to write about and work on decentralized tech full time with scuttlebutt being my primary focus. I've written about why I think it's so important here: https://adecentralizedworld.com/2020/03/what-is-scuttlebutt/ https://adecentralizedworld.com/2020/03/what-is-scuttlebutt/

alephnan 7y ago
> Decentralized social networks have been tried before, the two most well known are Diaspora and Mastadon. With these services there [is still moderation]... Scuttlebutt is how I believe the social web should function in the future. Even the Hackernews community which lean towards decentralization more so than the general public would still argue there is value in moderation.
- TimJRobinson 7y ago
  For sure, any network without any moderation is going to quickly fall into chaos. I'm working on a post on decentralized personalized moderation at the moment which should be out in the next day or two :)
  - moojd 7y ago
    I think this right here is a killer app for decentralized social networks and is something I've been thinking about a lot. I'm looking forward to your post!
- moojd 7y ago
  It's possible (and important) that moderation can be implemented in a decentralized community. Aether[0] has a really interesting take on it where it is essentially democratic. [0] https://getaether.net/ https://getaether.net/
- sneak 7y ago
  Moderation feeds can be published into decentralized networks just like the content messages themselves. Then readers can choose their set of preferred moderators (author/item killfile-stream publishers) to apply clientside.

seemslegit 7y ago

Uh, for something touting e2e encryption and security it would be better if the site did not serve over plain http by default

progre 7y ago
https adds nothing to a page when the only trafic is server -> client
- seemslegit 7y ago
  Of course it does, HTTP is never only server -> client - from preventing a passive eavesdropper from seeing what pages are being browsed on the server, cookies, UA fingerprinting etc. to active content modification in transit.
  - progre 7y ago
    Pages are still visable in the tls handshake, no coockies on this page (that would be client-> server traffic). But yeah, good point about the fingerprinting and content modification
    - seemslegit 7y ago
      No, they aren't - just the hostname (domain name).

cachestash 7y ago

npm install sigh...

camdenlock 7y ago
Yeah, this is where I lost interest too.
- crispyporkbites 7y ago
  I don’t understand why npm install is so bad here? It’s just a package manager. It’s not hard to write your implementation without npm if you prefer.
  - fabianhjr 7y ago
    it is, I tried with haskell and gave up when I noticed I couldn't easily get json serialization to be in the same order as the javascript implementation and therefor couldn't verify nor sign messages with interoperability with the current clients.
    - codygman 7y ago
      > I tried with haskell and gave up when I noticed I couldn't easily get json serialization to be in the same order as the javascript implementation and therefor couldn't verify nor sign messages with interoperability with the current clients. This gets you most of the way there: > With the Aeson library, the order of keys in objects is undefined due to objects being implemented as HashMaps. To allow user-specified key orders in the pretty-printed JSON, encodePretty' can be configured with a comparison function. - https://hackage.haskell.org/package/aeson-pretty-0.8.7/docs/Data-Aeson-Encode-Pretty.html#g:3 https://hackage.haskell.org/package/aeson-pretty-0.8.7/docs/... I'm not sure that messes with spacing or not though or you require specific space preserving requirements to interoperate with current clients. Even if not you can probably look at the source of encodePretty' and work out how to not mess with spacing or keep it all on one line. I've wanted a version of Aeson for a while that preserves order and all formatting for things like linters that are very unobtrusive. It seems your use case would have benefitted from that as well.

deleted 7y ago

[deleted]

Vendan 7y ago

Scuttlebutt is a neat concept, burdened by a bad protocol. Signing a message involves serializing a json object, signing it, adding the signature as a field on that json object, and then serializing it again. To verify, you deserialize the message into an object, remove the signature field, and then reserialize it, and verify the signature against that new serialization. This means that all the clients have to have a json serialization that 100% matches the node.js serialization that the majority (totality?) of current clients use. Any other implementation (I wrote one in Go) becomes a nightmare of chasing rarer and rarer differences that prevent verification, and if you miss one before you serialize your own message with it, suddenly you've poisoned your own journal... All in all, it's something to study and learn from, but I strongly recommend not becoming involved unless you are 100% happy with being tied into node.js.

alexh1 7y ago
This is how Cosmos (https://cosmos.network/ https://cosmos.network/) deals with signing transactions, caused _major_ headaches for me trying to figure out why my signatures were invalid when sending them from Node.js
- goliatone 7y ago
  I did something similar for a project that had clients in go and node. I solved this by flattening the object key paths to strings and sorting the keys, basically. You need to build that into the client serialization/deserialization, it feels clunky but I had 0 issues and has been working smoothly for a long while now.
sneak 7y ago
When I was faced with this (signing a structure), I serialized the json into base64, then put that base64 string as a value (along with the MAC) into a new json document. It of course increases deserialization overhead (json, verify, unbase64, inner json) but sidesteps this issue. I thought about sorting keys and other things like that, and the dozen edge cases and potential malleability issues dissuaded me for the compatibility issues mentioned above. How have others solved it?

Boss123123 7y ago

Thanks for this post really great to read same here I wrote at https://www.triveditech.com/ https://www.triveditech.com/

thulecitizen 7y ago

Awesome similar DWeb projects for those interested/inspired: Ceptr/Holochain: http://developer.holochain.org/ http://developer.holochain.org/ DAT protocol: https://datprotocol.github.io/how-dat-works/ https://datprotocol.github.io/how-dat-works/

jariel 7y ago

Looks very cool - but to the author, you'll have to write some introductory 'what is this documentation' because the single sentence on the landing page isn't quite enough. "It's a magofinisticks peer-to-peer storage functionomatic" is how it might read to some people. So what it is, what it does, the value of it, some general overview of the features and how they work would go a long way. Looks cool though.

miguelmota 7y ago

Here's a getting started with scuttlebutt tutorial I wrote last year for beginners https://miguelmota.com/blog/getting-started-with-secure-scuttlebutt/ https://miguelmota.com/blog/getting-started-with-secure-scut...

myself248 7y ago
There's an unstated assumption in all these things, that you already know some people in the network. If not, you can see all sorts of interesting things, but nobody knows you're replying to them, and it's a very lonely experience. If there's a way around that, I haven't figured it out, and all these tutorials aimed at people who've never heard of the network seem misplaced -- if you don't already have a personal invite, the network is useless to you, but if you already have a personal invite, the article is useless to you.

arj 7y ago

Can run directly (https://github.com/arj03/ssb-browser-demo https://github.com/arj03/ssb-browser-demo) in the browser as well.

nanomonkey 7y ago

The protocol guide is quite beautiful: https://ssbc.github.io/scuttlebutt-protocol-guide/ https://ssbc.github.io/scuttlebutt-protocol-guide/

aabbcc1241 7y ago

In addition, Manyverse is a social network built on SSB. Each user has a chain, and they sync and relay their friends chain.

dpc_pw 7y ago

SSB is great conceptually, but is plagued with bad design decisions, and with JS. I gave up on it, after trying it couple of times. If someone took it, and did it right, it might be a huge hit.

freelancercv 7y ago

Surprised, no one mentioned GUNDB, which does very similar thing and supports real time operation. Gundb: https://gun.eco https://gun.eco

snicky 7y ago

Why does the users' data have to be stored in a form of a linked list (or "blockchain")? Couldn't every peer in the network just hold his own database full of messages and blobs and sign them with their private key when requested by an invited follower? Edit: typo

capableweb 7y ago
Basically Scuttlebot is a collection of linked lists (or blockchains if you will) because you only have to discover one message to be able to fetch the rest of the related content, since it's all linked as a DAG.
- snicky 7y ago
  Yeah, so this is the design decision that I don't really understand. Each message on the blockchain contains the author's public key representing his identity. If I already know the identity of the author, I could just go ahead and befriend him and later ask him or other peers who are his friends to share his "feed" with me. If the feed lives in a relational DB, I could then use SQL to ask for particular messages like: "select * from posts order by created_at desc" or: "select * from posts order by created_at asc where created_at > <timestamp>" or: "select * from blobs where type = 'photo'" etc. In other words, I could query the feeds the way I want to. Also, this allows the author to delete the content, although this would also require to send some DELETE requests to peers mirroring the author's feed and those requests would have to be respected, but that's another story.
  - capableweb 7y ago
    The idea behind this is that it's streams all the way through, so there could be something more efficient to use for these "rolling" database that basically are streams of data. So instead of SQL, it's map-filter-reduce, a query engine that works over streams. https://github.com/ssbc/ssb-query/blob/master/README.md https://github.com/ssbc/ssb-query/blob/master/README.md - https://github.com/dominictarr/map-filter-reduce https://github.com/dominictarr/map-filter-reduce It's probably a result of going with pull-streams and basically architecture all scuttlebot tools/libs around pull-streams, so you want something that works over streams. And yeah, Scuttlebot is not built with SQL as it's query language, but you could probably easily write something that understands both SQL queries and the file format scuttlebot stores the messages in. Thing is, you don't want to send these queries to other peers (they are all local), because the system is setup to be offline-first and only reach out to the network when it has to, not just for listing latest messages. In general, you want all the action of fetching/syncing of data to happen exactly when you want it to happen, and then when you're done it doesn't continue until next "sync". So everything you do is local. But then, if "query the feeds the way I want to" is the reason you don't like it, then so be it. But if it's just that you don't understand it, then reading through and digging into the different repositories will give you a better viewpoint. Start at the ssb-query and look what's using it and what it's using.

soapdog 7y ago

Hey, this page is outdated. It is basically a list of APIs from ssb-server and some other related plugins, but there are entries there that are old and haven't been kept in sync with the actual modules. https://scuttlebutt.nz https://scuttlebutt.nz (nee https://ssb.nz https://ssb.nz) is a better link for all things SSB. The protocol guide at https://ssbc.github.io/scuttlebutt-protocol-guide/ https://ssbc.github.io/scuttlebutt-protocol-guide/ is your friend in understanding our little garden. Be aware that SSB grew much like a garden. It is not a protocol and ecosystem designed by committee with a cold and effective process. It grew from simple stuff into more complex stuff, and yes we all understand some of the challenges pilled upon all of us due to bad decisions in the past. There is a lot to love in SSB. Instead of going "npm install, meh", you should try it out. You don't need npm or nodejs to try SSB out, you can just pick any of the clients listed in the first page I linked. I develop one of those clients, patchfox, but it is not a full client so you'll need to bring your own ssb-server.