8 ms·
How to self-host all of Bluesky except the AppView (for now)
- heavensteeth 2y agoThis site is extremely snappy. Good work.
- __justplaying 2y agoThanks! Its code is available at https://github.com/aliceisjustplaying/whtwnd-blog https://github.com/aliceisjustplaying/whtwnd-blog, I intend to turn this into the template as the posts are stored on my PDS, on ATProto, using WhtWnd https://whtwnd.com/ https://whtwnd.com/ (And all of this is a fork of my friend's Samuel's blog, https://mozzius.dev https://mozzius.dev, see https://github.com/mozzius/mozzius.dev https://github.com/mozzius/mozzius.dev)
- apitman 2y agoYour site is stored on your PDS and available on whtwnd, but also hosted directly on your domain. How are they connected exactly?
- __justplaying 2y agoauthor here, should you have questions!
- theschmed 2y agoThanks for making yourself available to answer questions! Hopefully this is not a dumb question. Is plc.directory a single point of failure for BlueSky users who want to take advantage of the benefits of a did:plc? And if so, is that a permanent thing or down the road will there be multiple interoperating did:plc directories?
- __justplaying 2y agoyes it's a SPOF. not sure about the second question, but i do know there are plans to transfer its ownership to an independent foundation
- pfraze 2y agoTransferring to an independent org is what we're talking about now, yes. The backstory to PLC is that we picked up the DID standard and looked for an existing registry-method that would satisfy requirements¹. None of them really did. We then surveyed mechanisms for decentralized operation: DHTs, open blockchains, permissioned blockchains, and federated databases. Of them, the two blockchain variants seemed perhaps promising, but still premature since (as of 2022) you there's cost variability due to load and in some cases bad transaction latency (eg 10 minutes). We decided the best decision was to create PLC, which matches all of the requirements except for longterm meta governance. The way we designed it was to make the registry mechanics transferrable to a different protocol in the future, so that if for instance we decided (say) a DHT was suitable (it's not) we'd be able to use the same identifiers but change resolution and mutations to a new process. Then we started talking to other SMEs to get their take. Ultimately the solution that's gotten the most favorable response has been setting up an ICANN-style independent organization to operate it. This can be joined with a couple of interesting systems, such as mirrors which tail a certificate-transparency-style audit log, and which could even serve as transaction witnesses to indicate when the core registry might be rejecting updates ("write censorship"). What can I say, some things take time and stakeholder-building. Look up the history of DNS and Network Solutions Inc for a bit of a wild ride that people have forgotten about. One other thing I should point out is that the DID spec enables multiple registry methods. Atproto currently supports did:web, and if other methods show up which satisfy the requirements then we are interested. ¹ Secure against manipulation by the registry operators, longterm meta governance, highly available, reasonable transaction latency, reliably low cost that's not dogged by token speculation, low ecological impact.
- jazzyjackson 2y agoHey pfraze, forgive my ignorance but what role does DID serve that DNS doesn't? My favorite part about bsky is using TXT record to prove that I control my domain for username purposes, what's the downside to just generating a keypair, and using the fingerprint of the public key as my identity? (Maybe with some affordance for key rotation vis a vis KERI*) Not doubting youall weighed every possibility, just wondering what I'm missing *Key Event Receipt Infrastructure
- jervant 2y agoHow are Direct Messages implemented in Bluesky if anyone can access a firehose of all network activity?
- __justplaying 2y agoDMs are currently 1:1 only and closed source. They are working on/planning to build proper E2EE DMs that support group chats.
- moreati 2y agoWhat's in that 4.5 TB? e.g. message metadata? Message text? Media? What time window does it cover? A rolling N day window? Everything since year dot? Can it be pruned? e.g. only data of accounts followed or messages interacted with
- mintplant 2y agoWhat's the difference between social-app and the AppView?
- pfraze 2y agosocial-app is the client side, AppView is the backend api surface
- __justplaying 2y agoHow do I ask the mods to swap out the link to the actual post instead of my blog's front page? (...also, the title, as the original has the caveat)
- paulgb 2y ago@dang a better URL would be https://alice.bsky.sh/post/3laega7icmi2q https://alice.bsky.sh/post/3laega7icmi2q (I can't tell if Dan has an alert set up on his handle or whether he just sees everything, but hopefully that works :))
- __justplaying 2y agothanks!
- yorwba 2y agodang doesn't have an alert and he doesn't see everything. https://news.ycombinator.com/item?id=41317232 https://news.ycombinator.com/item?id=41317232 The official way to contact the mods is in the footer, i.e. email hn@ycombinator.com
- __justplaying 2y agowill email, thanks
- paulgb 2y agoAh thanks, good to know. I guess I've just been lucky with it and developed a superstition that it works.
- timerol 2y agoHe is also extremely active here, so there's a good chance he reads and responds to a random comment without an email. But email is the approved (and fastest) way to go about it
- Jtsummers 2y ago
- jonstaab 2y ago[flagged]
- __justplaying 2y agodidn't say it was cheap!
- bombcar 2y agoTen terabytes of spinning rust is only $100-$300 or so, that's not bad at all.
- jonstaab 2y agoMy point is not the current size, it's the eventual size if bluesky succeeds. Facebook ingests 100TB/day. Self-hosting a bluesky relay isn't (won't be) a thing.
- galactus 2y agoIt could be a thing. Not for individual tinkerers but for companies. The fact that today, with already 14 million users, is still possible for an individual to host it is amazing.
- nightpool 2y agoBut why is it required? Do you really need a copy of everyone's data locally? If the only way to self-host bluesky is to have an entire copy of the entire database, that seems like it's really bad from a scaling perspective.
- half-kh-hacker 2y agoWhat else would "self-hosting all of Bluesky" mean other than a copy of the entire site? If you just want to participate in the network host a PDS, which only stores your own posts.
- 98codes 2y agoThis is all academic for me until Bluesky gets the functionality to get an account back onto their main network, for DR if not peace of mind that an "undo" is possible.
- diggan 2y agoTotally understandable. Personally I don't use Bluesky for anything vital, it's just data that the world wouldn't be better/worse without anyways, so I'm gonna go and give it a try even if there is no undo. I love that people even has the choice, so much better than not even being able to.
- sureglymop 2y agoIt's great that you wrote this up! One thing I have found with many open source/selfhostable projects is just how much running them yourself can vary. It can go from a simple compose file with everything included to having to dig for obscure services and piece together how they all form the whole. For example, I recently looked into self hosting Zotero. It is so under documented and complex that there is almost no way one could self host that (even for just one user) without that being ones job. So one needs to make a distinction between something being open source and being feasible to use/maintain. In the end I gave up with Zotero. Even though it could have replaced Obsidian Notes, Calibre and Syncthing all at once for me.
- __justplaying 2y agoSelf-hosting/mirroring all these Bluesky components is currently a mixed bag as well though honestly the only outlier is the Relay, which is a beast. i currently have my copy of the PLC, a Jetstream with 2 days of data and a clone of the app on my laptop i play with sometimes and/or change things for an elaborate shitpost of Bluesky Nitro https://bsky.app/profile/alice.mosphere.at/post/3l7bpmmtiop2y https://bsky.app/profile/alice.mosphere.at/post/3l7bpmmtiop2... I don't self-host my PDS yet because there is no migration path back yet (but there will be). Though maybe I'll just yolo one day and do it anyways.
- diggan 2y ago> For example, I recently looked into self hosting Zotero. It is so under documented and complex that there is almost no way one could self host that I've come across this a lot too. But what I've found is that it mostly applies to open source projects that offer a hosted paid version, so it kind of makes sense they'll make the experience slightly worse than it could be (consciously or subconsciously), as it pushes people to their hosted solution. I don't particularly like it though. Doesn't seem to be the case for Zotero specifically, but your comment reminded me that I've noticed this more often lately.
- sbarre 2y agoYeah I tend to use ease of install for community editions of hosted paid open source projects as the leading indicator of how seriously they invest in (and support) their free/community version..
- ck2 2y agoI found it interesting it's almost impossible, very difficult to get real Bluesky stats This site tries but has limits: * https://bsky.jazco.dev/stats https://bsky.jazco.dev/stats They broke 14 million yesterday and it seems to be snowballing now since the election: * https://bsky.app/profile/jaz.bsky.social/post/3laetwhztdk2x https://bsky.app/profile/jaz.bsky.social/post/3laetwhztdk2x
- __justplaying 2y agohttps://bskycharts.edavis.dev/ https://bskycharts.edavis.dev/ is a good starting point for a number of charts
- jazzyjackson 2y agoIs it feasible to run a bluesky instance "on prem" and "offline" for instance as an airgapped corporate intranet ?
- nisten 2y agoGreat do I have to setup LDAP , oauth, and troubleshoot corporate-style single-signon systems for the next 6 months just to get a chat server running now....
- elfprince13 2y agoI think if you replaced the plc directory with a corporate domain that would be pretty straightforward?
- zzyzxd 2y agoSelfhosting is my hobby but I am also an SRE. I am hesitant to do this because the instruction is "too easy" -- "Simply open your firewall, download and run this installer.sh with sudo on your server and that's it!"[1]. How do I secure the webserver and the data? Where is the data on my disk? How to backup and restore? High availability? There might be detailed documentation somewhere, or I can even read the code. But these are the important things an open source software should tell its users right off the bat. 1: https://github.com/bluesky-social/pds/blob/main/README.md https://github.com/bluesky-social/pds/blob/main/README.md
- dawnerd 2y agoI was about to do this as well but their installer sketched me out. Why can't it just be some easy to follow docker instructions? They use docker too but instructions to set it up on your own is basically "read the installer script". Meanwhile mastodon is incredibly easy to self host w/ relays.
- benharri 2y agoi certainly wouldn't say mastodon easy to self host
- hagbard_c 2y agoPleroma [1] is and does the activitypub thing just as well. I installed it to see if it added anything worth keeping together with the other activitypub things I'm running (Peertube, Pixelfed and Lemmy, the latter two only for testing purposes, the first sees real use on several instances) and can vouch on the ease of getting it up and running. [1] https://docs.pleroma.social/backend/installation/otp_en/ https://docs.pleroma.social/backend/installation/otp_en/
- j45 2y agoAn installer script is often an early step, and much better than nothing... as well as a step towards docker. Here's the kicker, the install script could be called from a Dockerfile pretty easily, no? Sure, there might be things to sort out, but it doesn't seem unreasonable. I agree having a docker image is super handy and can be quick to try, as well as update, and put into a larger self-hosted environment how you need.
- nisten 2y agoIs the actual guide just this <400 word thing, or is it all those 15 different links on the post, or only some of them.... Does that... bureaucracy of documentation not infuriate anyone else or is it just me. I guess I'll try and reset my password to bluesky website, assuming it's this .app one, but then it's asking me to maybe select a provider ... of my password. Does whoemever made this user experience not have enough emotional intelligence realize how infuriating it is?
- __justplaying 2y agoThis was a quick and dirty post I put together primarily for people who are already on Bluesky and have dev experience, and peppered with appropriate links where you have actual guides and/or documentation for each bit.
- steveklabnik 2y ago> I guess I'll try and reset my password to bluesky website, assuming it's this .app one, but then it's asking me to maybe select a provider ... of my password. It's asking what the host of your data is. If you're not running your own server, then the default value of Bluesky itself is the correct one.
- mdaniel 2y agoAlso, yesterday someone posted[1] https://frontpage.fyi/ https://frontpage.fyi/ which seems like it's predominately Bluesky/ATprotocol news but since both of those interest me, if this blog link interests you then so might that link. It logs in with Bsky oauth2 federation 1: https://news.ycombinator.com/item?id=42081210 https://news.ycombinator.com/item?id=42081210
- elfprince13 2y agobut I thought that Bluesky wasn't meaningfully distributed /s
- jazzyjackson 2y agoIf you thought, past tense, you were probably right, but it's been in the oven for 3 years so it's finally approaching "fully baked"
- zxcvbnm69 2y ago[dead]
- jchw 2y agoI appreciate this effort. I've definitely been interested in how plausible it would be to, today, run another instance of the Bluesky AppView, mainly because AT proto seems promising, but to really meet it's full potential it needs independent operators with different sensibilities. I've been thinking a lot about the relay, though. 4.5 terabytes is, well... A lot, to say the least. If Bluesky grows 100x larger, running a relay will become pretty insanely expensive. I guess if the Bluesky organization remains fairly neutral about the relay part, it's not a huge deal, but: - It always eventually becomes hard to stay neutral. Eventually someone will get mad at something going through your network that isn't just obvious network abuse like SPAM. - It seems like drinking from the firehouse itself will eventually become expensive. Will it be possible for something this high bandwidth to remain freely-accessible?