30 ms·
Bluesky migrates to single-tenant SQLite
- pvg 3y agoThis seems like a very misleading title, the Bluesky PDS is the meant-for-selfhosting thing they distribute, not the bluesky service as experienced and used by most of its users.
- wmf 3y agoAFAIK there's only one version of the software so "the service" runs the same thing that you self-host. SQLite seems like it will simplify the single-user case though.
- jakebsky 3y agoThat's right. This is the same code Bluesky is running on our new PDS hosts. It's all open source. The main motivation in moving from a big central Postgres cluster to single tenant SQLite databases is to make hosting users much more efficient, inexpensive, and operationally simpler. But it's also part of the plan to run regional PDS hosts near users, increasing performance by decreasing end-to-end latency. The most experimental part of this setup is using Litestream to replicate these many SQLite databases (there are almost 2 million user repositories) to cloud storage. But we're not relying on this alone, we're also going to maintain standard SQLite ".backup" snapshots as well.
- whyrusleeping 3y agono this actually is moving every single user currently on the service into this setup. Everyone gets their own sqlite under the hood.
- typical182 3y agoThe “Personal” in PDS doesn’t mean it is only for self-hosting. Bluesky has a main PDS instance at https://bsky.social https://bsky.social that serves almost all of the Bluesky user base. There is a good overview of the architecture here: https://blueskyweb.xyz/blog/5-5-2023-federation-architecture https://blueskyweb.xyz/blog/5-5-2023-federation-architecture Here’s a snippet from the protocol roadmap they published 3-4 weeks ago [1]: Multiple PDS instances The Bluesky PDS (bsky.social) is currently a monolithic PostgreSQL database with over a million hosted repositories. We will be splitting accounts across multiple instances, using the protocol itself to help with scaling. [1] https://atproto.com/blog/2023-protocol-roadmap https://atproto.com/blog/2023-protocol-roadmap
- deleted 3y ago[deleted]
- focom 3y agoI am curious, does the HN folks know if bluesky is more active than nostr or the mastodon network?
- BryantD 3y agoLess active than Mastodon, more active than nostr. https://vqv.app/stats/chart https://vqv.app/stats/chart is useful for Bluesky and draws on the Bluesky firehose for data. https://stats.nostr.band/ https://stats.nostr.band/ seems useful for nostr.
- muglug 3y agoLess active than Mastodon, I'd assume more active than Nostr. But the interesting thing for me isn't activity — it's the people on there. Of the cohort who had >100k followers on Twitter, I think more of them post regularly on Bluesky than post on Mastodon. Bluesky definitely has a more cohesive feel, especially because there's currently just one instance & mod team.
- spookie 3y agoMastodon, and the Fediverse in general, make user interaction decisions on purpose to limit many of the issues common to social media. Think about: mob culture, addiction, and the like. I wonder if BlueSky intends to follow on those. For example, hiding user actions counts (repeats, favourites, etc...) until the user acts on one. Things like these may be strange for those accustumed to Twitter, but personally, is what makes me stick with smaller instances on the Fediverse.
- indigochill 3y agoMy bet is regardless of any initial good intentions, since BlueSky is a company, market pressures will inevitably force them into dark patterns like we see on every other commercial social network (going back to the early days of the companies, Facebook, Twitter, and even Google looked really good early on until all were corrupted by profit motive). My belief is that the profit motive is necessarily at odds with free communication. To me, the ActivityPub network (Mastodon and friends) is relatively unique in the social media space in having no direct commercial pressures (the protocol is developed by W3C) and therefore being inoculated against the causes for these dark patterns.
- lukevp 3y agoInteresting... I like the strategy of having each user be 1:1 with a DB. What would be done for data that needs to be aggregated across users though? If I'm subscribed to another user and they post, how does my DB get updated with that new post? Or is this meant just for durable data and not feed data (like profile data, which users are followed / not followed / etc.) and all the interactive stuff happens separately? I like that "connection pooling" is just limiting the number of open handles in a LRU cache. It's also interesting because instead of having to manage concurrency at the connection level, it handles it at the tenancy level since each DB connection is single-threaded. You could build up per-DB rate limiting on top of this pretty easily to prevent abuse by a given user. Is there a straightforward way to set up Litestream to handle any arbitrary number of DBs?
- wmf 3y agohttps://blueskyweb.xyz/blog/5-5-2023-federation-architecture https://blueskyweb.xyz/blog/5-5-2023-federation-architecture
- Retr0id 3y agoTo summarise the relevant details, the "AppView" service is responsible for the sorts of queries that aggregate across users, and that has its own database setup - I think postgres but I'm not 100% sure on that.
- jakebsky 3y agoYou're right, as usual. AppView is on a Postgres cluster with read replicas doing timeline generation (and other things) on-demand. We're in the process of moving it toward a beefy ScyllaDB cluster designed around a fanout-on-write system. The v1 backend system was optimized for rapid development and served us well. The v2 backend will be somewhat less flexible (no joins!) but is designed for much higher scale.
- manmal 3y agoDoes the BGS pull all the tenant‘s individual SQLite data? Or do the PDS push new posts to the BGS?
- xrd 3y agoIs Bluesky still invite only?
- nvy 3y agoYes. I have invite codes if you would like one. Email in my profile. Edit: they're all gone!
- lucb1e 3y agoWas browsing around your website (mentioned in profile), noticed https://0x85.org/contact.html https://0x85.org/contact.html only mentions Twitter and email. Maybe the bluesky omission is intentional, but probably it just hasn't been updated yet? I'm not on bsky myself, currently having fun on mastodon and I'm not familiar with bsky enough to know what I'm missing out on, but for other folks I figured I'd mention it
- nvy 3y agoHey thanks. It's just outdated, what with young kids and grad school. Appreciate the note.
- JosephK 3y agoIs your offer invite code available to other randoms like myself? I tried to register on bsky months ago and still haven't been approved.
- defatigable 3y agoI have some extra if you'd like one. Let me know how to get it to you and I will.
- azemetre 3y agoIf you're offering I'd love one. My email is my username on hackernews at gmail.com
- malkia 3y agoWhat do they mean by "Since SQLite does not support concurrent transactions" - it supports them, as long as you don't access the .db file through a file share (UNC, or NFS, etc) - https://www.sqlite.org/wal.html https://www.sqlite.org/wal.html I've been using this to update/read db from multiple threads/processes on the same machine. You can also do snapshotting with the sqlite backup API, if you want consistent view, and to not hold on transaction (or copy in-memory). But maybe I'm missing something here... Also haven't touched sqlite in years, so not sure...
- sinkwool 3y ago> Writers merely append new content to the end of the WAL file. Because writers do nothing that would interfere with the actions of readers, writers and readers can run at the same time. However, since there is only one WAL file, there can only be one writer at a time. I think the OP meant that updates have to run sequentially.
- starcraft2wol 3y agoWhich just means the lock happens at user scope in this case instead of per table or row. This limitation still causes so much confusion when it’s a completely reasonable design.
- mikrotikker 3y agoI've been importing data to sqlite databases running being actively written to for years. Just throws exception if the database is locked and I retry. Do 10k row batches, with a small sleep between. No issues. Helps if your use case doesn't really care about data being in order I guess.
- dilyevsky 3y agoProbably means there’s still no row-level locking at least last I checked, very limited table-level locking. The writer still grabs a lock on entire db per docs
- manmal 3y agoDid you disable auto checkpointing? Wouldn’t checkpointing result in potential corruption or at least data loss if two processes do that simultaneously? Or is that scenario exhaustively prevented with a lock file?
- lucasyvas 3y agoLove SQLite - in general there are many challenges with a schema or database per tenant setup of any kind though. Consider the luxury of row-level security in a shared instance where your migration either works or rolls back. Not now! If you are doing a data migration and failed to account for some unexpected data, now you have people on different schema versions until you figure it out. Now, yes, if you are at sharding scale this may occur anyway, but consider that before you hit that point, a single database is easiest. You will possibly want to combine the data for some reason in the future as well. Or, move ownership of resources atomically. I'm not opposed to this setup at all and it does have its place. But we are running away from schema-per-tenant setup at warp speed at work. There are so many issues if you don't invest in it properly and I don't think many are prepared when they initially have the idea. The funny thing is that about a decade ago, the app was born on a SQLite per tenant setup, then it moved to schema per tenant on Postgres, now it's finally moving to a single schema with RLS. So, the exact opposite progression.
- starcraft2wol 3y ago> now you have people on different schema versions until you figure it out. That can be a good thing if your product has say < 100 customers. As each might have different upgrade timelines and needs. I even know of business like this who do custom work for some so they essentially aren’t even running the same code (gasp). I guess it depends on the business structure.
- lucasyvas 3y agoTotally correct. But not a good thing in our case!
- KRAKRISMOTT 3y agohttps://blog.turso.tech/introducing-embedded-replicas-deploy-turso-anywhere-2085aa0dc242 https://blog.turso.tech/introducing-embedded-replicas-deploy... https://electric-sql.com/ https://electric-sql.com/
- dilyevsky 3y agoI dont know - I have experience working with monster DBs in production and never again. Under large enough load every change becomes risky because you can’t test performance corner cases fully. Having a free-tier user take out your prod because they found a non-indexed code path is also classic
- scorpion_farmer 3y agoThat looks like the PR from hell - 190 files changed, 143 commits? Mostly with names like "tidy" and "wip" Props to whoever actually reviewed that, you are a warrior
- 4death4 3y agoWhat if it was 190 files changed in 1 commit, would that make a difference?
- rgoulter 3y agoIt might. With commits like "typo", you might as well squash these into the commit which introduced the typo in the changeset. If there are changes across many files, and the changes were made automatically with some search-and-replace (or some refactoring tool).. by having a commit that's only that automatic change, it's easy to look at that commit and tell what the changes were. -- Presumably, non-automatic changes are going to be smaller. I guess roughly, if it makes sense to apply a changeset that changes 5 things, you'd want 5 commits. Having commits like "typo" means there are more commits; but squashing those 5 things together makes it harder to discern the granular change.
- numbsafari 3y ago> Props to whoever actually reviewed that, you are a warrior Or a ghost.
- lrx 3y agoI prefer to read the unified diff and commits don't matter as much.
- tehlike 3y agoThis is the answer.
- readline_prompt 3y agodon't know why, but recent teams around me have always made strict rules about number of commits in PRs. I just wanted to tell them the same thing you said: "Why don't you just look at the diffs?" curious for other opinions. (sorry not really about this particular topic)
- deleted 3y ago[deleted]
- paxys 3y agoCool, but maybe let people actually use your service before everyone forgets what it is?
- nyx2d 3y agoThey have over 1.8 million users currently, or do you mean PDSes specifically? Federation is in open beta on a test network, you can try it out today if you'd like.
- raverbashing 3y agoCome on, "over 1.8 million users" is not an impressive number These kind of movements makes me think they're not serious about scaling up. Wouldn't surprise me if then end up as an also-ran
- pmontra 3y agoMaybe not impressive but none of the services of my customers had or has 1.8 million users. And yet they do well (my customers.)
- rapnie 3y agoThat. And none of the big social media platforms were big at the start either.
- raverbashing 3y agoYes but 1.8Mi at a time where people are longing for a Twitter alternative is just leaving money on the table
- pmontra 3y agoThere are two usual strategies for growing: 1) low cost, organic and slow or 2) high cost, throw a lot of money at advertisement, saturate all media, grow quickly or bust. The exceptions are those rare products that despite a low cost marketing sell themselves so well that their organic growth is fast and in a few months everybody use them. Maybe Bluesky don't have the money to advertise or is not compelling enough. As one data point: I know about Mastodon but I think that I learned about Bluesky only today. I went to their site and there is nothing to explain how it works except that it's some social thing. I learned more by reading the comments here. Apparently it's being marketed at a very low cost.
- winrid 3y agoWhy sha256 hash the user into to get a two character target directory? Wouldn't md5 be much faster and solve the same problem?
- paulryanrogers 3y agoAt their scale maybe they're worried about collisions? Or, like me, they're drowning in security tooling from corporate and don't want to have to carve out exceptions for md5 usage in each.
- RaisingSpear 3y ago> At their scale maybe they're worried about collisions? With their scheme, collisions are already guaranteed to happen if they have >256 users.
- fodkodrasz 3y agoI guess parent meant abusable non-uniform distribution of collisions (they have collisions anyway as the take only the first two characters according to GP comment)
- winrid 3y agoIt could be they didn't want to explain the md5 usage, yeah. But that's kinda nuts if they do this every query.
- wmf 3y agoIt's probably not healthy to have broken cryptographic hashes running around. If you don't need a secure hash there are plenty of fast non-cryptographic hashes.
- winrid 3y agoThere's nothing about security here. By this logic you should probably stop using hashmaps, then? :)
- PrivateButts 3y agoI've got a bunch of invites if folks want them: bsky-social-etdu7-njigu bsky-social-2ktcs-uwoxg bsky-social-6f5nh-36gnq bsky-social-ciwro-3gzk5 bsky-social-y4h57-dxh3g
- thegeekpirate 3y agoGrabbed bsky-social-6f5nh-36gnq, thanks!
- pests 3y agoDamn, they all gone.
- ochronus 3y agoThere you go folks: bsky-social-h3d4w-u6yn4 bsky-social-74bqi-vkmcq bsky-social-n3fdq-46nxz bsky-social-yippe-32vdr bsky-social-l2fbt-xnscx
- shmde 3y agoGone in 60 seconds.
- arthurcolle 3y agoAny more?
- mythz 3y agoAlways happy to see more server SQLite/Litestream adoption which we've also been using to build our new Apps with. SQLite + Litestream is an even greater choice for tenant databases, that's vastly cheaper to replicate/backup to S3/R2 than expensive cloud managed databases [1] (up to 3900% cheaper vs SQLServer on Azure). [1] https://docs.servicestack.net/ormlite/litestream https://docs.servicestack.net/ormlite/litestream
- patates 3y agoWhat does 3900% cheaper mean? I don't get it.
- vizzah 3y agoyeah.. a weird way to say 39 times cheaper ;)
- throwaway167 3y ago100% of, say 42 is 42. So 100% less than 42 is 0. 3900% cheaper makes no sense.
- deleted 3y ago[deleted]
- andrewstuart 3y agoI sure hope they don’t ever want to change their db structure. Why not use Postgres with RBAC (Row Based Access Control).
- dathinab 3y ago- simpler db client - simpler cloud architecture - simpler resource management - simpler partial backups/restore - simpler compliance with law enforcement - partitioning might be easier, e.g. when handing "user account storage which should be undo-able for a while" (e.g. long term absent users data could be moved to cold storage, blocked/deleted users data could move to some scheduled for deletion space allowing undoing it for a while but then reliable auto deleting them, a copy of users data where crime detection triggered (e.g. CASM) could be moved to a quarantine space, etc. And each of the spaces can be completely different servers with different storage methods and retention policies, virtual access control and physical access control. Sure you can have all of that with RBAC + partitioning + triggers + roles in postgres, but it's the personal data store of a user so you don't need cross users FK constraint enforcement and it makes it much easier to make sure you don't miss anything wrt. access controll or forgetting to partition/move some columns of a new table etc.) - maybe simpler billing for storage ("just" size of DB) now simpler doesn't mean better, but often it pays of as long as you don't run into the limits of what is possible with the simpler architecture (and as far as I can tell you can shard this approach really nice, so there at least there shouldn't be scaling performance limits, scaling cost and future feature complexity limits might still apply)
- andrewstuart 3y agoYou didn’t systemically document “harder”.
- hknmtt 3y agogood luck with running updates.
- victorbjorklund 3y agoCan someone that knows more about bluesky explain what data is stored in sqlite and not? Because i assume it isnt messages etc between users.
- williamcotton 3y agoI assume that messages between users are stored in those SQLite DBs. Think email. When you send an email and CC five other people as well then seven people now have the same copy of the email stored on their email servers. That is, there’s no central database that contains a single email that is referenced by others. This is basically how sharding with relational DBs works as well. This sort of data denormalization is almost a requirement as applications scale and especially for many-to-many applications that have a high write to read ratio. Low write to read and you can get away with a single master to many slave relational DB architecture for quite astonishing numbers of requests and data!
- Hamuko 3y agoBy messages, do you mean direct messages (private messages between two parties)? Because Bluesky doesn't have those at the moment. There's only public messages broadcast to the world. Haven't done any research to determine if there are plans for direct messages.
- BigTuna 3y agoIt's all your posts and replies as a user. While they currently host the only* PDS themselves, the end goal is for every end user to have their own PDS. Inrupt/SOLID calls this concept a "pod". *(actually they just onboarded a second production PDS yesterday.. progress!)
- anonyfox 3y agoSlightly related: is Bluesky moderated good enough or do I get lots of rightwing and conspiracy crap like on twitter currently? I‘d really love to have some more civilized hub again that isn’t full of hate and anti-intellectualism.
- elAhmo 3y agoI think it is still too small and people seem quite nice there. But, that has its drawbacks, as I keep returning to Twitter due to the slow migration in the recent months. It is a shame, as it seems like a nice alternative that has some cool ideas.
- vehemenz 3y agoIt's anti-intellectual and uncivilized, but not because of rightwing conspiracy content. There is a strong culture of intolerance and censorship of viewpoints that diverge from the norm.
- _heimdall 3y agoAre you assuming that hate and anti-intellectualism are exclusively a rightwing thing?
- rchaud 3y agoOn Twitter, the place that hired Tucker Carlson after Fox News dumped him? Yeah it is. No need for "both sides"-ing on this one.
- _heimdall 3y agoOne example is hardly proof of an absolute though. Assuming all conspiracy or anti-intellectual thought comes from one side because Tucker Carlson is a huge logical misstep in my opinion
- rsynnott 3y agoNot exclusively, but on the mainstream internet in 2023? Yeah, more or less, bar a few tankies.
- gigatexal 3y agoAt a previous fintech role the company would store customer accounts as encrypted sqlite3 files on blob storage ... this worked out decently well for our access patterns.
- gigatexal 3y agoThis should make leaving the service rather simple. Download your sqlite file and throw up a simple local-only html front end to the data and you're solid.
- glub103011 3y ago[dead]
- tomashubelbauer 3y agoAnyone interested in joining Bluesky, please grab these. I have extra and I've already invited all my Twitter mutuals I wanted to invite. Edit: I'm all out now :)
- deleted 3y ago[deleted]
- da25 3y agoAll used :( Do you have anymore ?
- rapnie 3y agoWill the BGS also be federated, or is that to be the centralized big spider in Bluesky's web?
- fiatjaf 3y agoIn theory you can migrate between BGSes, but you can always just use one at any point in time. In practice no one will switch because it makes no sense to do it. If there happen to ever be more than one real BGS contender, it will be from something like Cloudflare that will just replicate everything Bluesky Inc decides.
- rapnie 3y agoI don't know if it does not make sense. AFAIU these BGSes could be special-purposed e.g. for a business, community or topic of interest. Why wouldn't it make sense to synchronise the collected data between these BGSes and get a combined view on the data? With just a single BGS we have another centralized big tech platform. I think decentralized BGSes are a major factor in how interested people are in becoming part of the ecosystem.
- jakebsky 3y agoThe BGS is a "dumb" relay and mirror of the network, so it generally shouldn't matter which one your client app is ultimately sourcing data from. But yes, anyone is free to operate a BGS. It does necessarily require a non-trivial amount of storage, compute, and bandwidth. A funded startup, well-funded non-profit, or any just about any cloud provider could likely afford to run one. It's also entirely possible to operate a BGS that only mirrors a slice of the network (for instance, only users in one country) if desired, which could in some cases make it affordable for a single user or small coop to operate.
- PurpleRamen 3y agoOn the surface, this looks like the worst combined with the awful. I hope someone will make a good article with some hard numbers to explain the benefits and analyze the assumed flaws, because this could be something really, fascinating to learn about.
- barnabee 3y agoCan you explain why this looks like the "worst combined with the awful" to you? To me, on the surface, particularly assuming you are building a distributed system to be run and deployed by many users, some of which are not professional sysadmins (which I believe is likely to be a goal here, and should be), this seems like quite a sane choice. I'd definitely expect a design goal to be avoiding the need to setup/configure/look after any additional database or other servers.
- PurpleRamen 3y agoThis looks like someone is building their own filebased database-system, in typescript, while still using mature features of database-servers. So instead of trusting the optimized, regularly maintained and battletested solution, they build something by themselves. This smells ugly, like something that will scale poor in performance, and will have security and tooling-problems. Simplification of installation seems not like a good enough reason to trust your whole backend on this. Installing and maintaining a database-server is not that hard today. This is well established and documented, unlike this. But I also don't know enough about this app, maybe this is just one of several options, meant for a specific usecase? Using this in a standalone desktop-app would make sense, while still offering a mature sql-backend available for server-installations.
- muttled 3y agoI'm not a professional coder, only side-projects. Never formally taught. I looked at the solution and thought it kinda sounds like something I'd come up with. Like when I didn't know how to use data tables and would hold data in an array of arrays to form the rows and columns. Somewhat clever, "works", but would probably make my professional coder friends vomit if I explained it to them.
- m3kw9 3y agoThat sounds like centralizing
- dbmnt 3y agoIf anyone needs a Bluesky invite, there are three in the About section of my profile page here (note: you will need to prepend "bsky-social-" to the code).