7 ms·
Scaling Bitbucket’s Database
- doubleunplussed 7y agoStill salty about them dropping mercurial support without providing any migration tools, and planning to just delete data without any archiving.
- meritt 7y agoI'm also salty. What's your plan? I'm either gonna migrate to git (and use a different host), or self-host something like hetapod.net
- bjoli 7y agoI went to sourcehut. I am very happy, but during the process I ended up actually having to learn a bit of git. Future projects will probably be git.
- patd 7y agoGitHub’s import feature worked well for me. You’re obviously going to lose the wiki, issues etc. But at least you’ll keep your repo history, branches, ...
- doubleunplussed 7y agoYeah I'm moving to git and github. I always pteferred mercurial for what I maintain are good reasons not just of personal taste, but I've been swimming against the tide for too long and it makes sense to go with git and github. If github goes downhill, at least I'll be in good company next time!
- compumike 7y agoAlso salty! For hg->git migration, https://github.com/frej/fast-export https://github.com/frej/fast-export worked smoothly for me. mkdir new-git-repo cd new-git-repo git init fast-export/hg-fast-export.sh -r ../path-to-old-hg-repo (probably want to translate .hgignore to .gitignore and commit it) git remote add origin git@path_to:git_server git push -u origin master git push --all
- johannes1234321 7y agoYes, it is fairly easy to convert the repo, but 1) this afaik couldn't be done inside the BB project, so one would have had to create a new repo and lose bug history etc. 2) It's a trust thing. If BB kills repos once, who doesn't say they won't kill others tomorrow? 3) Comfort: Import to GitHub is a click away
- michaelhoffman 7y agoSame here. I don't love that they aren't going to support Mercurial anymore. But deleting all our existing projects without providing a path for us to migrate or archive? I can't trust them anymore.
- adamfeldman 7y ago"Bitbucket uses PostgreSQL with one primary read-write database and N read-only replicas."
- loic-sharma 7y agoThat's really neat! I wonder, could you store the PostgreSQL log sequence numbers in a cookie to save the round-trip to Redis?
- nhoughto 7y agoI thought this too, but that then creates a likely unreasonable burden on the client. Not all user agents expect cookies either, think of all the varieties REST clients. Sounds like they would also need to internally reference the lsn too, so being able to look it up internally would be valuable. Feels like there is something there tho in terms of approach, esp since they are keying it on user id.
- nihil75 7y agoYea I'm not taking advice from a service that's down once a week
- izacus 7y agoDid you actually attempt to read the article or did you just come here to spew out needless negativity?
- chairmanwow1 7y agoHave you ever actually used Bitbucket in production? It has absurd latencies for every operation from website to even a git push taking full seconds to respond. It’s hard to take them seriously when that’s the tableau upon which they are advising us to build systems.
- skunkworker 7y agoA number of years ago I had all of my repos on bitbucket, but recently I’ve pulled everything as their peering can impact large git pulls and were unreliable. The transition with the account changeover was enough to make me jump ship to github completely.
- nihil75 7y agoThis.
- YawningAngel 7y agoI don't think it's needless. I think the fact that Bitbucket is profoundly second-rate in is quite relevant when discussing their fluffy blog posts about engineering.
- pm90 7y agoI’m not a big fan of Atlassian in general but the blog post provides a reasonable solution to an engineering problem. It doesn’t seem like a fluff piece.
- zzzeek 7y agoBitbucket (mostly via Atalassian) worked very hard to chase everyone off their platform in the vain hopes that the OSS community would start using Jira. Clearly they bet wrong, but unfortunately they aren't going to get any users back for a very long time.
- alexis_fr 7y agoFree private repos is a real upside. Central login where you can never understand what login you are using and what information they are collecting, nor on which website you are not whether you’ve suddenly integrated Confluence into a Bitbucket instance, is a real downside. God I hate SSO’ed companies, they went all the way of Google into Youtube.
- zzzeek 7y agogithub offers these now for individual accounts.
- agumonkey 7y agothis reminds me that I used to use bitbucket for the private repo and I just hosted some app code on .. github private.
- detaro 7y agoFree private repos are an upside over ... Github, which gives free accounts unlimited private repos? Gitlab, which does the same?
- Cyph0n 7y agoTo be fair, Github only added that recently.
- deleted 7y ago[deleted]
- progval 7y ago
- lykr0n 7y agoCurious that they use LSN tracking vs making the replication synchronous. I would be curious to see the performance numbers and the reasoning behind the choice.
- beoberha 7y agoSynchronous replication really slows down write performance and adds in the complexity of needing to determine if a replica is down. I think very few people use it in practice.
- Xorlev 7y ago+1, sync replication also reduces availability: any replica downtime becomes write unavailability until you decide that replica is "dead" and stop replicating to it. There's also hybrid setups of sync+async for durability reasons, sync to a single replica so that if your primary db suddenly catches fire you have a replica that's as up to date as possible, then read replicas using async replication.
- qeternity 7y ago10ms to do a redis get? What am I missing here...
- Xorlev 7y agoThey said that was their time budget, but didn't follow up with how expensive it ended up being in practice -- but agreed.
- qeternity 7y agoThe article says they were able to keep it close to their max budget, which actually suggests that it was higher than 10ms in practice
- avremel 7y agoAnd presumably query the replicas until an up-to-date one is found (based off the LSN of the user).
- qeternity 7y agoI’d have expected the LSN of the replicas to be in redis as well, but fair point. However even as a worst case given their number of replicas, I’d expect to be able to hit redis AND each replica in under 10ms
- ngrilly 7y ago> we find a replica that is as up to date as the user's saved LSN The article doesn't explain how they know the current LSN of each replica?
- avremel 7y agoIIUC they need to query replicas until they find one that is up to date. "We have to query Elasticache and replicas, which adds latency."
- rosstafarian0 7y agoIs it just me or does looking at write logs to determine read consistency just seem like a really bad way to do this vs a different load balancing strategy?
- hilbertseries 7y agoDefinitely feels like bandaid, over a more robust solution like sharding their main database by repository. Or splitting off some of their tables to another database. Their load of "millions of requests per hour", doesn't sound particularly high either. At some point their primary is going to get overloaded again by the writes. And they'll have added all of this machinery for nothing. They've also made the replicas an essential part of their stack. Whereas their previous stack could probably tolerate a certain amount of replicas going down or replica lag. They now have a system that will grow and be dependent on all the replicas being available. Until a heavy write load or heavy table alter, causes the replicas to become lagged at which point a higher percentage of traffic will go to the master and potentially cause downtime.
- pm90 7y agoYikes. Excellent point. Is sharding the only option here? It seems like having immensely shared Postgres is one way of solving the problem, but is that the only option? Wondering if eg cockroachDB would be a good fit
- hodgesrm 7y agoThis is a very common solution. The biggest advantage is that it allows you to use read replicas which are there for HA and otherwise idle.
- deleted 7y ago[deleted]
- saga92 7y agoOracle offers DML redirection for active dataguard setup where a write operation on the replica database is redirected to the primary database to allow applications that make infrequent writes to actively run on the Active Data Guard replica database. Also the write operation completes when the replica has seen the write from the primary thereby eliminating the race condition avoiding such complex reengineering
- rosstafarian0 7y agoA lot of databases have a concept of consistent reads. Even DynamoDB will allow you to specify consistent reads explicitly on a query and writing through DAX will take care of the scenario as you described as the read cache is immediately updated. This a common scenario.
- halestock 7y agoI found it interesting that their cloud version of bitbucket uses Django while their self-hosted stack is almost entirely Java.
- mithun 7y agoThat's because they bought Bitbucket. The self hosted version was internally developed and used to be called Stash, before they rebranded.
- caseyf7 7y agoDoes storing the LSNs in Redis and Elasticache create the same race condition they are trying to avoid or am I missing something?
- sedatk 7y agoStep 1: Remove Mercurial support.
- cryptonector 7y agoBitbucket is a perfect case for sharding. You could have one DB of users, and N DBs of user data, with the first referencing the latter, and then distribute user data among the latter. You could do this straight up with PostgreSQL without any special server-side software. You could also have a PG server using FDW to act as a proxy for all the DBs a client needs to be talking to. There are many other options too.
- hinkley 7y agoI am architecting a project that is at a smaller scale but bears some resemblance, and I’ve already settled that I will be very careful about GET vs POST/PUT semantics and route all traffic of the former only to the replicas and all of the latter through a connection pool to the primary. That, I expect, will hold me well into >1% of their traffic. I’m not sure if I’ve missed something with their LSN work or if this indicates that the GET semantics horse has already escaped their proverbial barn. Within the call-response of a single request, which they seem to be talking about, none of this should be necessary. Right? However a cascade of narrowly-spaced follow-up requests could easily catch you in this trick. Lie. Whatever you wish to call it.
- ykevinator 7y agoIt's amazing that Atlassian still exists.