5 ms·

Good overview, I know these sorts of pains well. Lots of hard questions and few definitive wins/right answers. How to organize a massive repository out in the o

by aseipp 4y ago

Good overview, I know these sorts of pains well. Lots of hard questions and few definitive wins/right answers. How to organize a massive repository out in the open is still an open question. On that note, recently, I've been experimenting with this project called josh, which basically is like 'git subtree on extreme steroids, functioning as a git proxy': https://josh-project.github.io/josh/ https://josh-project.github.io/josh/ It basically lets you unify/view many repositories as a single one, or equivalent to split a mono-repo into smaller sized units of work for CI, specific teams, etc. It's bidirectional, so you push and pull from josh and everything goes into a single linear history in the mono repo. And because it's bidirectional, people in the mono-repo can still do things like make large-scale atomic changes across all sub-repositories, and those get reflected. Josh currently isn't suitable for a lot of workloads due to various reasons (authentication is one that stands out), but it's actually the first tool I have seen that manages to offer BitKeeper-like "subtrees" that work really well, at scale, for large repos and teams. It requires some care to make sure "sub-trees" can be usable units of work, but it was one of the best features of BK in my opinion and really great for people doing one-off contributions, or isolating trees/changes to specific developers. I'd be interested to know if there are other open alternatives to this. It's a nice point in the design space between solutions like "integrate with the filesystem layer to do sparse clones" or "just split up the repos."

hirsin 4y ago
Curious why you say this isn't suitable for authN? This seems maximally suited for processes where changes (or more likely, additions) in a split out system (authn, authz) may immediately rely on behavior or interface changes elsewhere.
- mh- 4y ago
  I interpreted that to be an example reason, not an example workload.
- aseipp 4y ago
  I just mean that, I don't see how to properly integrate the josh-server with my SSH keys to do access, how to apply basic roles of who can push/read to what workspace, etc. If I wanted to do this at work "How does it integrate with authentication" is like a #1 concern from an infosec team who'd sign off. If the answer is "It doesn't, use something in front of it" that might be OK too. In fact I think authenticating and authorizing access to components of a monorepo is definitely in scope for its design, and could allow really powerful and cool things. But I just don't think it does that yet. Maybe it shouldn't. That's all I mean. It's not like a hard rule. There were some other things that I thought might hold me up, but I can't remember them now... Not sure if you've ever used it but there's some self-hosted software out there called 'gitolite' that does a lot of this. Perhaps they can be made to work together seamlessly...
  - vlad_ivanov 4y ago
    ACLs are in the works https://github.com/josh-project/josh/pull/561 https://github.com/josh-project/josh/pull/561 That being said, the overall scope of permissions and authentication is more complicated, more improvements is definitely needed
  - chrschilling 4y ago
    The whole topic of access control is way more work than you might expect: There is already an idea and initial implementation of path based ACLs for Josh. However, even if that concept was perfect and already implemented, it would be kind of useless. Why? To be useful in practice we would need a UI for code review in a monorepo. This UI would need to aware of the "workspaces" and ACLs in the repo and respect them when showing files and diffs to the users. As it stands now, Josh is used together with either Gerrit or GitHub (or similar). Patches or PRs are always being reviewed in the context of the full backing monorepo. As long as those are the only options to do code review, I don't see the value of having access control at the Git level. That being said, I am planing to create a new code review tool that does support these things, but it has a long way to go before it will be a serious alternative to the common tools used today.
tedmiston 4y ago
Are you saying you view the project as a curiosity or you have problems that it is actually solving that a git monorepo doesn't?
- aseipp 4y ago
  Not sure if you will see this but, yes, I do see it as solving some problems I have. They aren't necessarily problems a monorepo "doesn't solve", and could have other solutions, I'm just saying Josh is a powerful complement to the monorepo design and helps smooth out some issues. Choosing a monorepo vs many-repos is a tradeoff, there are consequences to choosing a monorepo, many solvable, and a tool like this just provides robust solutions to a couple of them, is what I'm saying. Actually one very common use case Josh solves that I've had in the past is "Merge a repository into another, while developers keep using the original as if nothing happens." This is important to keep teams moving while a migration happens. I have this problem right now at work; two repos that want to be one repo, one smaller and one bigger. You can use git filter-history to do this but Josh is significantly more powerful, and most importantly the team whose repository got "merged away" (i.e. got merged into the bigger repository) can keep working on their repository as if nothing happened, and you can eventually switch them over to the main repo. Normally you have to stop the whole train at once and move people over while some poor bastard has to surgically modify the git repository after they take the git repository down. But Josh allows to you to merge and incrementally migrate that repository, because you can now view one repository as a "workspace" of another. It's sort of like the difference between taking an optimistic lock vs a normal wait lock, in my mind. Josh lets you do "optimistic locking" when merging two repositories, rather than making every team stop -- serializing -- while it happens. Another common case is "I need to mirror a subset of my proprietary repository onto GitHub." Actually "I need to mirror subset X" in general. This is another monorepo problem that, while not super complex, is actually really nice to solve in this way because bidirectionality means people who patch the mirror downstream can still have their changes merged upstream. This isn't always possible for QA/workflow purposes (e.g. their downstream change could break something upstream, so many people choose to instead apply it the other way around), but it's something I've experimented with. Not unthinkable. I do think it's a very promising project with many real world applications. I'm still figuring out how best to organize and use it, just like we do with git.
tedmiston 4y ago
https://github.com/josh-project/josh https://github.com/josh-project/josh

by aseipp 4y ago

hirsin 4y ago

Curious why you say this isn't suitable for authN? This seems maximally suited for processes where changes (or more likely, additions) in a split out system (authn, authz) may immediately rely on behavior or interface changes elsewhere.

mh- 4y ago
I interpreted that to be an example reason, not an example workload.
aseipp 4y ago
I just mean that, I don't see how to properly integrate the josh-server with my SSH keys to do access, how to apply basic roles of who can push/read to what workspace, etc. If I wanted to do this at work "How does it integrate with authentication" is like a #1 concern from an infosec team who'd sign off. If the answer is "It doesn't, use something in front of it" that might be OK too. In fact I think authenticating and authorizing access to components of a monorepo is definitely in scope for its design, and could allow really powerful and cool things. But I just don't think it does that yet. Maybe it shouldn't. That's all I mean. It's not like a hard rule. There were some other things that I thought might hold me up, but I can't remember them now... Not sure if you've ever used it but there's some self-hosted software out there called 'gitolite' that does a lot of this. Perhaps they can be made to work together seamlessly...
- vlad_ivanov 4y ago
  ACLs are in the works https://github.com/josh-project/josh/pull/561 https://github.com/josh-project/josh/pull/561 That being said, the overall scope of permissions and authentication is more complicated, more improvements is definitely needed
- chrschilling 4y ago
  The whole topic of access control is way more work than you might expect: There is already an idea and initial implementation of path based ACLs for Josh. However, even if that concept was perfect and already implemented, it would be kind of useless. Why? To be useful in practice we would need a UI for code review in a monorepo. This UI would need to aware of the "workspaces" and ACLs in the repo and respect them when showing files and diffs to the users. As it stands now, Josh is used together with either Gerrit or GitHub (or similar). Patches or PRs are always being reviewed in the context of the full backing monorepo. As long as those are the only options to do code review, I don't see the value of having access control at the Git level. That being said, I am planing to create a new code review tool that does support these things, but it has a long way to go before it will be a serious alternative to the common tools used today.

tedmiston 4y ago

Are you saying you view the project as a curiosity or you have problems that it is actually solving that a git monorepo doesn't?

aseipp 4y ago
Not sure if you will see this but, yes, I do see it as solving some problems I have. They aren't necessarily problems a monorepo "doesn't solve", and could have other solutions, I'm just saying Josh is a powerful complement to the monorepo design and helps smooth out some issues. Choosing a monorepo vs many-repos is a tradeoff, there are consequences to choosing a monorepo, many solvable, and a tool like this just provides robust solutions to a couple of them, is what I'm saying. Actually one very common use case Josh solves that I've had in the past is "Merge a repository into another, while developers keep using the original as if nothing happens." This is important to keep teams moving while a migration happens. I have this problem right now at work; two repos that want to be one repo, one smaller and one bigger. You can use git filter-history to do this but Josh is significantly more powerful, and most importantly the team whose repository got "merged away" (i.e. got merged into the bigger repository) can keep working on their repository as if nothing happened, and you can eventually switch them over to the main repo. Normally you have to stop the whole train at once and move people over while some poor bastard has to surgically modify the git repository after they take the git repository down. But Josh allows to you to merge and incrementally migrate that repository, because you can now view one repository as a "workspace" of another. It's sort of like the difference between taking an optimistic lock vs a normal wait lock, in my mind. Josh lets you do "optimistic locking" when merging two repositories, rather than making every team stop -- serializing -- while it happens. Another common case is "I need to mirror a subset of my proprietary repository onto GitHub." Actually "I need to mirror subset X" in general. This is another monorepo problem that, while not super complex, is actually really nice to solve in this way because bidirectionality means people who patch the mirror downstream can still have their changes merged upstream. This isn't always possible for QA/workflow purposes (e.g. their downstream change could break something upstream, so many people choose to instead apply it the other way around), but it's something I've experimented with. Not unthinkable. I do think it's a very promising project with many real world applications. I'm still figuring out how best to organize and use it, just like we do with git.

tedmiston 4y ago

https://github.com/josh-project/josh https://github.com/josh-project/josh