5 ms·

Josh: Just One Single History

91 points by rapnie 3y ago

wbl 3y ago
I was unaware that people who used perforce and git would go decide that git needed to work like perforce.
skybrian 3y ago
To sum up, it looks like this is a server that provides smaller views of a large monorepo. A simple case is checking out a subdirectory of the repo and just getting the files and history for that subdirectory. They also advocate using these views to do builds. That way, your build can't depend on any files that are outside its view. It's a kind of sandboxing for builds. [1] > In particular when using C family languages, hidden dependencies on header files are easy to miss. For this reason limiting the visibility of files to the compiler by sandboxing is pretty much a requirement for reproducible builds. > With Josh, each deliverable gets its own virtual git repository with dependencies declared in the workspace.josh file. This means answering the above question becomes as simple as comparing commit ids. Furthermore due to the tree filtering each build is guaranteed to be perfectly sandboxed and only sees those parts of the monorepo that have actually been mapped. > This also means the deliverables to be re-built can be determined without cloning any repos like typically necessary with normal build tools. Bazel users might find this familiar. It does a similar thing by running build tools in a temporary directory that contains symlinks to your actual source code. But it's not something you actually need if you're not using a monorepo, since your package manager won't pull in other packages that you don't depend on. [1] https://josh-project.github.io/josh/usecases.html https://josh-project.github.io/josh/usecases.html
scubbo 3y ago
It's _amazing_ how much work monorepo fans put into "making a monorepo operate like a polyrepo", when they could just...
- danpalmer 3y ago
  I sort of get it. I don't think anyone coming from a multi-repo world really understands the full implications of a monorepo until they've worked in a large scale one (e.g. those at Google, Meta, or other places), so I think they try to keep certain kinds of practice that would ideally change in a monorepo world. And I also don't think Git scales to that (Microsoft just about made it work with a custom filesystem, Google and Meta both use custom source control, Meta's is based on Mercurial), and therefore it's often necessary to split the monorepo into many repos and there exist a lot of tools for treating a set of Git repos as one Monorepo.
  - scubbo 3y ago
    > I don't think anyone coming from a multi-repo world really understands the full implications of a monorepo until they've worked in a large scale one That's entirely fair. My sole experience is the one black-sheep monorepo at my own relatively-recently joined company, which is nowhere even close to approaching true large scale. Genuine question, though - what _are_ the advantages, as you see them (you didn't explicitly say as much, but I'm reading between the lines that you _can_ see some)? Every positive claim I've seen (primarily at https://monorepo.tools/ https://monorepo.tools/, but also elsewhere) feels either flimsy, or outright false: * "No overhead to create new projects - Use the existing CI setup" - I'm pretty confident that the amount of DX tooling work to make it super-smooth to create a new project is _dwarfed_ by the amount of work to make monorepos...work... * "Atomic commits across projects // One version of everything" - this is...actively bad? If I make a change to my library, I also have to change every consumer of it (or, worse, synchronize with them to make their changes at the same time before I can merge)? Whereas, in a polyrepo situation, I can publish the new version of my library, and decoupled consumers can update their consumption when they want to * "Developer mobility - Get a consistent way of building and testing applications" - it's perfectly easy to have a consistent experience across polyrepos, and or to have an inconsistent one in a monorepo. In fairness I will concede that a monorepo makes a consistent experience more _likely_, but that's a weak advantage at best. Monorepos _do_ make it significantly harder to _deliberately_ use different languages in different services, though, which is a perfectly cromulent thing to permit.
crotchfire 3y ago
Do not try and bend the spoon, that's impossible. Instead, only try to realize the truth...there is no spoon. Then you will see it is not the spoon that bends, it is only yourself -- what Pijul users say when they overhear git users arguing with each other about monorepos. https://pijul.org/ https://pijul.org/ Git insists that you impose an artificial ordering on commits which affect disjoint sets of files. And then people come up with heroic efforts like josh to try to bandage over that mistake. Or you could just not make the mistake in the first place...
- bognition 3y ago
  What alternative are you suggesting?
  - exDM69 3y ago
    Pijul is a version control system that is patch based (like Darcs) instead of snapshot based (like Git and Mercurial). Together with some mathematical models on the properties of patches it can deal with some merge situations that need to be resolved manually in Git. Pijul's website (and Darcs' before that) has some theory that may be helpful if you are interested in the details. The distinction between patch based and snapshot based is mostly invisible to the end user, except when it isn't.
    - gugagore 3y ago
      Couldn't git just recreate patches from snapshot? I mean, this is what happens when you show a commit or a diff. This is probably what you mean by how it's invisible to the user. When does it actually make a difference?
      - exDM69 3y ago
        Yes, git "recreates" the patches when you view a diff. But git can't decide that these two patches are independent of each other and thus their order does not matter so it'll give you a merge conflict where pijul doesn't. You can losslessly convert between the two models, but you can't efficiently apply patch theory (commutation etc) to a bunch of snapshots.
ixxie 3y ago
How does this improve monorepo workflow?
- rapnie 3y ago
  See the use cases: https://josh-project.github.io/josh/usecases.html https://josh-project.github.io/josh/usecases.html
rapnie 3y ago
Repo link: https://github.com/josh-project/josh https://github.com/josh-project/josh
jotaen 3y ago
> Josh combines the advantages of monorepos with those of multirepos by leveraging a blazingly-fast, incremental, and reversible implementation of git history filtering. It might be just me, but I have some serious trouble to understand what this actually is, and how I would use it specifically. From the homepage, I understand that this is some sort of utility around working with git histories in mono repos. But is Josh a client-side CLI tool? Is it a convention for structuring mono repos? A web service? A complete VCS? Or some combination of all that?
- p_l 3y ago
  It works by creating a proxy git remote which transparently rewrites history in ways that keep both sides in sync.
  - digdugdirk 3y ago
    Transparently in the "I don't have to worry about it" way? Or transparently in the "when something goes wrong I'll have no way to tell what happened" way?
    - parhamn 3y ago
      It's almost always one then the other, no?
    - p_l 3y ago
      Well, if something goes wrong you tend to have multiple repos that at worst will be recoverable with some amount of lost history. Benefit of Git being a DVCS. For most of the time you don't have to worry about it other than when you initially set things up.
alkonaut 3y ago
This is a great idea. Where do apply this though? The reason I use git at all is because of the support in other systems. It's not the best VCS, it's the best one that I can create a new repo for in Github or Azure Devops, or that I can link with issues in some issue mgmt system. If I'd need to self host a git repo, it would immediately lose most of its value. That's not to say it's useless longer term though. Other good extensions to git like Git LFS were also once in that situation, but wide adoption quickly meant that it's now a (mostly) integral part of git and every hosted git repository supports it. This would be very useful if it was adopted in the same way by the big repo hosters.
- aseipp 3y ago
  It doesn't manage a Git repository. It's a stateless proxy that sits in front of your existing Git repository. You just run it in front of your existing infrastructure. That's about it.
- k8svet 3y ago
  The site pretty clearly explains that the split repos are usable as normal git repos. And indeed I have a project I follow by cloning a josh managed repo. I didn't even know it was a josh repo until months after starting.
  - k8svet 3y ago
    Uh, what's going on with this thread? I'm pretty sure I didn't write this comment an hour ago (how it displays) and it didn't appear at the top of my comment history? Instead, appearing as "a day ago" in my history further down.
    - djsavvy 3y ago
      Weird. I don't see this comment, or the one you're referring to, on your profile either. Are you saying that you didn't write the comment, or that the timestamp is off?
    - macintux 3y ago
0xbeefeed 3y ago
[dead]

91 points by rapnie 3y ago

wbl 3y ago

I was unaware that people who used perforce and git would go decide that git needed to work like perforce.

To sum up, it looks like this is a server that provides smaller views of a large monorepo. A simple case is checking out a subdirectory of the repo and just getting the files and history for that subdirectory. They also advocate using these views to do builds. That way, your build can't depend on any files that are outside its view. It's a kind of sandboxing for builds. [1] > In particular when using C family languages, hidden dependencies on header files are easy to miss. For this reason limiting the visibility of files to the compiler by sandboxing is pretty much a requirement for reproducible builds. > With Josh, each deliverable gets its own virtual git repository with dependencies declared in the workspace.josh file. This means answering the above question becomes as simple as comparing commit ids. Furthermore due to the tree filtering each build is guaranteed to be perfectly sandboxed and only sees those parts of the monorepo that have actually been mapped. > This also means the deliverables to be re-built can be determined without cloning any repos like typically necessary with normal build tools. Bazel users might find this familiar. It does a similar thing by running build tools in a temporary directory that contains symlinks to your actual source code. But it's not something you actually need if you're not using a monorepo, since your package manager won't pull in other packages that you don't depend on. [1] https://josh-project.github.io/josh/usecases.html https://josh-project.github.io/josh/usecases.html

scubbo 3y ago

It's _amazing_ how much work monorepo fans put into "making a monorepo operate like a polyrepo", when they could just...

danpalmer 3y ago
I sort of get it. I don't think anyone coming from a multi-repo world really understands the full implications of a monorepo until they've worked in a large scale one (e.g. those at Google, Meta, or other places), so I think they try to keep certain kinds of practice that would ideally change in a monorepo world. And I also don't think Git scales to that (Microsoft just about made it work with a custom filesystem, Google and Meta both use custom source control, Meta's is based on Mercurial), and therefore it's often necessary to split the monorepo into many repos and there exist a lot of tools for treating a set of Git repos as one Monorepo.
- scubbo 3y ago
  > I don't think anyone coming from a multi-repo world really understands the full implications of a monorepo until they've worked in a large scale one That's entirely fair. My sole experience is the one black-sheep monorepo at my own relatively-recently joined company, which is nowhere even close to approaching true large scale. Genuine question, though - what _are_ the advantages, as you see them (you didn't explicitly say as much, but I'm reading between the lines that you _can_ see some)? Every positive claim I've seen (primarily at https://monorepo.tools/ https://monorepo.tools/, but also elsewhere) feels either flimsy, or outright false: * "No overhead to create new projects - Use the existing CI setup" - I'm pretty confident that the amount of DX tooling work to make it super-smooth to create a new project is _dwarfed_ by the amount of work to make monorepos...work... * "Atomic commits across projects // One version of everything" - this is...actively bad? If I make a change to my library, I also have to change every consumer of it (or, worse, synchronize with them to make their changes at the same time before I can merge)? Whereas, in a polyrepo situation, I can publish the new version of my library, and decoupled consumers can update their consumption when they want to * "Developer mobility - Get a consistent way of building and testing applications" - it's perfectly easy to have a consistent experience across polyrepos, and or to have an inconsistent one in a monorepo. In fairness I will concede that a monorepo makes a consistent experience more _likely_, but that's a weak advantage at best. Monorepos _do_ make it significantly harder to _deliberately_ use different languages in different services, though, which is a perfectly cromulent thing to permit.

crotchfire 3y ago

Do not try and bend the spoon, that's impossible. Instead, only try to realize the truth...there is no spoon. Then you will see it is not the spoon that bends, it is only yourself -- what Pijul users say when they overhear git users arguing with each other about monorepos. https://pijul.org/ https://pijul.org/ Git insists that you impose an artificial ordering on commits which affect disjoint sets of files. And then people come up with heroic efforts like josh to try to bandage over that mistake. Or you could just not make the mistake in the first place...

bognition 3y ago
What alternative are you suggesting?
- exDM69 3y ago
  Pijul is a version control system that is patch based (like Darcs) instead of snapshot based (like Git and Mercurial). Together with some mathematical models on the properties of patches it can deal with some merge situations that need to be resolved manually in Git. Pijul's website (and Darcs' before that) has some theory that may be helpful if you are interested in the details. The distinction between patch based and snapshot based is mostly invisible to the end user, except when it isn't.
  - gugagore 3y ago
    Couldn't git just recreate patches from snapshot? I mean, this is what happens when you show a commit or a diff. This is probably what you mean by how it's invisible to the user. When does it actually make a difference?
    - exDM69 3y ago
      Yes, git "recreates" the patches when you view a diff. But git can't decide that these two patches are independent of each other and thus their order does not matter so it'll give you a merge conflict where pijul doesn't. You can losslessly convert between the two models, but you can't efficiently apply patch theory (commutation etc) to a bunch of snapshots.

ixxie 3y ago

How does this improve monorepo workflow?

rapnie 3y ago
See the use cases: https://josh-project.github.io/josh/usecases.html https://josh-project.github.io/josh/usecases.html

rapnie 3y ago

Repo link: https://github.com/josh-project/josh https://github.com/josh-project/josh

jotaen 3y ago

> Josh combines the advantages of monorepos with those of multirepos by leveraging a blazingly-fast, incremental, and reversible implementation of git history filtering. It might be just me, but I have some serious trouble to understand what this actually is, and how I would use it specifically. From the homepage, I understand that this is some sort of utility around working with git histories in mono repos. But is Josh a client-side CLI tool? Is it a convention for structuring mono repos? A web service? A complete VCS? Or some combination of all that?

p_l 3y ago
It works by creating a proxy git remote which transparently rewrites history in ways that keep both sides in sync.
- digdugdirk 3y ago
  Transparently in the "I don't have to worry about it" way? Or transparently in the "when something goes wrong I'll have no way to tell what happened" way?
  - parhamn 3y ago
    It's almost always one then the other, no?
  - p_l 3y ago
    Well, if something goes wrong you tend to have multiple repos that at worst will be recoverable with some amount of lost history. Benefit of Git being a DVCS. For most of the time you don't have to worry about it other than when you initially set things up.

alkonaut 3y ago

This is a great idea. Where do apply this though? The reason I use git at all is because of the support in other systems. It's not the best VCS, it's the best one that I can create a new repo for in Github or Azure Devops, or that I can link with issues in some issue mgmt system. If I'd need to self host a git repo, it would immediately lose most of its value. That's not to say it's useless longer term though. Other good extensions to git like Git LFS were also once in that situation, but wide adoption quickly meant that it's now a (mostly) integral part of git and every hosted git repository supports it. This would be very useful if it was adopted in the same way by the big repo hosters.

aseipp 3y ago
It doesn't manage a Git repository. It's a stateless proxy that sits in front of your existing Git repository. You just run it in front of your existing infrastructure. That's about it.
k8svet 3y ago
The site pretty clearly explains that the split repos are usable as normal git repos. And indeed I have a project I follow by cloning a josh managed repo. I didn't even know it was a josh repo until months after starting.
- k8svet 3y ago
  Uh, what's going on with this thread? I'm pretty sure I didn't write this comment an hour ago (how it displays) and it didn't appear at the top of my comment history? Instead, appearing as "a day ago" in my history further down.
  - djsavvy 3y ago
    Weird. I don't see this comment, or the one you're referring to, on your profile either. Are you saying that you didn't write the comment, or that the timestamp is off?
  - macintux 3y ago

0xbeefeed 3y ago

[dead]