4 ms·

Whenever I hear smart and reasonable people argue well for both sides of an engineering issue, my experience is that it will turn out that we're arguing the wro

by ken 8y ago

Whenever I hear smart and reasonable people argue well for both sides of an engineering issue, my experience is that it will turn out that we're arguing the wrong question. The perspective is wrong. We can't get past thinking in terms of our old terminology. What we all really want is a VCS where repos can be combined and separated easily, or where one repo can gain the benefits of a monorepo without the drawbacks of one. Another crazy tech prediction from me: just as DVCS killed off pre-DVCS practically overnight, the thing that will quickly kill off DVCS is a new type of VCS where you can trivially combine/separate repos and sections of repos as needed. You can assign, at the repo level, sub-repos to include in this one, get an atomic commit hash for the state of the whole thing, and where my VCS client doesn't need to actually download every linked repo, but where tools are available to act like I have. (This will also enable it to replace the 10 different syntaxes I've had to learn for one project to reference another, via some Dependencies list and its generated Dependencies.lock list.) In a sense, we already have all of these features, in folders. You can combine and separate them, you can make a local folder mimic a folder on a remote system, and access its content without needing to download it all ahead of time. They just don't have any VCS features baked in. We've got {filesystems, network filesystems, and VCS}, and each of the three has some features the others would like! I don't have much money right now but I'd pay $1000 for a good solution to this. I'd use it for my home directory, my backups, my media server, etc.

jiveturkey 8y ago
With remote depots, perforce has the functionality you want, or nearly so. There used to be a free tier, which AFAICT is gone now. The google implementation internally supports virtual checkouts like you suggest.
- ken 8y ago
  I mean free and open source, of course. "Only exists as an internal Google project" is isomorphic, from my perspective, to "Does not exist".
  - dllthomas 8y ago
    > "Only exists as an internal Google project" is isomorphic, from my perspective, to "Does not exist". I mean, it shows that it's possible...
WorldMaker 8y ago
I don't think it is quite a revolution from/killer for the DVCS as you think. A lot of what you ask for can be done with DVCS tools if you go low level enough. The git graph supports more complex shapes in the raw than the git UI tools ('porcelains') tend to make it easy to work with. Especially when you throw in new tools like GitVFS for virtualizing some/all of the git object database. To some extent what you ask for is simply a UX problem in the DVCS world of how to make such power available in an easy-enough to use way, without providing too many additional new footguns. > Whenever I hear smart and reasonable people argue well for both sides of an engineering issue, my experience is that it will turn out that we're arguing the wrong question. Right, though my feeling is the root underlying discussion is actually sociopolitical rather than technical. The preference between monorepos and polyrepos seems more to do with the organizational structures of organizations and people than about the actual technical merits of either approach. I think the issue is that developers just feel safer trying to tie things to technical merits than engage with the problem at a sociopolitical level.
- candiodari 8y ago
  > tie things to technical merits than engage with the problem at a sociopolitical level. This hits the nail on the head. The real advantage of monorepos is that they are reproducible. Many-small-repos aren't. A monorepo, ideally one like bazel[1] pushes will produce the same bitstream every time you put "make". Projects terminated, going in a different direction, servers moved, down, forked, ... it all just doesn't matter. Your stuff works. Network or no network. The problem is that these repos allow programmers to be very undisciplined in a way that sabotages projects. In a monorepo you are updating libraries. In every individual project. That's a big plus, when it comes to "my software works", but of course it is work. Your software doesn't suddently stop working, but that's because it doesn't change, which means it won't change unless you make it change. Which means, new database driver ? You're upgrading it. No apt-get, no sysadmin deploying a shared library or dll install doing it for you. There is always a strong drive by higher up programmers and managers to fix "undisciplined" programming, and ... I've never seen this work. Or at least, I've never seen this do more than move the problem. The thing I don't understand about some languages, like Golang, is the (ridiculous) insistence on hermetic binaries (not using anything on the system, not even libc), and yet their very strong insistence on non-hermetic source (even to the point of sabotaging the tools for people who wish to do it, and sabotaging to some extent the community to achieve this). Either do one, or the other. I feel like C/C++'s support of both approaches is a superior option. You want shared (nobody does, but hey that's my opinion) ? You can do that. You want everything hermetic ? Not a problem. In my opinion sharing things between projects is a mistake. [1] https://bazel.build/ https://bazel.build/
Bjartr 8y ago
> You can assign, at the repo level, sub-repos to include in this one, get an atomic commit hash for the state of the whole thing, and where my VCS client doesn't need to actually download every linked repo, but where tools are available to act like I have. Mercurial subrepos work a lot like you describe.
rakoo 8y ago
> We've got {filesystems, network filesystems, and VCS}, and each of the three has some features the others would like! We actually have a fourth one that has nice features but missing what you exactly need: packages (and associated package managers). A package has (on top of build instructions) a list of files, and can depend on another package for working properly. At the top, you could have your own "machine.pkg" that defines all that is needed and is updated as some files are updated. A package can go from one machine to another and only the missing dependencies will be fetched. A package manager has the pro that any part can be defined as a package and abstracted away, and the con that packages aren't expected to change much so it'd become heavy to recursively update all packages, but that could be a way to hack something.

by ken 8y ago

jiveturkey 8y ago

With remote depots, perforce has the functionality you want, or nearly so. There used to be a free tier, which AFAICT is gone now. The google implementation internally supports virtual checkouts like you suggest.

ken 8y ago
I mean free and open source, of course. "Only exists as an internal Google project" is isomorphic, from my perspective, to "Does not exist".
- dllthomas 8y ago
  > "Only exists as an internal Google project" is isomorphic, from my perspective, to "Does not exist". I mean, it shows that it's possible...

WorldMaker 8y ago

I don't think it is quite a revolution from/killer for the DVCS as you think. A lot of what you ask for can be done with DVCS tools if you go low level enough. The git graph supports more complex shapes in the raw than the git UI tools ('porcelains') tend to make it easy to work with. Especially when you throw in new tools like GitVFS for virtualizing some/all of the git object database. To some extent what you ask for is simply a UX problem in the DVCS world of how to make such power available in an easy-enough to use way, without providing too many additional new footguns. > Whenever I hear smart and reasonable people argue well for both sides of an engineering issue, my experience is that it will turn out that we're arguing the wrong question. Right, though my feeling is the root underlying discussion is actually sociopolitical rather than technical. The preference between monorepos and polyrepos seems more to do with the organizational structures of organizations and people than about the actual technical merits of either approach. I think the issue is that developers just feel safer trying to tie things to technical merits than engage with the problem at a sociopolitical level.

candiodari 8y ago
> tie things to technical merits than engage with the problem at a sociopolitical level. This hits the nail on the head. The real advantage of monorepos is that they are reproducible. Many-small-repos aren't. A monorepo, ideally one like bazel[1] pushes will produce the same bitstream every time you put "make". Projects terminated, going in a different direction, servers moved, down, forked, ... it all just doesn't matter. Your stuff works. Network or no network. The problem is that these repos allow programmers to be very undisciplined in a way that sabotages projects. In a monorepo you are updating libraries. In every individual project. That's a big plus, when it comes to "my software works", but of course it is work. Your software doesn't suddently stop working, but that's because it doesn't change, which means it won't change unless you make it change. Which means, new database driver ? You're upgrading it. No apt-get, no sysadmin deploying a shared library or dll install doing it for you. There is always a strong drive by higher up programmers and managers to fix "undisciplined" programming, and ... I've never seen this work. Or at least, I've never seen this do more than move the problem. The thing I don't understand about some languages, like Golang, is the (ridiculous) insistence on hermetic binaries (not using anything on the system, not even libc), and yet their very strong insistence on non-hermetic source (even to the point of sabotaging the tools for people who wish to do it, and sabotaging to some extent the community to achieve this). Either do one, or the other. I feel like C/C++'s support of both approaches is a superior option. You want shared (nobody does, but hey that's my opinion) ? You can do that. You want everything hermetic ? Not a problem. In my opinion sharing things between projects is a mistake. [1] https://bazel.build/ https://bazel.build/

Bjartr 8y ago

> You can assign, at the repo level, sub-repos to include in this one, get an atomic commit hash for the state of the whole thing, and where my VCS client doesn't need to actually download every linked repo, but where tools are available to act like I have. Mercurial subrepos work a lot like you describe.

rakoo 8y ago

> We've got {filesystems, network filesystems, and VCS}, and each of the three has some features the others would like! We actually have a fourth one that has nice features but missing what you exactly need: packages (and associated package managers). A package has (on top of build instructions) a list of files, and can depend on another package for working properly. At the top, you could have your own "machine.pkg" that defines all that is needed and is updated as some files are updated. A package can go from one machine to another and only the missing dependencies will be fetched. A package manager has the pro that any part can be defined as a package and abstracted away, and the con that packages aren't expected to change much so it'd become heavy to recursively update all packages, but that could be a way to hack something.