8 ms·

I've worked both at Google (only as an intern, though) and at other very very big companies with gargantuan code bases. At that scale, with software that is con

by anyfoo 8y ago

I've worked both at Google (only as an intern, though) and at other very very big companies with gargantuan code bases. At that scale, with software that is constantly in flux, pretty much the last thing you want is having to keep compatibility between several versions of a component. It's bad enough if you have to do it for external reasons, but if the only reason is so that "others in the company have a choice" then... no, just no. You might think this ought to be trivial by having clear API contracts, but that's a) not how things work in practice if all code is effectively owned by the same, overarching entity and, more importantly, b) now you have an enormous effort to transition between incompatible API revisions instead of just being able to do lockstep changes, for no real gain. Even if you manage to pull that off (again, for what benefit?), it will bite you that 1.324.2564 behaves subtly different from 1.324.5234 even though the intent was just to add a new option and they otherwise ought to have no extensional changes in behavior.

whack 8y ago
It took me a while to figure out that you're disagreeing with me, because your last paragraph is a perfect example of why monorepos are so dangerous. Imagine a tooling team on a different continent that makes some changes this afternoon. Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, all your services end up broken as a result. In a versioned world, you can still freeze your dependency at 1.324.5234, and migrate only when you want to, and when you're feeling confident about it. In a monorepo world, you don't have a choice. You've been forcefully migrated as soon as the tooling team decides to make the change on their end. They had the best of intentions, but that doesn't always translate to a good outcome. FWIW, I'm currently working at a large famous company that uses a monorepo. Color me not-impressed. I do think that having a single repository for an entire team/project is a good idea. Hundreds of different projects and teams who've never seen one another? Not so much.
- shub 8y ago
  > Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, all your services end up broken as a result. Someone makes a commit to library code and production magically breaks? How does that happen?
  - YokoZar 8y ago
    > Someone makes a commit to library code and production magically breaks? How does that happen? With insufficient tests and broken release processes.
    - shub 8y ago
      Yeah, if you’re pushing to prod automatically without a really good test suite then I would say the problem isn’t with the person who wrote a bug.
- Eridrus 8y ago
  > The next morning, all your services end up broken as a result. I mean, this is the argument for having good integration tests. At some point someone has to figure out if the new code will break a system; if you don't have good integration tests you're basically left eyeballing the changes, and sure eyeballing changes can work fine in small teams, but at some point you need good tests. I did some work on a ranking system last year, and by definition there's no way to roll that out incrementally, because, well, it is the central component deciding what thing to do/show, and you have to change the world at once, there is literally no other option. So you need good ways of evaluating these wide reaching changes. "If you liked it then you shoulda put a test on it" :)
  - shados 8y ago
    > I mean, this is the argument for having good integration tests. Maybe companies like Google have very, very strong code hygiene, but at most places I've worked, sooner or later there's a project that had a tight deadline and someone thought it smart to cut corners on the tests. Or the test s are there but they're bad. Or incomplete. Or worse, some system was just too hard to test and not updated frequently enough, so it requires manual testing. Having "eventual consistency" for this is quite nice. Push a breaking change, update what you can, run tests, speak with owners, get deployed what you can. Keep tab of what you couldn't. Then do what you have to do to get the stuff tested (even if it means manual) and gradually become consistent as these things get pushed to prod...and hopefully next time its easier.
- anyfoo 8y ago
  > In a versioned world, you can still freeze your dependency at 1.324.5234, and migrate only when you want to, and when you're feeling confident about it. The correct course of action is to either reverse/fix the code change to the library you depend on, or if your code is clearly using the library wrong and can be easily fixed, to do that. Not to let the whole ecosystem slowly spiral out of control. Either way, the point is that it will force the issue to be resolved, quickly, and the code base to move forward. The tools/libraries you depend on are themselves dependent on other libraries and tools. They may have done changes that are necessary to continue working, which you are not picking up if you stay behind. They will do IPC and RPC and always rely on their infrastructure being current. >In a monorepo world, you don't have a choice. You've been forcefully migrated Yes, and that's good, because: > and migrate only when you want to, and when you're feeling confident about it. ... does not help in moving the code forward. If your change will break others, you need to coordinate with those others so that the transition happens gracefully, not let them live on what amounts to unsupported (and slowly more incompatible) code.
  - mbrock 8y ago
    “Either way, the point is that it will force the issue to be resolved, quickly, and the code base to move forward.” This is also known as continuous integration.
  - whack 8y ago
    >> In a monorepo world, you don't have a choice. You've been forcefully migrated > Yes, and that's good In your projects, have you configured your build system to always auto-pull the latest version of every single dependency you have? If not, you're not practicing what you've claimed above. FWIW, java-maven used to allow specifying LATEST/RELEASE versions, so that the latest version will always be auto-pulled on every build. They later removed that option entirely, because they realized how dangerous that is. https://stackoverflow.com/questions/30571/how-do-i-tell-maven-to-use-the-latest-version-of-a-dependency https://stackoverflow.com/questions/30571/how-do-i-tell-mave...
perlgeek 8y ago
I thought that Google deploys new versions gradually (first to 1% of users, and if that doesn't show errors, to more and more). Which implies that there are at least two version of an application or service running. How does that work when you don't keep APIs stable, at least at the service boundaries?

by anyfoo 8y ago

whack 8y ago

It took me a while to figure out that you're disagreeing with me, because your last paragraph is a perfect example of why monorepos are so dangerous. Imagine a tooling team on a different continent that makes some changes this afternoon. Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, all your services end up broken as a result. In a versioned world, you can still freeze your dependency at 1.324.5234, and migrate only when you want to, and when you're feeling confident about it. In a monorepo world, you don't have a choice. You've been forcefully migrated as soon as the tooling team decides to make the change on their end. They had the best of intentions, but that doesn't always translate to a good outcome. FWIW, I'm currently working at a large famous company that uses a monorepo. Color me not-impressed. I do think that having a single repository for an entire team/project is a good idea. Hundreds of different projects and teams who've never seen one another? Not so much.

shub 8y ago
> Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, all your services end up broken as a result. Someone makes a commit to library code and production magically breaks? How does that happen?
- YokoZar 8y ago
  > Someone makes a commit to library code and production magically breaks? How does that happen? With insufficient tests and broken release processes.
  - shub 8y ago
    Yeah, if you’re pushing to prod automatically without a really good test suite then I would say the problem isn’t with the person who wrote a bug.
Eridrus 8y ago
> The next morning, all your services end up broken as a result. I mean, this is the argument for having good integration tests. At some point someone has to figure out if the new code will break a system; if you don't have good integration tests you're basically left eyeballing the changes, and sure eyeballing changes can work fine in small teams, but at some point you need good tests. I did some work on a ranking system last year, and by definition there's no way to roll that out incrementally, because, well, it is the central component deciding what thing to do/show, and you have to change the world at once, there is literally no other option. So you need good ways of evaluating these wide reaching changes. "If you liked it then you shoulda put a test on it" :)
- shados 8y ago
  > I mean, this is the argument for having good integration tests. Maybe companies like Google have very, very strong code hygiene, but at most places I've worked, sooner or later there's a project that had a tight deadline and someone thought it smart to cut corners on the tests. Or the test s are there but they're bad. Or incomplete. Or worse, some system was just too hard to test and not updated frequently enough, so it requires manual testing. Having "eventual consistency" for this is quite nice. Push a breaking change, update what you can, run tests, speak with owners, get deployed what you can. Keep tab of what you couldn't. Then do what you have to do to get the stuff tested (even if it means manual) and gradually become consistent as these things get pushed to prod...and hopefully next time its easier.
anyfoo 8y ago
> In a versioned world, you can still freeze your dependency at 1.324.5234, and migrate only when you want to, and when you're feeling confident about it. The correct course of action is to either reverse/fix the code change to the library you depend on, or if your code is clearly using the library wrong and can be easily fixed, to do that. Not to let the whole ecosystem slowly spiral out of control. Either way, the point is that it will force the issue to be resolved, quickly, and the code base to move forward. The tools/libraries you depend on are themselves dependent on other libraries and tools. They may have done changes that are necessary to continue working, which you are not picking up if you stay behind. They will do IPC and RPC and always rely on their infrastructure being current. >In a monorepo world, you don't have a choice. You've been forcefully migrated Yes, and that's good, because: > and migrate only when you want to, and when you're feeling confident about it. ... does not help in moving the code forward. If your change will break others, you need to coordinate with those others so that the transition happens gracefully, not let them live on what amounts to unsupported (and slowly more incompatible) code.
- mbrock 8y ago
  “Either way, the point is that it will force the issue to be resolved, quickly, and the code base to move forward.” This is also known as continuous integration.
- whack 8y ago
  >> In a monorepo world, you don't have a choice. You've been forcefully migrated > Yes, and that's good In your projects, have you configured your build system to always auto-pull the latest version of every single dependency you have? If not, you're not practicing what you've claimed above. FWIW, java-maven used to allow specifying LATEST/RELEASE versions, so that the latest version will always be auto-pulled on every build. They later removed that option entirely, because they realized how dangerous that is. https://stackoverflow.com/questions/30571/how-do-i-tell-maven-to-use-the-latest-version-of-a-dependency https://stackoverflow.com/questions/30571/how-do-i-tell-mave...

perlgeek 8y ago

I thought that Google deploys new versions gradually (first to 1% of users, and if that doesn't show errors, to more and more). Which implies that there are at least two version of an application or service running. How does that work when you don't keep APIs stable, at least at the service boundaries?