6 ms·

The repository structure seems a bit bad in that it takes months for changes to propagate, but how else should a project as big as windows be handled at the sou

The repository structure seems a bit bad in that it takes months for changes to propagate, but how else should a project as big as windows be handled at the source-level?

eitally 16y ago
Well, creating a common vision would have been a good start, as would having had a single owner. There is a point where it becomes more valuable for managers to delegate and step away than it is for them to try and keep abreast of the minutiae. In order for delegation to work there must be comprehensive trust. In my experience, large organizations usually fail miserably at this combination of trust, vision, and delegation [at some point of scaling]. The better you are the better you can scale, but even revered learning organizations like 3M, Bell Labs, and Xerox PARC relied [mostly] on small, self-organizing organic project teams with very little management oversight... but oodles of charisma & responsibility.
btilly 16y ago
For an alternative watch http://www.youtube.com/watch?v=sMql3Di4Kgc http://www.youtube.com/watch?v=sMql3Di4Kgc to see how Google does it. Basically most of the company develops on one branch, but no changelist can be checked in without going through peer review. (The talk is about the tool that Google uses for peer review.) It seems to work quite well. Now you can ask how the project size compares. Well http://en.wikipedia.org/wiki/Source_lines_of_code#Example http://en.wikipedia.org/wiki/Source_lines_of_code#Example claims that Windows Server 2003 had 50 million lines of code. http://code.google.com/opensource/ http://code.google.com/opensource/ claims that Google has released over 15 million lines of code across various projects. I guarantee that Google has not released most of its code. Based on those data points I wouldn't be surprised if the sizes were comparable, at least to within an order of magnitude. (However Google's source code is probably significantly better. Google has not been around as long, and does not have the same kinds of accumulated legacy issues that Microsoft does.) Random disclaimer, I work for Google and I like how Google does it.
- emehrkay 16y ago
  Thanks, I'll check that video out.
- macrael 16y ago
  Is it really fair to compare the amount of code in windows server to the total amount of code that google maintains? Google has many different software projects, what is the biggest? Search? Android? Gmail? Compare one project to a server operating system and my guess is that the OS is going to be an order of magnitude larger. An OS is an enormously complicated piece of software with lots of co-dependent parts. That's what led the to this pretty horribly complicated source management system.
  - wtallis 16y ago
    Google's stuff may be a lot more modular than Windows, but that's not to say that there aren't inter-dependencies. Google's web services all share just a few back-end technologies (MapReduce, GFS, BigTable) that are comparable to an OS kernel, with things like GMail and Docs hanging off like subsystems. But even things like Gmail, Docs, Calender, Search, etc. aren't even close to independent. The GMail people need to collaborate with the Docs people to handle attachments and the Calendar people to handle to-do lists and event invitations and the GTalk and Voice people to implement the voice and video chat and phone calling. The Picasa people need to collaborate with the the Google Earth and Maps people to handle geotagging, and the GWT (probably) people to deal with the web interface. The Android people need to collaborate with everybody who works on the front-ends of all the other products in order to develop mobile versions or integrate them more deeply in to the OS. Even if the "collaboration" in many of these instances is mostly limited to using an API published by the other group, there's still the need to make sure that those APIs are sufficient to accomplish the task at hand, and obviously Google's UI consistency between products is more than an accident. I think that to the extent that Microsoft's code is more tightly coupled than Google's or that of a desktop Linux distribution, it's mostly their own fault.
  - joshu 16y ago
    android is kept in git, not the main archive.
adambyrtek 16y ago
Thinking about that, looks like this is a perfect example to prove why continuous integration is such a great idea.
city41 16y ago
VS2010 was the largest source repository I've ever worked on. It is structured very similar to Windows, all large projects at MS are as far as I know. It ended up being my job was almost entirely dedicated to just maintaining our branches: pushing our updates up to Main, and pulling in fresh changes about once a week. DevDiv has a term for this: BFD, "Branch Facilitator Developer", which for most teams is a full time job of nothing but merging code. For me it was about 70% of my job. Each branch took about 120 gigs of space on disk, and I regularly worked with about 5 branches. Merging our changes into main was a huge ordeal. We had to merge, do a full stack build (which took about 24 hours), and run every single team's tests, which took another 24 to 48 hours. From there it was a bookkeeping task of checking in on all the tests that failed and resolving them, doing performance testing, etc etc. All in all merging to main took roughly a week to pull off. The reverse was always interesting: pulling in new bits from Main to our branch was like playing Russian roulette, what was going to break this time? To add a little spice to all of this, it all took place on alpha builds of Team Foundation Server that was being developed at the same time as everything else. Fun times :)

The repository structure seems a bit bad in that it takes months for changes to propagate, but how else should a project as big as windows be handled at the source-level?

eitally 16y ago

Well, creating a common vision would have been a good start, as would having had a single owner. There is a point where it becomes more valuable for managers to delegate and step away than it is for them to try and keep abreast of the minutiae. In order for delegation to work there must be comprehensive trust. In my experience, large organizations usually fail miserably at this combination of trust, vision, and delegation [at some point of scaling]. The better you are the better you can scale, but even revered learning organizations like 3M, Bell Labs, and Xerox PARC relied [mostly] on small, self-organizing organic project teams with very little management oversight... but oodles of charisma & responsibility.

btilly 16y ago

For an alternative watch http://www.youtube.com/watch?v=sMql3Di4Kgc http://www.youtube.com/watch?v=sMql3Di4Kgc to see how Google does it. Basically most of the company develops on one branch, but no changelist can be checked in without going through peer review. (The talk is about the tool that Google uses for peer review.) It seems to work quite well. Now you can ask how the project size compares. Well http://en.wikipedia.org/wiki/Source_lines_of_code#Example http://en.wikipedia.org/wiki/Source_lines_of_code#Example claims that Windows Server 2003 had 50 million lines of code. http://code.google.com/opensource/ http://code.google.com/opensource/ claims that Google has released over 15 million lines of code across various projects. I guarantee that Google has not released most of its code. Based on those data points I wouldn't be surprised if the sizes were comparable, at least to within an order of magnitude. (However Google's source code is probably significantly better. Google has not been around as long, and does not have the same kinds of accumulated legacy issues that Microsoft does.) Random disclaimer, I work for Google and I like how Google does it.

emehrkay 16y ago
Thanks, I'll check that video out.
macrael 16y ago
Is it really fair to compare the amount of code in windows server to the total amount of code that google maintains? Google has many different software projects, what is the biggest? Search? Android? Gmail? Compare one project to a server operating system and my guess is that the OS is going to be an order of magnitude larger. An OS is an enormously complicated piece of software with lots of co-dependent parts. That's what led the to this pretty horribly complicated source management system.
- wtallis 16y ago
  Google's stuff may be a lot more modular than Windows, but that's not to say that there aren't inter-dependencies. Google's web services all share just a few back-end technologies (MapReduce, GFS, BigTable) that are comparable to an OS kernel, with things like GMail and Docs hanging off like subsystems. But even things like Gmail, Docs, Calender, Search, etc. aren't even close to independent. The GMail people need to collaborate with the Docs people to handle attachments and the Calendar people to handle to-do lists and event invitations and the GTalk and Voice people to implement the voice and video chat and phone calling. The Picasa people need to collaborate with the the Google Earth and Maps people to handle geotagging, and the GWT (probably) people to deal with the web interface. The Android people need to collaborate with everybody who works on the front-ends of all the other products in order to develop mobile versions or integrate them more deeply in to the OS. Even if the "collaboration" in many of these instances is mostly limited to using an API published by the other group, there's still the need to make sure that those APIs are sufficient to accomplish the task at hand, and obviously Google's UI consistency between products is more than an accident. I think that to the extent that Microsoft's code is more tightly coupled than Google's or that of a desktop Linux distribution, it's mostly their own fault.
- joshu 16y ago
  android is kept in git, not the main archive.

adambyrtek 16y ago

Thinking about that, looks like this is a perfect example to prove why continuous integration is such a great idea.

city41 16y ago

VS2010 was the largest source repository I've ever worked on. It is structured very similar to Windows, all large projects at MS are as far as I know. It ended up being my job was almost entirely dedicated to just maintaining our branches: pushing our updates up to Main, and pulling in fresh changes about once a week. DevDiv has a term for this: BFD, "Branch Facilitator Developer", which for most teams is a full time job of nothing but merging code. For me it was about 70% of my job. Each branch took about 120 gigs of space on disk, and I regularly worked with about 5 branches. Merging our changes into main was a huge ordeal. We had to merge, do a full stack build (which took about 24 hours), and run every single team's tests, which took another 24 to 48 hours. From there it was a bookkeeping task of checking in on all the tests that failed and resolving them, doing performance testing, etc etc. All in all merging to main took roughly a week to pull off. The reverse was always interesting: pulling in new bits from Main to our branch was like playing Russian roulette, what was going to break this time? To add a little spice to all of this, it all took place on alpha builds of Team Foundation Server that was being developed at the same time as everything else. Fun times :)