6 ms·

Torvalds once said [1] that the only scalable software development methodology is open source, and I tend to agree for two reasons: 1. The project structure, t

by abiro 6y ago

Torvalds once said [1] that the only scalable software development methodology is open source, and I tend to agree for two reasons: 1. The project structure, tooling and documentation that lets new contributors jump in quickly makes software development easy to scale. In Coasian terms, the transaction costs around development are minimized. 2. It enforces hard interfaces and it clearly separates development from operations. Lack of discipline around these issues is a source of much accidental complexity. [1] I can’t seem to find the quote, I read it in an interview a few years ago and stuck with me since.

kasperni 6y ago
> the only scalable software development methodology is open source Some companies such as Microsoft, Google, Amazon would disagree.
- abiro 6y ago
  The Linux kernel had 20k contributors since its inception, you’d be hard pressed to find those numbers for a single project at any company.
  - kasperni 6y ago
    Yes, the linux kernel is a big codebase but so are a lot of others [1]. And you will find lots of private companies not mentioned that have long-standing projects with >10 mil LOC. If you took the 100 biggest code bases in the world. I would be surprised if more than 10% of them were open source. [1] https://www.informationisbeautiful.net/visualizations/million-lines-of-code/ https://www.informationisbeautiful.net/visualizations/millio...
    - oblio 6y ago
      I know a medium sized French-American multinational. You've probably not even heard of it. A decade ago they had multiple products that had several million lines of code. Their entire platforms probably had 100 million. I can't even imagine how many contributors they had, the oldest project was started in the 80's and it probably had 1000 contributors over time, at least. And again, that's for a middle size company you haven't even heard of. FAANG, Microsoft, Oracle, IBM for sure will have stuff dwarfing that.
      - abhijat 6y ago
        Someone from oracle posted here a while ago, where they described Oracle 12.2 has 25 million lines of C https://news.ycombinator.com/item?id=18442941 https://news.ycombinator.com/item?id=18442941
        marcosdumay 6y ago
        And yet it's a less appealing product than Postgres. An explosion on the number of lines of code is one of the way development teams fail.
        _dps 6y ago
        Is it actually less appealing? My understanding is that DBAs consider Oracle very good, just very (very) expensive. This also lines up with my experience tuning queries against Oracle vs Postgres backends. The folk wisdom seems, to me, to be Postgres is 99% as good in the common cases and 90-95% as good in the difficult cases.
        marcosdumay 6y ago
        Oracle DBAs consider Oracle clearly superior. SQL Server DBAs that know all the ways to optimize SQL Servers consider SQL Server clearly superior. I don't know of any PostgreSQL DBA, just generalists that work with Postgres, so I don't know about them. The truth is that Oracle is ridden with coherence bugs, and have a much worse performance picture out of the box than Postgres. But while improving the Postgres performance requires deep digging into the DBMS itself, Oracle has a lot you can optimize just on the SQL interface. But there are plenty of cases where Oracle just can not become as fast as out of the box Postgres, and many where it could in theory, but a bug prevents you from doing things right. Overall, three is no definitive ordering on the speed of the "good 3" DBMS. It always depends on what you are doing.
      - qznc 6y ago
        SAP told me their framework is a billion LoC of Java and 30k tables in the database. That was years ago so it has probably grown further. It is only the framework, so no useful application yet.
  - kgeist 6y ago
    How many of those 20k contributors worked on drivers, and how many - on the actual core (~150kloc)? Every driver is like a separate subproject, having 20k people work on hundreds/thousands of drivers (unrelated to each other) wouldn't sound as impressive.
    - thrower123 6y ago
      That's the scary thing. How many real, core contributors does even something like the Linux kernel have? People who have actually written more than a couple patches that landed and stayed in the kernel. I'd be astounded if it's much more than a hundred. Most open-source projects have one to maybe five real contributors, outside the drive-by pull requests to fix some bug.
  - misterS 6y ago
    I guess (and I have no way to verify this, it's really just a guess) that the number of developers that worked on the Windows code base in its 35 year history is in the same order, if not higher.
  - rcxdude 6y ago
    Consider the bit at the end of this blog post: https://devblogs.microsoft.com/oldnewthing/20180326-00/?p=98335 https://devblogs.microsoft.com/oldnewthing/20180326-00/?p=98... Bonus chatter: In a discussion of Windows source control, one person argued that git works great on large projects and gave an example of a large git repo: “For example, the linux kernel repository at roughly eight years old is 800–900MB in size, has about 45,000 files, and is considered to have heavy churn, with 400,000 commits.” I found that adorable. You have 45,000 files. Yeah, call me when your repo starts to get big. The Windows repo has over three million files. Four hundred thousand commits over eight years averages to around [130] commits per day. This is heavy churn? That’s so cute. You know what we call a day with [130] commits? “Catastrophic network outage.”
    - jka 6y ago
      To get closer to an apples-to-apples comparison, it'd be necessary to know whether the commit counts in each case include all development branches for all development groups. By design, git-based development can be highly distributed. Also, even if we normalized both cases for 'code files only' and/or 'kernel code only', there could still be architectural, code style, and development process differences that lead to different metrics for each project.
    - deleted 6y ago
      [deleted]
    - AlexCoventry 5y ago
      He's really not selling Microsoft as a fun place to work, is he?
dragonwriter 6y ago
> the only scalable software development methodology is open source “open source” isn’t a development methodology, or even a distinct set of methodologies. > The project structure, tooling and documentation that lets new contributors jump in quickly makes software development easy to scale. Plenty of open source projects don’t have that, nor is there anything restricting those things to open source projects. It is true that some open source projects, because they see the value in new developers jumping in quickly, prioritize having structure, documentation, and tooling that supports that. It’s also true that some proprietary software projects do, too, because the project owner sees value in that.
- abiro 6y ago
  There are books written about open source development. Don’t confuse somebody dumping source files on Github with open source development.
  - oblio 6y ago
    There are a gazillion ways to develop open source. It's more like an ethos than a methodology.
    - marcosdumay 6y ago
      > There are a gazillion ways to develop open source. Yet every large project behaves in a similar way, with only a few larger variations.
      - oblio 6y ago
        That's why one of the seminal books about Open Source is the Cathedral and the Bazaar? Because "there is only one way to do things"? :-)
        marcosdumay 6y ago
        Hum... The Bazaar on that book is a very specific way to do things. So specific that I don't think anybody really follows it.
qznc 6y ago
Since Open Source software drops liability by definition, a huge complexity factor disappears.
rakoo 6y ago
To go further I think it's not exactly open source but remote-first that made software development scalable. If you grossly simplify it to its innermost core, making development scalable means that if you have 10 times more people you can have 10 times more features/bugfixes/speed improvements/... The only way to do that is to make sure that an additional developer doesn't depend on other developers to work, and that can only happen if everything is properly documented, the build instructions are up-to-date, the processes are clear, basically anyone can start from scratch and get up to speed without the help of anyone else. That kind of organization traditionally doesn't happen in on-site companies, where newcomers are followed by senior people, they have to follow some introduction course to familiarize themselves with the processes, they have to ask many questions every day, they need to be synchronized with other persons which brings some inefficiency beacuse everyone works at their own pace, etc... This all disappears when everything is properly documentend and every contributor can work in the middle of the night if they wish. I think the Gitlab Handbook goes over this quite well and describes a framework to implement that kind of organization but the rules are retrospectively obvious for people already used to open-source (https://about.gitlab.com/company/culture/all-remote/guide/ https://about.gitlab.com/company/culture/all-remote/guide/): - write down everything - discussion should happen asynchronously. Any synchronous discussion (by text or call) should be only very small points. Whatever the type, write down the conclusions of that discussions - Everything is public (to the organization), including decisions taken, issues, processes
- marcosdumay 6y ago
  Yeah, remote-first is important, but it's not only it. Another very relevant factor is that people can just clone stuff, create new projects, and everything moves independently. So the open source development model has a pile of solutions for dependency management that team based development doesn't adopt. But the one thing I don't get is why team based development doesn't adopt those solution. They are not expensive. Yet, even when I was able to dictate into everybody's requirements, I wasn't able to force teams into adopting them. Instead they insisted on synchronizing themselves by much more expensive and less reliable means. My guess is that most developers never dug any deep into open source and have no idea how it's done.

by abiro 6y ago

kasperni 6y ago

> the only scalable software development methodology is open source Some companies such as Microsoft, Google, Amazon would disagree.

abiro 6y ago
The Linux kernel had 20k contributors since its inception, you’d be hard pressed to find those numbers for a single project at any company.
- kasperni 6y ago
  Yes, the linux kernel is a big codebase but so are a lot of others [1]. And you will find lots of private companies not mentioned that have long-standing projects with >10 mil LOC. If you took the 100 biggest code bases in the world. I would be surprised if more than 10% of them were open source. [1] https://www.informationisbeautiful.net/visualizations/million-lines-of-code/ https://www.informationisbeautiful.net/visualizations/millio...
  - oblio 6y ago
    I know a medium sized French-American multinational. You've probably not even heard of it. A decade ago they had multiple products that had several million lines of code. Their entire platforms probably had 100 million. I can't even imagine how many contributors they had, the oldest project was started in the 80's and it probably had 1000 contributors over time, at least. And again, that's for a middle size company you haven't even heard of. FAANG, Microsoft, Oracle, IBM for sure will have stuff dwarfing that.
    - abhijat 6y ago
      Someone from oracle posted here a while ago, where they described Oracle 12.2 has 25 million lines of C https://news.ycombinator.com/item?id=18442941 https://news.ycombinator.com/item?id=18442941
      - marcosdumay 6y ago
        And yet it's a less appealing product than Postgres. An explosion on the number of lines of code is one of the way development teams fail.
        _dps 6y ago
        Is it actually less appealing? My understanding is that DBAs consider Oracle very good, just very (very) expensive. This also lines up with my experience tuning queries against Oracle vs Postgres backends. The folk wisdom seems, to me, to be Postgres is 99% as good in the common cases and 90-95% as good in the difficult cases.
        marcosdumay 6y ago
        Oracle DBAs consider Oracle clearly superior. SQL Server DBAs that know all the ways to optimize SQL Servers consider SQL Server clearly superior. I don't know of any PostgreSQL DBA, just generalists that work with Postgres, so I don't know about them. The truth is that Oracle is ridden with coherence bugs, and have a much worse performance picture out of the box than Postgres. But while improving the Postgres performance requires deep digging into the DBMS itself, Oracle has a lot you can optimize just on the SQL interface. But there are plenty of cases where Oracle just can not become as fast as out of the box Postgres, and many where it could in theory, but a bug prevents you from doing things right. Overall, three is no definitive ordering on the speed of the "good 3" DBMS. It always depends on what you are doing.
    - qznc 6y ago
      SAP told me their framework is a billion LoC of Java and 30k tables in the database. That was years ago so it has probably grown further. It is only the framework, so no useful application yet.
- kgeist 6y ago
  How many of those 20k contributors worked on drivers, and how many - on the actual core (~150kloc)? Every driver is like a separate subproject, having 20k people work on hundreds/thousands of drivers (unrelated to each other) wouldn't sound as impressive.
  - thrower123 6y ago
    That's the scary thing. How many real, core contributors does even something like the Linux kernel have? People who have actually written more than a couple patches that landed and stayed in the kernel. I'd be astounded if it's much more than a hundred. Most open-source projects have one to maybe five real contributors, outside the drive-by pull requests to fix some bug.
- misterS 6y ago
  I guess (and I have no way to verify this, it's really just a guess) that the number of developers that worked on the Windows code base in its 35 year history is in the same order, if not higher.
- rcxdude 6y ago
  Consider the bit at the end of this blog post: https://devblogs.microsoft.com/oldnewthing/20180326-00/?p=98335 https://devblogs.microsoft.com/oldnewthing/20180326-00/?p=98... Bonus chatter: In a discussion of Windows source control, one person argued that git works great on large projects and gave an example of a large git repo: “For example, the linux kernel repository at roughly eight years old is 800–900MB in size, has about 45,000 files, and is considered to have heavy churn, with 400,000 commits.” I found that adorable. You have 45,000 files. Yeah, call me when your repo starts to get big. The Windows repo has over three million files. Four hundred thousand commits over eight years averages to around [130] commits per day. This is heavy churn? That’s so cute. You know what we call a day with [130] commits? “Catastrophic network outage.”
  - jka 6y ago
    To get closer to an apples-to-apples comparison, it'd be necessary to know whether the commit counts in each case include all development branches for all development groups. By design, git-based development can be highly distributed. Also, even if we normalized both cases for 'code files only' and/or 'kernel code only', there could still be architectural, code style, and development process differences that lead to different metrics for each project.
  - deleted 6y ago
    [deleted]
  - AlexCoventry 5y ago
    He's really not selling Microsoft as a fun place to work, is he?

dragonwriter 6y ago

> the only scalable software development methodology is open source “open source” isn’t a development methodology, or even a distinct set of methodologies. > The project structure, tooling and documentation that lets new contributors jump in quickly makes software development easy to scale. Plenty of open source projects don’t have that, nor is there anything restricting those things to open source projects. It is true that some open source projects, because they see the value in new developers jumping in quickly, prioritize having structure, documentation, and tooling that supports that. It’s also true that some proprietary software projects do, too, because the project owner sees value in that.

abiro 6y ago
There are books written about open source development. Don’t confuse somebody dumping source files on Github with open source development.
- oblio 6y ago
  There are a gazillion ways to develop open source. It's more like an ethos than a methodology.
  - marcosdumay 6y ago
    > There are a gazillion ways to develop open source. Yet every large project behaves in a similar way, with only a few larger variations.
    - oblio 6y ago
      That's why one of the seminal books about Open Source is the Cathedral and the Bazaar? Because "there is only one way to do things"? :-)
      - marcosdumay 6y ago
        Hum... The Bazaar on that book is a very specific way to do things. So specific that I don't think anybody really follows it.

qznc 6y ago

Since Open Source software drops liability by definition, a huge complexity factor disappears.

rakoo 6y ago

To go further I think it's not exactly open source but remote-first that made software development scalable. If you grossly simplify it to its innermost core, making development scalable means that if you have 10 times more people you can have 10 times more features/bugfixes/speed improvements/... The only way to do that is to make sure that an additional developer doesn't depend on other developers to work, and that can only happen if everything is properly documented, the build instructions are up-to-date, the processes are clear, basically anyone can start from scratch and get up to speed without the help of anyone else. That kind of organization traditionally doesn't happen in on-site companies, where newcomers are followed by senior people, they have to follow some introduction course to familiarize themselves with the processes, they have to ask many questions every day, they need to be synchronized with other persons which brings some inefficiency beacuse everyone works at their own pace, etc... This all disappears when everything is properly documentend and every contributor can work in the middle of the night if they wish. I think the Gitlab Handbook goes over this quite well and describes a framework to implement that kind of organization but the rules are retrospectively obvious for people already used to open-source (https://about.gitlab.com/company/culture/all-remote/guide/ https://about.gitlab.com/company/culture/all-remote/guide/): - write down everything - discussion should happen asynchronously. Any synchronous discussion (by text or call) should be only very small points. Whatever the type, write down the conclusions of that discussions - Everything is public (to the organization), including decisions taken, issues, processes

marcosdumay 6y ago
Yeah, remote-first is important, but it's not only it. Another very relevant factor is that people can just clone stuff, create new projects, and everything moves independently. So the open source development model has a pile of solutions for dependency management that team based development doesn't adopt. But the one thing I don't get is why team based development doesn't adopt those solution. They are not expensive. Yet, even when I was able to dictate into everybody's requirements, I wasn't able to force teams into adopting them. Instead they insisted on synchronizing themselves by much more expensive and less reliable means. My guess is that most developers never dug any deep into open source and have no idea how it's done.