7 ms·

Several years ago, the project's inadequate funding became known in the media and Stenn received partial funding from the Linux Foundation's Core Infras

by gbrown_ 10y ago

Several years ago, the project's inadequate funding became known in the media and Stenn received partial funding from the Linux Foundation's Core Infrastructure Initiative, which was started after the discovery of how the minimal resources of the OpenSSL project left systems vulnerable to the Heartbleed vulnerability. No no no no. Yes more funding and resources are good for these things but Heartbleed did not come about because of that. It came about to due broken development practices and the developers focusing on adding more features rather than working through the issues people had reported in their bug tracker. The work done in NTPsec echos this in what seems to be a repeat of OpenSSL/ LibreSSL with NTP/ NTPsec. Yes forking is the "easy way out" in these circumstances and it's a shame to see efforts split in such projects but in reality it's often what's needed to get things moving in the right direction.

falcolas 10y ago
> broken development practices and the developers focusing on adding more features rather than working through the issues Both of those problems are easily attributed to a lack of funding. Fulfilling feature requests generates money, fixing bugs does not. When you can only afford to spend 10 hours a week on a project, you're going to end up cutting corners to work around those time limits.
- HedgeMage 10y ago
  In the case of Network Time Foundation, the nonprofit wrapper for NTP classic, the problem was largely mismanagement of funding. NTF funded Harlan Stenn's work on NTP, but funded no other developers. It did, however, fund a fundraiser and two part-time administrative staff. The two administrative staff came to one FTE (full-time-equivalent) total, and I forget how much the fundraiser was working. In any case, the org had funded more FTE in overhead than in technical staff. By contrast, the temporary rescue team funded a project-manager-slash-information-security-officer, three developers, and a bright intern who did a great deal of documentation work. Granted, these were not all full-time positions, and I was able to lean on my parent organization for administrative support, so while paperwork was minimal I didn't have to fund it directly. NTPSec's staff has varied over time, but at the time I stepped down to hand the ISO role off to my successor, they had the following funded positions: a project manager, an ISO, two developers, and a sysadmin-slash-developer. (Note: not all of these were full-time positions, and some were funded by third parties rather than by NTPSec directly.) Open source infrastructure software projects, when well run, do not spend over half their resources on non-development work. It's just not responsible. It's how you lose donor confidence, and how you fail to maintain good software engineering practice even when you have the resources to do better. This has been a running theme in open source failures in the last few years: "it will be better if we throw more money at it!". Sometimes this is true: there are developers out there simply burning out for lack of resources and splitting their attention between OSS work and making a living. However, often, there is mismanagement at work, or the project doesn't have a good enough talent pool to pull from to use funds effectively when they do get funds. I love when the problems are purely technical, because it's a clean fix and everyone thanks me and I walk away quietly. When the underlying problems are social, they tend to fester, because nobody really wants to be in the hot seat over the disagreements that happened.
  - falcolas 10y ago
    That's quite the stretch, to claim that the two part time administrators is anywhere near half the value of a full time software engineer. If you take average wages, two part time administrative employees would cost you somewhere around $40,000 a year (2 * 12 * 30 * 52, rounded up). One software developer's value is conservatively $120,000 a year (likely more; $80,000 salary, $40,000 payroll taxes & benefits). By those estimated numbers (I'm not in a position to look up the actual numbers right now, but they're probably available), that's only around 25% for administrative overhead, for a three person team. Not that bad for a foundation. And certainly nowhere near "half their resources" as you claim. I'm glad that NTPSec appears to be relatively well funded, but knocking on NTF for being less well funded and having administrative work that they don't want to pawn off on an already overworked developer is pretty nasty business.
HedgeMage 10y ago
This is pretty much what happened. We spent a few months working with Mr. Stenn, and ultimately he did not agree to pursue strategies to correct the underlying problems that caused NTP's security and stability issues. Simply patching known vulns and moving on would have been a temporary solution: more vulns were lurking. NTPSec was born to give the code base another chance, to evolve with a different strategy. In the end, I tend to feel that this is a strength of OSS: different groups are free to do things different ways, and if people are paying attention, software quality should win out. Since Eric and the rest of my team started working on the NTP code base in early 2015, we've eliminated over 50% of its vulnerabilities before they were disclosed simply by applying good software engineering practice where it hadn't been. In the year before my O'Reilly presentation, it was more like 80 or 85 percent. Everything we hadn't eliminated by disclosure or discovery time was fixed promptly. There are other NTP protocol implementations besides NTP classic or NTPSec that are worth considering for some users. However, we felt that refactoring the reference implementation was necessary due to its use in many less-mainstream, but often highly-critical (in a life-critical or economically-critical or critical-to-scientific-research sense) applications. The non-NTP-related implementations don't always do what high speed trading houses need, or scientific installations built on aging but extremely precise equipment need, or controls system interfaces need, and on and on and on. We just didn't have a drop-in replacement available for all of the things that weren't web servers, workstations, and other commodity applications. The "rift" article is now subscriber-only, so I can't respond there to its many inaccuracies (I was passed a PDF by someone who cached it, this is the only way I was able to read it). I was never contacted about it by the author, and I don't feel it was a fair treatment of the subject. That's okay. I learned a long time ago that fixing a mess will make some people thank you and some people angry with you. It wouldn't have become a mess by the time I found it if there weren't a cost to fixing it. People who fear controversy will have a hard time making a difference in the world. I'm at work, but I'll do my best to answer any questions fired at me today on this thread. If there's something you want to know, ask!
- philh 10y ago
  > In the podcast, Sons depicted NTP as a faltering project run by out-of-touch developers. According to Sons, the build system was on one server whose root password had been lost. > Stenn denied many of Sons's statements outright. For example, asked about Sons's story about losing the root password, he dismissed it as "a complete fabrication." Unless either you or Stenn is outright lying (neither of which seems likely, on priors), this seems like a strange misunderstanding to crop up. Do you know what's going on with this?

by gbrown_ 10y ago

falcolas 10y ago

> broken development practices and the developers focusing on adding more features rather than working through the issues Both of those problems are easily attributed to a lack of funding. Fulfilling feature requests generates money, fixing bugs does not. When you can only afford to spend 10 hours a week on a project, you're going to end up cutting corners to work around those time limits.

HedgeMage 10y ago
In the case of Network Time Foundation, the nonprofit wrapper for NTP classic, the problem was largely mismanagement of funding. NTF funded Harlan Stenn's work on NTP, but funded no other developers. It did, however, fund a fundraiser and two part-time administrative staff. The two administrative staff came to one FTE (full-time-equivalent) total, and I forget how much the fundraiser was working. In any case, the org had funded more FTE in overhead than in technical staff. By contrast, the temporary rescue team funded a project-manager-slash-information-security-officer, three developers, and a bright intern who did a great deal of documentation work. Granted, these were not all full-time positions, and I was able to lean on my parent organization for administrative support, so while paperwork was minimal I didn't have to fund it directly. NTPSec's staff has varied over time, but at the time I stepped down to hand the ISO role off to my successor, they had the following funded positions: a project manager, an ISO, two developers, and a sysadmin-slash-developer. (Note: not all of these were full-time positions, and some were funded by third parties rather than by NTPSec directly.) Open source infrastructure software projects, when well run, do not spend over half their resources on non-development work. It's just not responsible. It's how you lose donor confidence, and how you fail to maintain good software engineering practice even when you have the resources to do better. This has been a running theme in open source failures in the last few years: "it will be better if we throw more money at it!". Sometimes this is true: there are developers out there simply burning out for lack of resources and splitting their attention between OSS work and making a living. However, often, there is mismanagement at work, or the project doesn't have a good enough talent pool to pull from to use funds effectively when they do get funds. I love when the problems are purely technical, because it's a clean fix and everyone thanks me and I walk away quietly. When the underlying problems are social, they tend to fester, because nobody really wants to be in the hot seat over the disagreements that happened.
- falcolas 10y ago
  That's quite the stretch, to claim that the two part time administrators is anywhere near half the value of a full time software engineer. If you take average wages, two part time administrative employees would cost you somewhere around $40,000 a year (2 * 12 * 30 * 52, rounded up). One software developer's value is conservatively $120,000 a year (likely more; $80,000 salary, $40,000 payroll taxes & benefits). By those estimated numbers (I'm not in a position to look up the actual numbers right now, but they're probably available), that's only around 25% for administrative overhead, for a three person team. Not that bad for a foundation. And certainly nowhere near "half their resources" as you claim. I'm glad that NTPSec appears to be relatively well funded, but knocking on NTF for being less well funded and having administrative work that they don't want to pawn off on an already overworked developer is pretty nasty business.

HedgeMage 10y ago

This is pretty much what happened. We spent a few months working with Mr. Stenn, and ultimately he did not agree to pursue strategies to correct the underlying problems that caused NTP's security and stability issues. Simply patching known vulns and moving on would have been a temporary solution: more vulns were lurking. NTPSec was born to give the code base another chance, to evolve with a different strategy. In the end, I tend to feel that this is a strength of OSS: different groups are free to do things different ways, and if people are paying attention, software quality should win out. Since Eric and the rest of my team started working on the NTP code base in early 2015, we've eliminated over 50% of its vulnerabilities before they were disclosed simply by applying good software engineering practice where it hadn't been. In the year before my O'Reilly presentation, it was more like 80 or 85 percent. Everything we hadn't eliminated by disclosure or discovery time was fixed promptly. There are other NTP protocol implementations besides NTP classic or NTPSec that are worth considering for some users. However, we felt that refactoring the reference implementation was necessary due to its use in many less-mainstream, but often highly-critical (in a life-critical or economically-critical or critical-to-scientific-research sense) applications. The non-NTP-related implementations don't always do what high speed trading houses need, or scientific installations built on aging but extremely precise equipment need, or controls system interfaces need, and on and on and on. We just didn't have a drop-in replacement available for all of the things that weren't web servers, workstations, and other commodity applications. The "rift" article is now subscriber-only, so I can't respond there to its many inaccuracies (I was passed a PDF by someone who cached it, this is the only way I was able to read it). I was never contacted about it by the author, and I don't feel it was a fair treatment of the subject. That's okay. I learned a long time ago that fixing a mess will make some people thank you and some people angry with you. It wouldn't have become a mess by the time I found it if there weren't a cost to fixing it. People who fear controversy will have a hard time making a difference in the world. I'm at work, but I'll do my best to answer any questions fired at me today on this thread. If there's something you want to know, ask!

philh 10y ago
> In the podcast, Sons depicted NTP as a faltering project run by out-of-touch developers. According to Sons, the build system was on one server whose root password had been lost. > Stenn denied many of Sons's statements outright. For example, asked about Sons's story about losing the root password, he dismissed it as "a complete fabrication." Unless either you or Stenn is outright lying (neither of which seems likely, on priors), this seems like a strange misunderstanding to crop up. Do you know what's going on with this?