5 ms·

> Mark Perlin, is said to have argued against source code analysis by claiming that the program, consisting of 170,000 lines of MATLAB code, is so dense it woul

> Mark Perlin, is said to have argued against source code analysis by claiming that the program, consisting of 170,000 lines of MATLAB code, is so dense it would take eight and a half years to review at a rate of ten lines an hour. So it’s definitely riddled with bugs. And I can’t imagine that much matlab code following rigorous software engineering practices.

bryanrasmussen 6y ago
and surprisingly all of the code is of equal importance so you really need to review each line sequentially! Instead of finding stuff that you think is most likely to relate to what you're trying to figure out and debug from there. Wow I would like to see this marvel of engineering myself!
- Scandiravian 6y ago
  Kind of makes me wonder: if the argument is that the code is too complex to review and understand, does that mean the company is not doing code reviews themselves?
  - michaelmior 6y ago
    If the code was developed in less than 8.5 years, it seems like the answer must be yes based on their previous claims.
    - gambiting 6y ago
      Well I mean if something requires 8 man years of work you can do it in a year with 8 people. Not always, but usually.
      - agravier 6y ago
        On the contrary, such linear scaling would be quite exceptional. I'm speaking from experience but you don't need to trust me; I invite you read any book on software engineering management, starting from The Mythical Man-Month by Brooks.
        bryanrasmussen 6y ago
        the mythical man month was first published in 1975, I think the typical applications programmers work on today have changed significantly since then and encompass many different disciplines (to be thought professional) - so many disciplines that one developer is likely to be the master of all. It is true that there is a communication overhead to adding more people so it will not scale linearly, but if a single developer has taken 8 years to build something in our era it seems likely that having 8 people might get it done say 1 and a half to two years.
      - markwkw 6y ago
        I would claim that if something requires 8 man years, that it will most definitely take more than a year to develop with 8 people. Communication takes time, coordination takes time, there is an incremental cost to each news person added to a team. From experience, perhaps with 2-3 people who happen to gel well together you may get close to proportional scaling of output, but with 8 it’s really unlikely in the real world.
      - watwut 6y ago
        I would find it extremely unlikely to work that way.
  - bryanrasmussen 6y ago
    this has actually only happened to me a couple times but it has happened - someone tells me Bryan, go look at the code X did in Y, figure out if we refactor. X would then tell me - that code is really complicated is full of algorithms! I go and look at the code realize that for what it is trying to do can be cut down from 10 pages of printed code to less than 1 and it was incredibly simple what actually needed to be done. In short when someone tells me the stuff is too complicated because too clever and advanced I tend to disbelieve them. that said I have of course written my too complicated stuff lots of times, but if asked I don't say it was because I'm clever. names anonymized so as to not accidentally hurt anyone's feelings. on edit: actually one time the code was clever but not especially difficult, they just used the algorithms line because they didn't want anyone messing with their stuff.
    - Scandiravian 6y ago
      I think there's a bias towards judging things to be "clever" if they're hard to understand It's a cliche to have a "what idiot wrote this" outburst, then realise it's your own code, because most of us have written our fair share of "clever" code My boss explicitly stated that he doesn't want to see any "clever" or "smart" code in our product - write code based on simple fundamentals, benchmark before deciding to optimise, and be respectful in your reviews I like my boss a lot
  - watwut 6y ago
    You can do code review on every single commit while not being able to make overall analysis of the massive codebase.
    - Scandiravian 6y ago
      I completely agree, but if the reviewer isn't able to (with some amount of accuracy) predict the impact the committed code will have on overall behaviour, then there's very limited value on doing the review in the first place
      - watwut 6y ago
        In any larger project, the reviewer are not able to predict impact from reading commit. More importantly, typical reviewer have only small partia area where he has good idea about which commit is bad idea. He however does not understand whole codebase. Knowing what the whole does and knowing what my module does are two different things.
        Scandiravian 6y ago
        I again completely agree with you Looking back at my reply, I think I should have added a bit of background to clarify my comment My master's degree is in bioinformatics and I worked in the biotech industry until about a year ago. I mainly worked as a consultant for top 20 pharma companies, but also did work on different in-house projects and in academia From my experience in the industry, I find it very unlikely that the software mentioned in the article is structured in a modular way. I've yet to see good software practices outside one or two academic projects. Most pharma companies still use copying and renaming folders as version control. Naturally I'm sceptical of any code coming from the biotech industry On top of that, it's written in MATLAB. I have only ever seen this used by statisticians and university researchers, never by software engineers I'm therefore willing to bet, that when the reviewers open the source code, they'll find unstructured mess of spaghetti code, that has never been refactored, reviewed or tested So yes - I agree in all your points, but I find it unlikely that they're being applied to this particular project
syshum 6y ago
I also cant image anyone thinking that would be a winning legal argument... "This software is too complex to look at so just trust us" Really... that is what they went with...
- avs733 6y ago
  unfortunately, the technical illiteracy of much of the US judiciary has made this effective.
- delecti 6y ago
  Yeah, that's one of those "defenses" that feels like a confession.
Scandiravian 6y ago
My exact thoughts. This sounds like a classic example of launching a prototype created by domain specialists (biostatisticians and bioinformaticians) as production software and skipping on the expensive stuff, like sound development practices
- gww 6y ago
  There is so little emphasis in production software development in bioinformatics and biostatistics. Despite a lot of groups open sourcing their code it's is nearly unusable and not reproducible due to hard coding, ignoring edge cases, and dumping the majority of the code on a single giant R or python function.
  - Scandiravian 6y ago
    That's my experience as well. My master's is in bioinformatics and I worked for several years in biotech. I got frustrated because my concerns that my team's development practices were causing issues on a regular basis, were ignored. I was continuously able to predict what issues we would run into, but no-one seemed to care - I even had a manager tell me, that it was good that our software was buggy, since the client would continue paying us to fix it I've since left the biotech industry. There's a limit to how many times I want to run my head against that particular wall
  - natechols 6y ago
    It's a real problem, and I've been struggling with it for two decades, but even so I am legitimately impressed (and not in a good way) if they have 170,000 lines of Matlab code in their production software. That takes a really special combination of productivity and cluelessness, even for academic specialists. Regardless of the facts of this particular case, it should be absolutely horrifying that anyone's freedom is left up to a gigantic pile of unaudited Matlab code. (That said I am almost certain he added some zeros to the number, I have a hard time imagining what they're doing that could be that complex.)
    - Scandiravian 6y ago
bitexploder 6y ago
I have done a lot of source code review in my time. For security assessments. Our general rule of thumb was about 10k lines per week that we can really get deep on. 10 lines an hour would only be for the most dense code and critical path stuff. They will need a reviewer that knows the domain (DNA), but it’s perfectly reasonable to review that code on a weeks/months time scale, definitely not years.
moron4hire 6y ago
That's like an accountant, accused of embezalling, refusing to hand over the ledgers because "there are just too many records to go through". Yeah-no, that's kind of the whole point. We want to find out what you've been hiding in that wall of paper.
partyboat1586 6y ago
>170,000 lines of MATLAB code This is a deep problem. Many scientists don't understand software engineering and more and more need to write bigger and bigger programs. And most of the time they don't open source their code. Open source science.

bryanrasmussen 6y ago

and surprisingly all of the code is of equal importance so you really need to review each line sequentially! Instead of finding stuff that you think is most likely to relate to what you're trying to figure out and debug from there. Wow I would like to see this marvel of engineering myself!

Scandiravian 6y ago
Kind of makes me wonder: if the argument is that the code is too complex to review and understand, does that mean the company is not doing code reviews themselves?
- michaelmior 6y ago
  If the code was developed in less than 8.5 years, it seems like the answer must be yes based on their previous claims.
  - gambiting 6y ago
    Well I mean if something requires 8 man years of work you can do it in a year with 8 people. Not always, but usually.
    - agravier 6y ago
      On the contrary, such linear scaling would be quite exceptional. I'm speaking from experience but you don't need to trust me; I invite you read any book on software engineering management, starting from The Mythical Man-Month by Brooks.
      - bryanrasmussen 6y ago
        the mythical man month was first published in 1975, I think the typical applications programmers work on today have changed significantly since then and encompass many different disciplines (to be thought professional) - so many disciplines that one developer is likely to be the master of all. It is true that there is a communication overhead to adding more people so it will not scale linearly, but if a single developer has taken 8 years to build something in our era it seems likely that having 8 people might get it done say 1 and a half to two years.
    - markwkw 6y ago
      I would claim that if something requires 8 man years, that it will most definitely take more than a year to develop with 8 people. Communication takes time, coordination takes time, there is an incremental cost to each news person added to a team. From experience, perhaps with 2-3 people who happen to gel well together you may get close to proportional scaling of output, but with 8 it’s really unlikely in the real world.
    - watwut 6y ago
      I would find it extremely unlikely to work that way.
- bryanrasmussen 6y ago
  this has actually only happened to me a couple times but it has happened - someone tells me Bryan, go look at the code X did in Y, figure out if we refactor. X would then tell me - that code is really complicated is full of algorithms! I go and look at the code realize that for what it is trying to do can be cut down from 10 pages of printed code to less than 1 and it was incredibly simple what actually needed to be done. In short when someone tells me the stuff is too complicated because too clever and advanced I tend to disbelieve them. that said I have of course written my too complicated stuff lots of times, but if asked I don't say it was because I'm clever. names anonymized so as to not accidentally hurt anyone's feelings. on edit: actually one time the code was clever but not especially difficult, they just used the algorithms line because they didn't want anyone messing with their stuff.
  - Scandiravian 6y ago
    I think there's a bias towards judging things to be "clever" if they're hard to understand It's a cliche to have a "what idiot wrote this" outburst, then realise it's your own code, because most of us have written our fair share of "clever" code My boss explicitly stated that he doesn't want to see any "clever" or "smart" code in our product - write code based on simple fundamentals, benchmark before deciding to optimise, and be respectful in your reviews I like my boss a lot
- watwut 6y ago
  You can do code review on every single commit while not being able to make overall analysis of the massive codebase.
  - Scandiravian 6y ago
    I completely agree, but if the reviewer isn't able to (with some amount of accuracy) predict the impact the committed code will have on overall behaviour, then there's very limited value on doing the review in the first place
    - watwut 6y ago
      In any larger project, the reviewer are not able to predict impact from reading commit. More importantly, typical reviewer have only small partia area where he has good idea about which commit is bad idea. He however does not understand whole codebase. Knowing what the whole does and knowing what my module does are two different things.
      - Scandiravian 6y ago
        I again completely agree with you Looking back at my reply, I think I should have added a bit of background to clarify my comment My master's degree is in bioinformatics and I worked in the biotech industry until about a year ago. I mainly worked as a consultant for top 20 pharma companies, but also did work on different in-house projects and in academia From my experience in the industry, I find it very unlikely that the software mentioned in the article is structured in a modular way. I've yet to see good software practices outside one or two academic projects. Most pharma companies still use copying and renaming folders as version control. Naturally I'm sceptical of any code coming from the biotech industry On top of that, it's written in MATLAB. I have only ever seen this used by statisticians and university researchers, never by software engineers I'm therefore willing to bet, that when the reviewers open the source code, they'll find unstructured mess of spaghetti code, that has never been refactored, reviewed or tested So yes - I agree in all your points, but I find it unlikely that they're being applied to this particular project

syshum 6y ago

I also cant image anyone thinking that would be a winning legal argument... "This software is too complex to look at so just trust us" Really... that is what they went with...

avs733 6y ago
unfortunately, the technical illiteracy of much of the US judiciary has made this effective.
delecti 6y ago
Yeah, that's one of those "defenses" that feels like a confession.

Scandiravian 6y ago

My exact thoughts. This sounds like a classic example of launching a prototype created by domain specialists (biostatisticians and bioinformaticians) as production software and skipping on the expensive stuff, like sound development practices

gww 6y ago
There is so little emphasis in production software development in bioinformatics and biostatistics. Despite a lot of groups open sourcing their code it's is nearly unusable and not reproducible due to hard coding, ignoring edge cases, and dumping the majority of the code on a single giant R or python function.
- Scandiravian 6y ago
  That's my experience as well. My master's is in bioinformatics and I worked for several years in biotech. I got frustrated because my concerns that my team's development practices were causing issues on a regular basis, were ignored. I was continuously able to predict what issues we would run into, but no-one seemed to care - I even had a manager tell me, that it was good that our software was buggy, since the client would continue paying us to fix it I've since left the biotech industry. There's a limit to how many times I want to run my head against that particular wall
- natechols 6y ago
  It's a real problem, and I've been struggling with it for two decades, but even so I am legitimately impressed (and not in a good way) if they have 170,000 lines of Matlab code in their production software. That takes a really special combination of productivity and cluelessness, even for academic specialists. Regardless of the facts of this particular case, it should be absolutely horrifying that anyone's freedom is left up to a gigantic pile of unaudited Matlab code. (That said I am almost certain he added some zeros to the number, I have a hard time imagining what they're doing that could be that complex.)
  - Scandiravian 6y ago

bitexploder 6y ago

I have done a lot of source code review in my time. For security assessments. Our general rule of thumb was about 10k lines per week that we can really get deep on. 10 lines an hour would only be for the most dense code and critical path stuff. They will need a reviewer that knows the domain (DNA), but it’s perfectly reasonable to review that code on a weeks/months time scale, definitely not years.

moron4hire 6y ago

That's like an accountant, accused of embezalling, refusing to hand over the ledgers because "there are just too many records to go through". Yeah-no, that's kind of the whole point. We want to find out what you've been hiding in that wall of paper.

partyboat1586 6y ago

>170,000 lines of MATLAB code This is a deep problem. Many scientists don't understand software engineering and more and more need to write bigger and bigger programs. And most of the time they don't open source their code. Open source science.