6 ms·

Quite the dog's breakfast: > In the first version of the software, there were 70 full copies of 4 different OpenSSL versions, ranging from 0.9.8 to 1.0.2k (inc

by abraae 8y ago

Quite the dog's breakfast: > In the first version of the software, there were 70 full copies of 4 different OpenSSL versions, ranging from 0.9.8 to 1.0.2k (including one from a vendor SDK) with partial copies of 14 versions, ranging from 0.9.7d to 1.0.2k, those partial copies numbering 304. Fragments of 10 versions, ranging from 0.9.6 to 1.0.2k, were also found across the codebase, with these normally being small sets of files that had been copied to import some particular functionality."

wjnc 8y ago
Can someone explain to me how you can understand the culture in a team of programmers that leads to this being acceptable? Just a few small guidelines could help alleviate things like this, right? Modularization and a tiny sprinkle of code review and this is solved. Perhaps some (big) initial effort to clean up if you've started with code bases of different individuals with the goal of integration.
- dagw 8y ago
  understand the culture in a team of programmers In large companies there is never 'a' team of programmers, there are N teams of programmers with varying amounts of communications and cooperation going on between them, where "varying" often includes being completely unaware of the other teams existence. N different teams working independently of each other each implement a variation of feature X to solve their own problem. The someone decides it's time to merge the whole thing together and it's easier to just use the N different implementations of X rather than to try to refactor and debug a single implementation of X that works for all N cases.
  - dalore 8y ago
    Don't forget N varies over time. Programmers come and go.
- TheHegemon 8y ago
  What I've noticed is that in general that as long as there some activity (even if it's bad) it is considered success. I'm currently knee-deep in a code base where the exact same function was nearly copied several dozen times and trying to fix it gets shot down as a "waste of time". I say nearly because in some places the same exact variable is misspelled many different ways.
- bloopernova 8y ago
  Using this: "Just get it out the door and we'll fix it later". I've heard that from most managers I've known. Instead of this: "We need to audit then refactor the existing codebase to bring it up release quality. It will delay release by 2 months, but we are dealing with customer PII so we have to be as stable as possible to prevent break-ins." Or this: "Don't deploy the prototype to production"
- PhantomGremlin 8y ago
  the culture in a team of programmers The "culture" behind this is endemic to industry and isn't anything unique to Huawei. People have already commented on this not being a "team". Let me focus on "programmers". What probably happened is that many of Huawei's programmers are EEs who happened to take a programming course or two in college. But learning to think and behave like a good programmer is much more than that. It requires a certain mindset. It may require a certain amount of avocation and enthusiasm, beyond what is taught in an introductory programming course. It requires one to grok good software practices. Most non-CS majors in college (USA or China or whatever country) probably don't get much of that cultural indoctrination during the few programming courses they take. The engineer wants to pass/ace the programming course. Learning the aesthetics or nuances isn't a priority. Programming is just too easy. Type a few lines into a file, compile, debug syntax errors, tweak until you get something that looks right. There's no rigor involved. No style. No aesthetic. Don Knuth called it "The Art of Computer Programming". I submit that many engineers don't think of programming as an art. It's just a means to an end. E.g. here's an example from a large codebase that was extensively reviewed. The context was Toyota's unintended acceleration problem of about 15 years ago. The design review found things like: Other egregious deviations from standard practice were the number of global variables in the system. (A variable is a location in memory that has a number in it. A global variable is any piece of software anywhere in the system can get to that number and read it or write it.) The academic standard is zero. Toyota had more than 10,000 global variables.[1] None of this will ever change until and unless a company's top level management understands the problem and mandates change. Additionally, there's always an incredible push to bring new products to market. That's 1000x more important than it is to follow best software practices. [1] http://www.safetyresearch.net/blog/articles/toyota-unintended-acceleration-and-big-bowl-%E2%80%9Cspaghetti%E2%80%9D-code http://www.safetyresearch.net/blog/articles/toyota-unintende...
  - jshowa3 8y ago
    Sometimes the academic standard isn't realistic because software that large is often written by many people in different stages with little thought to architecture. Coupled with the fact that software design is so ambiguous (i.e. why you get papers with the title meme "considered harmful" to the n power for certain software practices) I don't blame people for being confused. For example, people can make legitimate cases for use of goto, continue, and break which academics consider bad practice because it interrupts program flow. My guess is the primary reason it had 10,000 global variables is because of lack of knowledge, but also lack of structure. And to restructure the code with that many pieces of global memory would probably require extensively rewriting. And I can't tell you how many articles I've read that tell you to never rewrite applications because it implies a death sentence for the company (ie. Netscape).
- onethree 8y ago
  Someone once described the difference between working with Cisco and with Huawei as being (and I'm paraphrasing) with Cisco if you have a feature request or bug report, you'll submit it through to your account manager, it will work its way through the Cisco machine, and eventually it might get added/fixed. With Huawei, if you spend enough they'll just give you someone to develop whatever it is you want. That came from someone with a ccie so I guess they have some level of interest in making Cisco sound more organised, but it would explain why Huawei have issues like this
- AFascistWorld 8y ago
  That it worked, and it's enough.
deleted 8y ago
[deleted]
zimmerfrei 8y ago
That sounds like bad software engineering to most, but if each OpenSSL copy was actually in a separate Docker image, reactions would suddenly be much milder I suspect.
- yjftsjthsd-h 8y ago
  To be fair, while much of that boils down to what is considered normal, Docker images are also far easier to update and maintain then manually grafted together libraries.
- xemdetia 8y ago
  This is a very upsetting comment. Docker is one of the things that is making modern security compliance about 10 times harder than it was before even on 10k-100k systems. Especially in government contexts where using the approved/certified/correct cryptographic modules (FIPS 140-2) is an absolute requirement. FIPS 140-2 in particular requires not only the appropriate package to be available (commonly OpenSSL) but also for the application to opt into a FIPS 140-2 mode after linking. There is also the fact that cryptographic libraries that are certified are often on a 2-3 month delay from the non-certified modules. You have to verify that the Host OS's kernel, the runtime that is providing the containing environment, and every single Docker image that is used to run an application, ensure that each application is running in an appropriate FIPS 140-2 mode inside the container, and also verify that every container is being appropriately kept up to date. Using Docker does not eschew responsibilities you have to ensure that your OpenSSL and other materials is up to date, especially since Docker doesn't give you any protections when it comes to encryption over the wire or encryption with data at rest which is exactly where a cryptographic library with a vulnerability like in the original article leaves a vulnerability where there shouldn't be one.
diggan 8y ago
I think this is a widespread problem, especially with things like npm where small modules are advocated for. Taking a random example from GitHub, installing the npm modules for Polymer/polymer yields duplicate dependencies of @types/node (4.9.1, 6.0.118, 9.6.41, 10.12.18, 10.12.21) dom5 (1.3.6, 1.1.0, 3.0.1) and many more modules... I'm pretty sure you can find similar examples from most popular open source projects.
- viraptor 8y ago
  Including multiple versions of openssl is not as trivial as adding another dependency in the packages file. It actually takes a bit of effort / linker fun to include multiple versions without a conflict. It also takes a special kind of person to both be capable of this and not realise this shouldn't be done.
  - twic 8y ago
    My guess would be that the firmware includes multiple binaries which run as separate processes. If there are 70 tools which speak SSL, it wouldn't be a huge surprise if there were 70 copies of OpenSSL.
  - yjftsjthsd-h 8y ago
    Yeah; it's kind of impressive, in a way. It reminds me of my reaction when I first found out that Mac OS / Darwin's kernel was built by taking Mach and then grafting parts of BSD on top: "Wait, you can do that!?"

by abraae 8y ago

wjnc 8y ago

Can someone explain to me how you can understand the culture in a team of programmers that leads to this being acceptable? Just a few small guidelines could help alleviate things like this, right? Modularization and a tiny sprinkle of code review and this is solved. Perhaps some (big) initial effort to clean up if you've started with code bases of different individuals with the goal of integration.

dagw 8y ago
understand the culture in a team of programmers In large companies there is never 'a' team of programmers, there are N teams of programmers with varying amounts of communications and cooperation going on between them, where "varying" often includes being completely unaware of the other teams existence. N different teams working independently of each other each implement a variation of feature X to solve their own problem. The someone decides it's time to merge the whole thing together and it's easier to just use the N different implementations of X rather than to try to refactor and debug a single implementation of X that works for all N cases.
- dalore 8y ago
  Don't forget N varies over time. Programmers come and go.
TheHegemon 8y ago
What I've noticed is that in general that as long as there some activity (even if it's bad) it is considered success. I'm currently knee-deep in a code base where the exact same function was nearly copied several dozen times and trying to fix it gets shot down as a "waste of time". I say nearly because in some places the same exact variable is misspelled many different ways.
bloopernova 8y ago
Using this: "Just get it out the door and we'll fix it later". I've heard that from most managers I've known. Instead of this: "We need to audit then refactor the existing codebase to bring it up release quality. It will delay release by 2 months, but we are dealing with customer PII so we have to be as stable as possible to prevent break-ins." Or this: "Don't deploy the prototype to production"
PhantomGremlin 8y ago
the culture in a team of programmers The "culture" behind this is endemic to industry and isn't anything unique to Huawei. People have already commented on this not being a "team". Let me focus on "programmers". What probably happened is that many of Huawei's programmers are EEs who happened to take a programming course or two in college. But learning to think and behave like a good programmer is much more than that. It requires a certain mindset. It may require a certain amount of avocation and enthusiasm, beyond what is taught in an introductory programming course. It requires one to grok good software practices. Most non-CS majors in college (USA or China or whatever country) probably don't get much of that cultural indoctrination during the few programming courses they take. The engineer wants to pass/ace the programming course. Learning the aesthetics or nuances isn't a priority. Programming is just too easy. Type a few lines into a file, compile, debug syntax errors, tweak until you get something that looks right. There's no rigor involved. No style. No aesthetic. Don Knuth called it "The Art of Computer Programming". I submit that many engineers don't think of programming as an art. It's just a means to an end. E.g. here's an example from a large codebase that was extensively reviewed. The context was Toyota's unintended acceleration problem of about 15 years ago. The design review found things like: Other egregious deviations from standard practice were the number of global variables in the system. (A variable is a location in memory that has a number in it. A global variable is any piece of software anywhere in the system can get to that number and read it or write it.) The academic standard is zero. Toyota had more than 10,000 global variables.[1] None of this will ever change until and unless a company's top level management understands the problem and mandates change. Additionally, there's always an incredible push to bring new products to market. That's 1000x more important than it is to follow best software practices. [1] http://www.safetyresearch.net/blog/articles/toyota-unintended-acceleration-and-big-bowl-%E2%80%9Cspaghetti%E2%80%9D-code http://www.safetyresearch.net/blog/articles/toyota-unintende...
- jshowa3 8y ago
  Sometimes the academic standard isn't realistic because software that large is often written by many people in different stages with little thought to architecture. Coupled with the fact that software design is so ambiguous (i.e. why you get papers with the title meme "considered harmful" to the n power for certain software practices) I don't blame people for being confused. For example, people can make legitimate cases for use of goto, continue, and break which academics consider bad practice because it interrupts program flow. My guess is the primary reason it had 10,000 global variables is because of lack of knowledge, but also lack of structure. And to restructure the code with that many pieces of global memory would probably require extensively rewriting. And I can't tell you how many articles I've read that tell you to never rewrite applications because it implies a death sentence for the company (ie. Netscape).
onethree 8y ago
Someone once described the difference between working with Cisco and with Huawei as being (and I'm paraphrasing) with Cisco if you have a feature request or bug report, you'll submit it through to your account manager, it will work its way through the Cisco machine, and eventually it might get added/fixed. With Huawei, if you spend enough they'll just give you someone to develop whatever it is you want. That came from someone with a ccie so I guess they have some level of interest in making Cisco sound more organised, but it would explain why Huawei have issues like this
AFascistWorld 8y ago
That it worked, and it's enough.

deleted 8y ago

[deleted]

zimmerfrei 8y ago

That sounds like bad software engineering to most, but if each OpenSSL copy was actually in a separate Docker image, reactions would suddenly be much milder I suspect.

yjftsjthsd-h 8y ago
To be fair, while much of that boils down to what is considered normal, Docker images are also far easier to update and maintain then manually grafted together libraries.
xemdetia 8y ago
This is a very upsetting comment. Docker is one of the things that is making modern security compliance about 10 times harder than it was before even on 10k-100k systems. Especially in government contexts where using the approved/certified/correct cryptographic modules (FIPS 140-2) is an absolute requirement. FIPS 140-2 in particular requires not only the appropriate package to be available (commonly OpenSSL) but also for the application to opt into a FIPS 140-2 mode after linking. There is also the fact that cryptographic libraries that are certified are often on a 2-3 month delay from the non-certified modules. You have to verify that the Host OS's kernel, the runtime that is providing the containing environment, and every single Docker image that is used to run an application, ensure that each application is running in an appropriate FIPS 140-2 mode inside the container, and also verify that every container is being appropriately kept up to date. Using Docker does not eschew responsibilities you have to ensure that your OpenSSL and other materials is up to date, especially since Docker doesn't give you any protections when it comes to encryption over the wire or encryption with data at rest which is exactly where a cryptographic library with a vulnerability like in the original article leaves a vulnerability where there shouldn't be one.

diggan 8y ago

I think this is a widespread problem, especially with things like npm where small modules are advocated for. Taking a random example from GitHub, installing the npm modules for Polymer/polymer yields duplicate dependencies of @types/node (4.9.1, 6.0.118, 9.6.41, 10.12.18, 10.12.21) dom5 (1.3.6, 1.1.0, 3.0.1) and many more modules... I'm pretty sure you can find similar examples from most popular open source projects.

viraptor 8y ago
Including multiple versions of openssl is not as trivial as adding another dependency in the packages file. It actually takes a bit of effort / linker fun to include multiple versions without a conflict. It also takes a special kind of person to both be capable of this and not realise this shouldn't be done.
- twic 8y ago
  My guess would be that the firmware includes multiple binaries which run as separate processes. If there are 70 tools which speak SSL, it wouldn't be a huge surprise if there were 70 copies of OpenSSL.
- yjftsjthsd-h 8y ago
  Yeah; it's kind of impressive, in a way. It reminds me of my reaction when I first found out that Mac OS / Darwin's kernel was built by taking Mach and then grafting parts of BSD on top: "Wait, you can do that!?"