11 ms·

This is what happens when devs are presented with a very complicated problem, extremely short deadline and enormous amount of pressure.

gruez 9y ago
>extremely short deadline they had 6 months.
- sddfd 9y ago
  Are you sure that that particular team inside Microsoft had full six months? Intel had 6 months.
  - gerdesj 9y ago
    I'm pretty sure they had around six months give or take a day or so of oh shit in Intel. Of course, Intel may have actually simply broken the glass on a dusty old plan of action "In the event of ..."
    - zer00eyz 9y ago
      Disaster plans are funny things. I have had the misfortune of having to pull them out twice in my career - in both cases they offered little in the way of guidance for the particular situation that came up. The set of unknown unknowns that are typically missed make most of them unless in all but the most narrow of cases, because many companies write them, and then forget them. Especialy if they are as large as intel.
      - gerdesj 9y ago
        Very true, although I'm glad to say I have not had to break out one of my own yet for real. My first experience of a full on DR test was pretty humbling - NetWare servers backed up by the Unix troops via Legato. It turned out that the backups were good but restored at a pathetically slow speed (no reflection on the Unix systems but I suspect the Novell TSAs were a bit shag at the time). We updated "time to restore" estimations and moved on, after adding one or two other results of lessons learned. Do test your plans (this is not aimed at you personally zer00eyz - you probably know better than most). There are a lot of unknowns but the basic model of a real DR plan is pretty sound these days, if you can afford it or wing it in some way. An example: Another site, a suitable distance away. On that site there is enough infra to run the basics - wifi, a few ethernet ports, telephony etc. There should also be enough hypervisor and storage for that. Some backups are delivered there as well as on site. Hypervisor replicas are created from the backups (or directly) depending on RPO requirements and bandwidth available. The only thing that should be able to routinely access the backup files is the backup system (certainly not "Domain Admins" or other such nonsense". Ensure that what is written is verified. Now test it 8) .... regularly
        zer00eyz 9y ago
        Ok now I have to share a story... The company in question had a rather large on site server room (raised floor, fire suppression) and a massive generator to deal with any power issues as well as redundant connectivity. This room was literally the backup incase their "real" data center went off line. The problem is that the room was "convenient" so there were plenty of things that lived ONLY there (mistake one) - When the substation for the office went, and the generator started everything looked fine. The problem was that no one had ever run the generator for that long... after a few hours it simply crapped out (over heated, problem two). A quick trip to home depot got them generators and extension cords that let them get the few boxes that were critical back up - however one box decided to not only fault, but to take it's data with it. This is when I got a rather frantic call "did I still have the code from the project I did?" - they offered to cut me a check for $2000 if I would go home right then and simply LOOK for it. Lucky for them I had it - and the continuity portion of the DR plan got revisited. In hind sight after I said I had the code, I probably could have asked them to put another zero on the end of the check and they would have done it just to be a functioning business come 6am.
        gerdesj 9y ago
        I didn't even have to show some leg to get you to recant the dit. Thank you - I'm happy to listen to (nearly) everything. "I probably could have asked them to put another zero" - ahem that's not the IT Consultant's Way exactly. We have far more polite ways of extracting loot. We are not lawyers and should have morals.
      - TeMPOraL 9y ago
        Reminds me of that The Expanse quote: "I have a file with 900 pages of analysis and contingency plans for war with Mars, including fourteen different scenarios about what to do if they develop an unexpected new technology. My file for what to do if an advanced alien species comes calling is three pages long, and it begins with 'Step 1: Find God'."
  - campuscodi 9y ago
    Microsoft had 2-3 months, tops.
    - gerdesj 9y ago
      Source, proof of assertion?
      - bonzini 9y ago
        As far as I know he's right. The news was given first to Amazon and Microsoft sometime in August. Consider one month for testing and preparing for release, that gives three months to build a solution for all supported operating systems. Two months to do it for the most recent version and one month for backporting to the older ones sounds about right. Maybe a few weeks more, but that's it.
      - campuscodi 9y ago
        Intel's own press releases. I'm not gonna go digging into old links just because some random dude on the Internet can't use Google.
- Aloha 9y ago
  If the problem is complex enough, six months may be a short deadline.
  - azinman2 9y ago
    Especially since it comes as a surprise, and there’s already an existing train on its way to the next station with its own timetable.
    - cakes 9y ago
      that surprise could include "management says we don't have to do anything :/" <5.99 months pass>, management: "we have to patch this and it needs to be done yesterday."
      - TeMPOraL 9y ago
        That would be extreme, but I can entirely imagine it taking many weeks for the true importance of this problem to correctly propagate across all management levels.
- maltalex 9y ago
  > they had 6 months. You say that like 6 months is automatically a lot of time. "She had 6 months to give birth". Yeah, only it takes 9, so 6 is short. Consider the scope and depth of the issue and the fact that they probably couldn't involve too many people on this effort.
  - walterbell 9y ago
    Other operating system maintainers had only days/weeks in Jan 2018.
    - deleted 9y ago
      [deleted]
    - loufe 9y ago
      We're all throwing darts in the dark here with regards to the resource they gave the problem and its difficulty. I think the real takeaway is just that it could be something other than just stupidity through and through.
    - mannykannot 9y ago
      Unless you can be sure their response solved their problem without introducing others, it's not evidence of sufficient time.
  - dmix 9y ago
    A single person with a fixed 9 month biological timeframe is the analogy you chose to compare 6 months of a billion dollar company's software development time by potentially hundreds of developers (for better or worse) but importantly for an extremely critical class of bugs and therefore that's "just how it is"? Come on, software is hard but when you fix a vulnerability and expose a far far worse one, and had months to plan, execute, and test it, then you are most certainly justified in being criticized. It's not like we're saying the code is shoddy and needs work, which is entirely excusable in a short timeframe. It's that they've left users far worse off in the end then from where they started.
    - 9y ago
- yashap 9y ago
  I wonder how long the actual devs fixing it had? From what I hear from friends who work there, Microsoft is a sprawling bureaucracy with many layers of management, where decisions are far from quick. I'd imagine that after Intel/whoever let Microsoft know about the exploits, it went through many levels of prioritization, negotiation about which team would work on it, not being brought into sprints because of other features already being worked on, etc. Most likely there were people with minimal knowledge of the relevant tech making all these prioritization decisions. Wouldn't shock me at all if there was very little actual dev work done for the first few months, and then it was all super rushed at the end. Quite possibly the devs with the required knowledge didn't even know this was in the pipeline for months. That's par for the course at every decently large company I've worked at (i.e. 100+ devs), and at a beast like Microsoft I imagine it'd be way worse.
  - lmilcin 9y ago
    I remember that Microsoft was able to deliver critical fixes practically overnight. This assumed that once you see the problem the fix is pretty straightforward. Unfortunately Spectre and Meltdown aren't straightforward and go to the very heart of how the OS works. It's not at all easy to fix this when you have enormous amount of software working on top of it depending on every little quirk your solution provides.
waldbeere 9y ago
or maybe because M$ change name to hindusoft
gwbas1c 9y ago
These kinds of things should be part of an automated test suite. Specifically, the kind of tests that were written years ago. Honestly, Microsoft is really big into automated testing. I'm surprised this slipped through.
- astrange 9y ago
  I don't think there are any OS kernels that practice test-driven development - most of them don't even have code coverage working. It's also very hard to test for a problem you haven't thought of yet.
yuhong 9y ago
Yea, it is probably the biggest change to the Windows kernel in a security update.
Tobba_ 9y ago
This is what happens when you don't have a QA department.
- efdee 9y ago
  Are you implying that Microsoft doesn't do QA?
  - Tobba_ 9y ago
    If they do, whatever issues they're occupied with finding would call for an exorcism.
- bonzini 9y ago
  This is something that you find through code review, not testing. Apart from regression testing, but that presupposes that you encountered the issue before.

gruez 9y ago

>extremely short deadline they had 6 months.

sddfd 9y ago
Are you sure that that particular team inside Microsoft had full six months? Intel had 6 months.
- gerdesj 9y ago
  I'm pretty sure they had around six months give or take a day or so of oh shit in Intel. Of course, Intel may have actually simply broken the glass on a dusty old plan of action "In the event of ..."
  - zer00eyz 9y ago
    Disaster plans are funny things. I have had the misfortune of having to pull them out twice in my career - in both cases they offered little in the way of guidance for the particular situation that came up. The set of unknown unknowns that are typically missed make most of them unless in all but the most narrow of cases, because many companies write them, and then forget them. Especialy if they are as large as intel.
    - gerdesj 9y ago
      Very true, although I'm glad to say I have not had to break out one of my own yet for real. My first experience of a full on DR test was pretty humbling - NetWare servers backed up by the Unix troops via Legato. It turned out that the backups were good but restored at a pathetically slow speed (no reflection on the Unix systems but I suspect the Novell TSAs were a bit shag at the time). We updated "time to restore" estimations and moved on, after adding one or two other results of lessons learned. Do test your plans (this is not aimed at you personally zer00eyz - you probably know better than most). There are a lot of unknowns but the basic model of a real DR plan is pretty sound these days, if you can afford it or wing it in some way. An example: Another site, a suitable distance away. On that site there is enough infra to run the basics - wifi, a few ethernet ports, telephony etc. There should also be enough hypervisor and storage for that. Some backups are delivered there as well as on site. Hypervisor replicas are created from the backups (or directly) depending on RPO requirements and bandwidth available. The only thing that should be able to routinely access the backup files is the backup system (certainly not "Domain Admins" or other such nonsense". Ensure that what is written is verified. Now test it 8) .... regularly
      - zer00eyz 9y ago
        Ok now I have to share a story... The company in question had a rather large on site server room (raised floor, fire suppression) and a massive generator to deal with any power issues as well as redundant connectivity. This room was literally the backup incase their "real" data center went off line. The problem is that the room was "convenient" so there were plenty of things that lived ONLY there (mistake one) - When the substation for the office went, and the generator started everything looked fine. The problem was that no one had ever run the generator for that long... after a few hours it simply crapped out (over heated, problem two). A quick trip to home depot got them generators and extension cords that let them get the few boxes that were critical back up - however one box decided to not only fault, but to take it's data with it. This is when I got a rather frantic call "did I still have the code from the project I did?" - they offered to cut me a check for $2000 if I would go home right then and simply LOOK for it. Lucky for them I had it - and the continuity portion of the DR plan got revisited. In hind sight after I said I had the code, I probably could have asked them to put another zero on the end of the check and they would have done it just to be a functioning business come 6am.
        gerdesj 9y ago
        I didn't even have to show some leg to get you to recant the dit. Thank you - I'm happy to listen to (nearly) everything. "I probably could have asked them to put another zero" - ahem that's not the IT Consultant's Way exactly. We have far more polite ways of extracting loot. We are not lawyers and should have morals.
    - TeMPOraL 9y ago
      Reminds me of that The Expanse quote: "I have a file with 900 pages of analysis and contingency plans for war with Mars, including fourteen different scenarios about what to do if they develop an unexpected new technology. My file for what to do if an advanced alien species comes calling is three pages long, and it begins with 'Step 1: Find God'."
- campuscodi 9y ago
  Microsoft had 2-3 months, tops.
  - gerdesj 9y ago
    Source, proof of assertion?
    - bonzini 9y ago
      As far as I know he's right. The news was given first to Amazon and Microsoft sometime in August. Consider one month for testing and preparing for release, that gives three months to build a solution for all supported operating systems. Two months to do it for the most recent version and one month for backporting to the older ones sounds about right. Maybe a few weeks more, but that's it.
    - campuscodi 9y ago
      Intel's own press releases. I'm not gonna go digging into old links just because some random dude on the Internet can't use Google.
Aloha 9y ago
If the problem is complex enough, six months may be a short deadline.
- azinman2 9y ago
  Especially since it comes as a surprise, and there’s already an existing train on its way to the next station with its own timetable.
  - cakes 9y ago
    that surprise could include "management says we don't have to do anything :/" <5.99 months pass>, management: "we have to patch this and it needs to be done yesterday."
    - TeMPOraL 9y ago
      That would be extreme, but I can entirely imagine it taking many weeks for the true importance of this problem to correctly propagate across all management levels.
maltalex 9y ago
> they had 6 months. You say that like 6 months is automatically a lot of time. "She had 6 months to give birth". Yeah, only it takes 9, so 6 is short. Consider the scope and depth of the issue and the fact that they probably couldn't involve too many people on this effort.
- walterbell 9y ago
  Other operating system maintainers had only days/weeks in Jan 2018.
  - deleted 9y ago
    [deleted]
  - loufe 9y ago
    We're all throwing darts in the dark here with regards to the resource they gave the problem and its difficulty. I think the real takeaway is just that it could be something other than just stupidity through and through.
  - mannykannot 9y ago
    Unless you can be sure their response solved their problem without introducing others, it's not evidence of sufficient time.
- dmix 9y ago
  A single person with a fixed 9 month biological timeframe is the analogy you chose to compare 6 months of a billion dollar company's software development time by potentially hundreds of developers (for better or worse) but importantly for an extremely critical class of bugs and therefore that's "just how it is"? Come on, software is hard but when you fix a vulnerability and expose a far far worse one, and had months to plan, execute, and test it, then you are most certainly justified in being criticized. It's not like we're saying the code is shoddy and needs work, which is entirely excusable in a short timeframe. It's that they've left users far worse off in the end then from where they started.
  - 9y ago
yashap 9y ago
I wonder how long the actual devs fixing it had? From what I hear from friends who work there, Microsoft is a sprawling bureaucracy with many layers of management, where decisions are far from quick. I'd imagine that after Intel/whoever let Microsoft know about the exploits, it went through many levels of prioritization, negotiation about which team would work on it, not being brought into sprints because of other features already being worked on, etc. Most likely there were people with minimal knowledge of the relevant tech making all these prioritization decisions. Wouldn't shock me at all if there was very little actual dev work done for the first few months, and then it was all super rushed at the end. Quite possibly the devs with the required knowledge didn't even know this was in the pipeline for months. That's par for the course at every decently large company I've worked at (i.e. 100+ devs), and at a beast like Microsoft I imagine it'd be way worse.
- lmilcin 9y ago
  I remember that Microsoft was able to deliver critical fixes practically overnight. This assumed that once you see the problem the fix is pretty straightforward. Unfortunately Spectre and Meltdown aren't straightforward and go to the very heart of how the OS works. It's not at all easy to fix this when you have enormous amount of software working on top of it depending on every little quirk your solution provides.

waldbeere 9y ago

or maybe because M$ change name to hindusoft

gwbas1c 9y ago

These kinds of things should be part of an automated test suite. Specifically, the kind of tests that were written years ago. Honestly, Microsoft is really big into automated testing. I'm surprised this slipped through.

astrange 9y ago
I don't think there are any OS kernels that practice test-driven development - most of them don't even have code coverage working. It's also very hard to test for a problem you haven't thought of yet.

yuhong 9y ago

Yea, it is probably the biggest change to the Windows kernel in a security update.

Tobba_ 9y ago

This is what happens when you don't have a QA department.

efdee 9y ago
Are you implying that Microsoft doesn't do QA?
- Tobba_ 9y ago
  If they do, whatever issues they're occupied with finding would call for an exorcism.
bonzini 9y ago
This is something that you find through code review, not testing. Apart from regression testing, but that presupposes that you encountered the issue before.