7 ms·

There are many interesting lessons from this tale, but the conclusion that automated deployment would have saved the day seems a bit of a jump. Automation does

by personZ 13y ago

There are many interesting lessons from this tale, but the conclusion that automated deployment would have saved the day seems a bit of a jump. Automation does not protect you from either automated devastation, gaps, or human errors. Your automation tools, just as with written instructions, require configuration -- a list of servers, for instance. Automation can be bullet-proof when it's a continuous deployment situation, but is less reliable when you do infrequent deployments, as such a financial firm does. I say this having been in a firm where we moved from "a list of deployment steps" to "fully automated" for our quarterly builds, and the result was much, much, much worse than it was before. We could certainly have resolved this (for instance having a perfect replica of production), but the amount of delta and work and testing we did on our deployment process vastly, and by several magnitudes, exceeded our manual process. An observer did not validate the deployment (which should be the case whether automated or not for such a deploy). They ignored critical warning messages sent by the system pre-trading (the system was warning them that it was a SNAFU situation). Systems in a cluster didn't verify versions with each other. Configuration did not demand a version. Most importantly for a system of this sort, they didn't have a trade gateway that they could easily see what the system was doing, and gate abnormal behaviors quickly and easily (such a system should be as simple as possible, the premise being that it's an intermediate step between the decision/action systems and the market. The principal is exactly the same as sending a mass customer mailing to a holding pen for validation to ensure that your macros are correct, people aren't multi-sent, to do throttling, etc).

MartinCron 13y ago
Automation can be bullet-proof when it's a continuous deployment situation, but is less reliable when you do infrequent deployments If your deployment pipeline is fully automated, why aren't you making lots of little deployments? The safest change to make is the smallest change possible, after all.
- btgeekboy 13y ago
  Our deployment can be done in one command in about 15 minutes, start to finish. Yet, we only deploy once every few weeks. Why? There's more to releasing code than simply pushing it live: QA has to review it, documentation has to be written/updated, marketing may need to write a press release, sales and customers may need to be notified, etc.
  - wpietri 13y ago
    "Has to" is pretty strong. You folks have chosen to work that way, but other people do it differently.
    - akerl_ 13y ago
      I hope you're trolling, but rolling out high-impact code as is described in this article and comment thread without serious review and QA is akin to business suicide, as demonstrated by the article.
      - hft_throwaway 13y ago
        The code was QAed, but they didn't test old and new versions against each other. Version A could accept a flag and run obsolete logic that would lose control of its orders but never sent it, so this problem never happened. Version B sent this flag and the receiver would send RPI orders with it. Put a Version B sender and a Version A receiver together and you end up with a disaster. From a systems perspective, my takeaways on this are: -Don't re-use a message for a semantically different purpose in a distributed system where you're running different software versions (even in cases where you don't plan to, really, since you may roll back or end up running the wrong code by mistake) -Version your messages so anything that changes their meaning can only be accepted by a receiver that follows that protocol -QA old and new builds against one another If you really want to look at the root cause of this, it's cultural. Trading desks don't want to spend development time on things that don't generate PnL. Traders want to try lots of ideas so many features are built that don't get used. Code cleanup gets put on the back burner. Developers do sketchy stuff like re-purposing a message field because it's annoying or time-consuming to deploy a new format. If traders aren't developers themselves, they may underestimate the risk of pressuring operations & devs to work more quickly. Things like this are probably the biggest risk faced by automated traders, and the good shops take it very seriously. I've never been scared of any loss due to poor trading, but losses due to software errors can be astonishing and happen faster than you can stop them.
      - wpietri 13y ago
        I'm not trolling at all. Let me take his points: > QA has to review it QA review is one approach to quality, but it's far from the only one. In Lean Manufacturing, heavy QA is seen as wasteful, covering up for upstream problems. Their approach is to eliminate the root problems. That let Toyota kick the asses of the US car manufacturers in the 80s. >documentation has to be written/updated This to me smells of a phasist approach, with disconnected groups of specialists. Some people work with cross-functional teams, so that everything important (e.g., both code and user documentation) is updated at the same time. >marketing may need to write a press release, sales and customers may need to be notified This is confusing releasing code with making features active for most users. You can do them together, but it's not the only way. Feature flags and gradual rollouts are two other options. More broadly, in this case rolling out the code with serious review and QA was also business suicide. The "do more QA" approach is trying to decrease MTBF, with the goal of nothing bad happening ever. But there's another approach: to minimize MTTR (or, more accurately, to minimize impact of issues). Shops like that are much better at recovering from issues. Rather than trying to pretend they will never make mistakes, they assume they will and work to be ready for it.
- teacup50 13y ago
  No, the safest change to make is one that has been reviewed and tested thoroughly and found to be correct. Plenty of harm can be found in even small changes; the reality is that continuous deployment only works for companies that can afford to regularly push minutely broken software to their customers.
  - wpietri 13y ago
    Wealthfront has been doing continuous deployment for years, and they're financially liable for any bugs. You might want to reconsider your opinion.
    - teacup50 13y ago
      No, they're financially liable for financially impacting bugs. Broken web UI annoys your customers, but is unlikely to bankrupt you.
      - wpietri 13y ago
        I'm not getting your point. Is there some software that doesn't have the occasional minor UI bug? My point is that Wealthfront is doing Continuous Deployment in an environment where even small changes can cause substantial harm, and they seem to be doing fine at it.
        teacup50 13y ago
        Yes, there is software that is designed with the goal of never wasting the user's time or goodwill with buggy or ill conceived UX or implementation. As for "doing fine at it", that they exist doesn't prove that, or even define what "fine" is.
        wpietri 13y ago
        I'm still not getting your point. So what if software is designed with a goal of being magical? The question is how good the real-world results are. From hearing them talk at events and reading their blog, they seem to be doing fine. Surviving for 6 years and their recent C round from a number of smart people also suggests they are doing fine. If you have some substantial indication that they aren't, I look forward to reading it.
vacri 13y ago
The bit that surprised me the most was the lack of killswitch (or a halt or a pause); that a human supervisor couldn't invoke a "holy shit!" button.
- harmegido 13y ago
  This probably surprised regulators and the exchanges too, as it is a requirement for automated trading.
- yummyfajitas 13y ago
  A "killswitch" is not a trivial thing to build. If you simply run `kill -9 pid`, you might be holding a large position, or worse, you might be holding some naked shorts. (In fact, you almost certainly are.) This is risky. It can result in failures to deliver, make you vulnerable to large market movements, etc. Another form of "killswitch" is to not open any new positions, but still attempt to close out your old positions with varying degrees of aggressiveness. But if your system is wildly broken, this might not be doing what you think it's doing. As I understand it, this happened to Knight.
  - ramchip 13y ago
    I think that's a strange way to approach a killswitch. The point of a killswitch is to stop everything NOW, so the system can be checked. After the problem is understood, then it's safe to restart strategies gradually and unwind positions. A system shouldn't attempt to send more orders when it's in an unpredictable state. I believe many regulators require such a switch.
  - hft_throwaway 13y ago
    You know that saying about finding yourself in a hole? A "close out my risk" button is fine for situations like losing money or whatever, but if you have no clue what your orders, trades and risk even are, the only sensible thing to do is stop making it worse.
- theorique 13y ago
  A kill switch for a system that holds complex and rapidly changing state (open positions, market data) is quite different from, and much more complicated than, a kill switch for a system that can be simply halted at a given instant.
rossjudson 13y ago
This has nothing to do with automation of deployments. Any part of an automated deployment can fail. At scale a single failure cannot be allowed to halt the deployment, either. This is an architectural mistake. Distributed systems must always be able to operate in an environment with a variety of versions, without ill effects. They repurposed a flag and then failed to test the mixed environment. Hindsight is 20/20, of course.
carlsborg 13y ago
Posted a note about the human aspect of uptime recently: http://zvzzt.wordpress.com/2012/08/16/a-note-on-uptime http://zvzzt.wordpress.com/2012/08/16/a-note-on-uptime

by personZ 13y ago

MartinCron 13y ago

Automation can be bullet-proof when it's a continuous deployment situation, but is less reliable when you do infrequent deployments If your deployment pipeline is fully automated, why aren't you making lots of little deployments? The safest change to make is the smallest change possible, after all.

btgeekboy 13y ago
Our deployment can be done in one command in about 15 minutes, start to finish. Yet, we only deploy once every few weeks. Why? There's more to releasing code than simply pushing it live: QA has to review it, documentation has to be written/updated, marketing may need to write a press release, sales and customers may need to be notified, etc.
- wpietri 13y ago
  "Has to" is pretty strong. You folks have chosen to work that way, but other people do it differently.
  - akerl_ 13y ago
    I hope you're trolling, but rolling out high-impact code as is described in this article and comment thread without serious review and QA is akin to business suicide, as demonstrated by the article.
    - hft_throwaway 13y ago
      The code was QAed, but they didn't test old and new versions against each other. Version A could accept a flag and run obsolete logic that would lose control of its orders but never sent it, so this problem never happened. Version B sent this flag and the receiver would send RPI orders with it. Put a Version B sender and a Version A receiver together and you end up with a disaster. From a systems perspective, my takeaways on this are: -Don't re-use a message for a semantically different purpose in a distributed system where you're running different software versions (even in cases where you don't plan to, really, since you may roll back or end up running the wrong code by mistake) -Version your messages so anything that changes their meaning can only be accepted by a receiver that follows that protocol -QA old and new builds against one another If you really want to look at the root cause of this, it's cultural. Trading desks don't want to spend development time on things that don't generate PnL. Traders want to try lots of ideas so many features are built that don't get used. Code cleanup gets put on the back burner. Developers do sketchy stuff like re-purposing a message field because it's annoying or time-consuming to deploy a new format. If traders aren't developers themselves, they may underestimate the risk of pressuring operations & devs to work more quickly. Things like this are probably the biggest risk faced by automated traders, and the good shops take it very seriously. I've never been scared of any loss due to poor trading, but losses due to software errors can be astonishing and happen faster than you can stop them.
    - wpietri 13y ago
      I'm not trolling at all. Let me take his points: > QA has to review it QA review is one approach to quality, but it's far from the only one. In Lean Manufacturing, heavy QA is seen as wasteful, covering up for upstream problems. Their approach is to eliminate the root problems. That let Toyota kick the asses of the US car manufacturers in the 80s. >documentation has to be written/updated This to me smells of a phasist approach, with disconnected groups of specialists. Some people work with cross-functional teams, so that everything important (e.g., both code and user documentation) is updated at the same time. >marketing may need to write a press release, sales and customers may need to be notified This is confusing releasing code with making features active for most users. You can do them together, but it's not the only way. Feature flags and gradual rollouts are two other options. More broadly, in this case rolling out the code with serious review and QA was also business suicide. The "do more QA" approach is trying to decrease MTBF, with the goal of nothing bad happening ever. But there's another approach: to minimize MTTR (or, more accurately, to minimize impact of issues). Shops like that are much better at recovering from issues. Rather than trying to pretend they will never make mistakes, they assume they will and work to be ready for it.
teacup50 13y ago
No, the safest change to make is one that has been reviewed and tested thoroughly and found to be correct. Plenty of harm can be found in even small changes; the reality is that continuous deployment only works for companies that can afford to regularly push minutely broken software to their customers.
- wpietri 13y ago
  Wealthfront has been doing continuous deployment for years, and they're financially liable for any bugs. You might want to reconsider your opinion.
  - teacup50 13y ago
    No, they're financially liable for financially impacting bugs. Broken web UI annoys your customers, but is unlikely to bankrupt you.
    - wpietri 13y ago
      I'm not getting your point. Is there some software that doesn't have the occasional minor UI bug? My point is that Wealthfront is doing Continuous Deployment in an environment where even small changes can cause substantial harm, and they seem to be doing fine at it.
      - teacup50 13y ago
        Yes, there is software that is designed with the goal of never wasting the user's time or goodwill with buggy or ill conceived UX or implementation. As for "doing fine at it", that they exist doesn't prove that, or even define what "fine" is.
        wpietri 13y ago
        I'm still not getting your point. So what if software is designed with a goal of being magical? The question is how good the real-world results are. From hearing them talk at events and reading their blog, they seem to be doing fine. Surviving for 6 years and their recent C round from a number of smart people also suggests they are doing fine. If you have some substantial indication that they aren't, I look forward to reading it.

vacri 13y ago

The bit that surprised me the most was the lack of killswitch (or a halt or a pause); that a human supervisor couldn't invoke a "holy shit!" button.

harmegido 13y ago
This probably surprised regulators and the exchanges too, as it is a requirement for automated trading.
yummyfajitas 13y ago
A "killswitch" is not a trivial thing to build. If you simply run `kill -9 pid`, you might be holding a large position, or worse, you might be holding some naked shorts. (In fact, you almost certainly are.) This is risky. It can result in failures to deliver, make you vulnerable to large market movements, etc. Another form of "killswitch" is to not open any new positions, but still attempt to close out your old positions with varying degrees of aggressiveness. But if your system is wildly broken, this might not be doing what you think it's doing. As I understand it, this happened to Knight.
- ramchip 13y ago
  I think that's a strange way to approach a killswitch. The point of a killswitch is to stop everything NOW, so the system can be checked. After the problem is understood, then it's safe to restart strategies gradually and unwind positions. A system shouldn't attempt to send more orders when it's in an unpredictable state. I believe many regulators require such a switch.
- hft_throwaway 13y ago
  You know that saying about finding yourself in a hole? A "close out my risk" button is fine for situations like losing money or whatever, but if you have no clue what your orders, trades and risk even are, the only sensible thing to do is stop making it worse.
theorique 13y ago
A kill switch for a system that holds complex and rapidly changing state (open positions, market data) is quite different from, and much more complicated than, a kill switch for a system that can be simply halted at a given instant.

rossjudson 13y ago

This has nothing to do with automation of deployments. Any part of an automated deployment can fail. At scale a single failure cannot be allowed to halt the deployment, either. This is an architectural mistake. Distributed systems must always be able to operate in an environment with a variety of versions, without ill effects. They repurposed a flag and then failed to test the mixed environment. Hindsight is 20/20, of course.

carlsborg 13y ago

Posted a note about the human aspect of uptime recently: http://zvzzt.wordpress.com/2012/08/16/a-note-on-uptime http://zvzzt.wordpress.com/2012/08/16/a-note-on-uptime