3 ms·

I gotta agree with HelloNurse here. When that kind of money is on the line you set up a production version that ingests real data at real speed, and produces th

I gotta agree with HelloNurse here. When that kind of money is on the line you set up a production version that ingests real data at real speed, and produces the same output / response but doesn't actually spend / affect any money. You watch it for at least a week and see what happens. I'm sorry to say it, but the loss of money here was not the JVMs fault. It was putting code with insufficient load testing in a production environment and crossing your fingers despite the fact that serious money was on the line.

anonymoushn 6y ago
A complete-fidelity production-like environment would be a couple orders of magnitude more expensive than the 1-day loss, so I disagree.
- HelloNurse 6y ago
  You don't need another complete set of servers: the new production servers, before beginning actual service, are available for brutal stress tests and all sorts of other experiments while the old servers running the old software are still in use and undisturbed. If the system is completely new and there is no old software on old servers, it's even easier. Considering that software mishaps in securities trading can annihilate a company of any size, not only cost you as much as a bunch of servers, lack of testing appears very reckless.
  - anonymoushn 6y ago
    > You don't need another complete set of servers: the new production servers, before beginning actual service, are available for brutal stress tests and all sorts of other experiments while the old servers running the old software are still in use and undisturbed. If the system is completely new and there is no old software on old servers, it's even easier. We did not generally physically swap out servers each time we changed a line of code. Rather, we had two identical sets of servers, one for production and one for test. For test, we had some additional machines to run matching engines. If we wanted to make the test environment more closely match an actual venue, we would need many more machines that run matching engines. That still might not be sufficient to produce identical output timings in prod and test, though, because venues generally do not publish the sequence of all incoming ticks they receive during a day, the source code of their matching engine, what hardware they are using, their kernel version, and so on. > Considering that software mishaps in securities trading can annihilate a company of any size, not only cost you as much as a bunch of servers, lack of testing appears very reckless. It's strange that you replied to me suggesting we had no production-like test environment ~10 hours after I replied to you saying that this code did fine in our production-like test environment.

anonymoushn 6y ago

A complete-fidelity production-like environment would be a couple orders of magnitude more expensive than the 1-day loss, so I disagree.

HelloNurse 6y ago
You don't need another complete set of servers: the new production servers, before beginning actual service, are available for brutal stress tests and all sorts of other experiments while the old servers running the old software are still in use and undisturbed. If the system is completely new and there is no old software on old servers, it's even easier. Considering that software mishaps in securities trading can annihilate a company of any size, not only cost you as much as a bunch of servers, lack of testing appears very reckless.
- anonymoushn 6y ago
  > You don't need another complete set of servers: the new production servers, before beginning actual service, are available for brutal stress tests and all sorts of other experiments while the old servers running the old software are still in use and undisturbed. If the system is completely new and there is no old software on old servers, it's even easier. We did not generally physically swap out servers each time we changed a line of code. Rather, we had two identical sets of servers, one for production and one for test. For test, we had some additional machines to run matching engines. If we wanted to make the test environment more closely match an actual venue, we would need many more machines that run matching engines. That still might not be sufficient to produce identical output timings in prod and test, though, because venues generally do not publish the sequence of all incoming ticks they receive during a day, the source code of their matching engine, what hardware they are using, their kernel version, and so on. > Considering that software mishaps in securities trading can annihilate a company of any size, not only cost you as much as a bunch of servers, lack of testing appears very reckless. It's strange that you replied to me suggesting we had no production-like test environment ~10 hours after I replied to you saying that this code did fine in our production-like test environment.