6 ms·

LSE Halts

101 points by ashishb4u 16y ago

http://online.wsj.com/article/SB10001424052748704150604576165753761876730.html?mod=googlenews_wsj

famoreira 16y ago
Does anyone know if this halt is related to the recent switch over to the new Linux based platform? (http://news.ycombinator.com/item?id=686116 http://news.ycombinator.com/item?id=686116)
- drstrangevibes 16y ago
  it is a new platform, hard to tell but I wouldnt go so far as to say linux is to blame, its more likely to be exchange software, they've gone from native to fix 5, so its using a new protocol, new os, new software stack and most likely new hardware stack
  - tallanvor 16y ago
    I don't think famoreira was trying to blame Linux. --That's just the part of the new system that you can reference easily.
    - famoreira 16y ago
      Yes, I didn't mean to blame Linux. I was just referring to the system in the way it was made famous (replacing a Microsoft based system with a Linux one).
  - IDisposableHero 16y ago
    I don't think that "linux is to blame" or even that any choice of programming languages or other tools are to blame. It's a large system with extra-ordinarily tight performance requirements, and a lot of integration with external systems, and by nature hard to phase in gradually. That kind of system is very hard to get right first time regardless of your toolset. I'd be interested to hear more about thier methods to deal with these requirements and risks, less about thier platform choice. IIRC, both this and the previous failed system were outsourced, which doesn't exactly fill me with confidence.
- known 16y ago
  I believe this is due to a glitch in the Application and not in Linux per se.
ashishb4u 16y ago
Nothing official yet. (http://www.ft.com/cms/s/0/9bf8033a-40bb-11e0-9a37-00144feabdc0.html?ftcamp=rss#axzz1ExjV27Jt http://www.ft.com/cms/s/0/9bf8033a-40bb-11e0-9a37-00144feabd...)
drstrangevibes 16y ago
this is the official place for updates, although its not very responsive at the moment,probably due to load http://liveservice.londonstockexchangegroup.com/en/incidents/active/lse http://liveservice.londonstockexchangegroup.com/en/incidents...
- pnathan 16y ago
  It's been archived - here's the link to this incident in the archive: http://liveservice.londonstockexchangegroup.com/en/incidents/archived/85-friday-25022011-075443 http://liveservice.londonstockexchangegroup.com/en/incidents...
JonnieCache 16y ago
I wonder if it's HFT related? A group of rogue algorithms gone exponential? There are certainly people worried about that...
- deleted 16y ago
  [deleted]
tpatke 16y ago
Where is the fail over? If ever a system needed 99.9999...
- tallanvor 16y ago
  Unfortunately sometimes fail over isn't a solution. Until they've at least narrowed down the problem, a fail over system may make things worse. Personally, I'm curious about the testing that was done - given the complaints and problems that have been reported so far, did they even try to run the new and old systems simultaneously for any period of time to verify that the various components could all keep up properly? Or at the very least, did they try to test the system at full load for a decent period of time?
- JonnieCache 16y ago
  Failover in this scenario would require instant, 100% replication of a near-zero latency system. I'm no expert, but I'm fairly sure having trades 'disappear' or any other loss of consistency would be worse than downtime in this case.
mtkd 16y ago
"And as a final note, solely for the seriously nerdy techies, there's been some talk about the failure being due to F# error." http://ftalphaville.ft.com/marketslive/ http://ftalphaville.ft.com/marketslive/
- timrobinson 16y ago
  I've no idea where they got this from, although I mentioned it on Twitter as a joke: https://twitter.com/tim_g_robinson/status/41086872876892160 https://twitter.com/tim_g_robinson/status/41086872876892160
- drstrangevibes 16y ago
  F# on linux?
  - timrobinson 16y ago
    http://www.mono-project.com/Release_Notes_Mono_2.10#Languages http://www.mono-project.com/Release_Notes_Mono_2.10#Language...
  - viraptor 16y ago
    Via mono. If that's true... that would be an interesting news for the project (not because of the bug itself of course...).
    - timrobinson 16y ago
      If the LSE were using F# we'd have heard a lot more about it already. For instance, from Microsoft.
      - viraptor 16y ago
        Maybe not if they were using F# on mono. Their PR would have to work very hard to put this in a way that doesn't say "LSE is using our language on a foreign platform using a completely alternative implementation, which has nothing to do with us apart some ECMA specs - and they're still successful without MS involvement. Yes, we just confirmed that you can trust your highly critical .NET environment to the alternative versions of runtime."
alecco 16y ago
What if all these systems were open sourced? Including all the load tests.
- viraptor 16y ago
  At this size, some obfuscation might be good. If you find a bug and crash some browser, it's not a big deal. If you find some problem in LSE that can be triggered from outside, you're dealing with realworld $M transactions. Also, unless you have a huge farm of servers it would be useless to you - and basically anyone really.
- rbanffy 16y ago
  > What if all these systems were open sourced? LSE considers its software to be a competitive advantage.
  - drstrangevibes 16y ago
    .....er sure
    - rbanffy 16y ago
      When they make it work, it could be ;-)
ig1 16y ago
The problem is that the LSE can't hire top tier talent. There are plenty of people who are experts in building hugely concurrent and fail-safe exchange software, and they pretty much all work for investment banks. The LSE can't afford to compete with $1000/day salaries so they end up hiring second-tier developers. And unfortunately second-tier developers are a lot lot worse than first-tier developers.
- laktek 16y ago
  I don't think it's about talent, but more of a problem of proper planning and execution. It seems things had been rushed to meet the tight demands, causing the chaos. When LSE acquired MilleniumIT, to redo the platform, they claimed to have developed world fastest trading platform[1] and had enough talent under their belt developing trading systems around the globe for last decade. [1] http://www.wallstreetandtech.com/exchanges/showArticle.jhtml?articleID=220000655 http://www.wallstreetandtech.com/exchanges/showArticle.jhtml...
  - ig1 16y ago
    None of the exchanges have the required talent. Being the best exchange technology company is like being the best team in a minor league. If you're good at extreme low latency stuff you're going to be working for a bank. At a bank being great at low-latency can mean tens of millions of dollars of profit for your trading desk and a huge bonus for your personally. At an exchange being great a low-latency means ... well not much, your exchange gets a bit more flow and you get a pat on the back.
    - deleted 16y ago
      [deleted]
    - chollida1 16y ago
      > At an exchange being great a low-latency means ... well not much, your exchange gets a bit more flow and you get a pat on the back. Exchanges pay out bigger bonus' than you'd imagine. 7 figures for a programmer is high but no where out of the ordinary. Says a guy who has worked for an exchange.
alecco 16y ago
Sri Lanka? http://uk.reuters.com/article/2009/09/16/uk-lse-it-sb-idUKTRE58F4TC20090916 http://uk.reuters.com/article/2009/09/16/uk-lse-it-sb-idUKTR... (via /B @ftalphaville comments)
timrobinson 16y ago
"Continuous trading has now resumed in all order driven trading services."

101 points by ashishb4u 16y ago

http://online.wsj.com/article/SB10001424052748704150604576165753761876730.html?mod=googlenews_wsj

Does anyone know if this halt is related to the recent switch over to the new Linux based platform? (http://news.ycombinator.com/item?id=686116 http://news.ycombinator.com/item?id=686116)

drstrangevibes 16y ago
it is a new platform, hard to tell but I wouldnt go so far as to say linux is to blame, its more likely to be exchange software, they've gone from native to fix 5, so its using a new protocol, new os, new software stack and most likely new hardware stack
- tallanvor 16y ago
  I don't think famoreira was trying to blame Linux. --That's just the part of the new system that you can reference easily.
  - famoreira 16y ago
    Yes, I didn't mean to blame Linux. I was just referring to the system in the way it was made famous (replacing a Microsoft based system with a Linux one).
- IDisposableHero 16y ago
  I don't think that "linux is to blame" or even that any choice of programming languages or other tools are to blame. It's a large system with extra-ordinarily tight performance requirements, and a lot of integration with external systems, and by nature hard to phase in gradually. That kind of system is very hard to get right first time regardless of your toolset. I'd be interested to hear more about thier methods to deal with these requirements and risks, less about thier platform choice. IIRC, both this and the previous failed system were outsourced, which doesn't exactly fill me with confidence.
known 16y ago
I believe this is due to a glitch in the Application and not in Linux per se.

ashishb4u 16y ago

Nothing official yet. (http://www.ft.com/cms/s/0/9bf8033a-40bb-11e0-9a37-00144feabdc0.html?ftcamp=rss#axzz1ExjV27Jt http://www.ft.com/cms/s/0/9bf8033a-40bb-11e0-9a37-00144feabd...)

drstrangevibes 16y ago

this is the official place for updates, although its not very responsive at the moment,probably due to load http://liveservice.londonstockexchangegroup.com/en/incidents/active/lse http://liveservice.londonstockexchangegroup.com/en/incidents...

pnathan 16y ago
It's been archived - here's the link to this incident in the archive: http://liveservice.londonstockexchangegroup.com/en/incidents/archived/85-friday-25022011-075443 http://liveservice.londonstockexchangegroup.com/en/incidents...

JonnieCache 16y ago

I wonder if it's HFT related? A group of rogue algorithms gone exponential? There are certainly people worried about that...

deleted 16y ago
[deleted]

tpatke 16y ago

Where is the fail over? If ever a system needed 99.9999...

tallanvor 16y ago
Unfortunately sometimes fail over isn't a solution. Until they've at least narrowed down the problem, a fail over system may make things worse. Personally, I'm curious about the testing that was done - given the complaints and problems that have been reported so far, did they even try to run the new and old systems simultaneously for any period of time to verify that the various components could all keep up properly? Or at the very least, did they try to test the system at full load for a decent period of time?
JonnieCache 16y ago
Failover in this scenario would require instant, 100% replication of a near-zero latency system. I'm no expert, but I'm fairly sure having trades 'disappear' or any other loss of consistency would be worse than downtime in this case.

mtkd 16y ago

"And as a final note, solely for the seriously nerdy techies, there's been some talk about the failure being due to F# error." http://ftalphaville.ft.com/marketslive/ http://ftalphaville.ft.com/marketslive/

timrobinson 16y ago
I've no idea where they got this from, although I mentioned it on Twitter as a joke: https://twitter.com/tim_g_robinson/status/41086872876892160 https://twitter.com/tim_g_robinson/status/41086872876892160
drstrangevibes 16y ago
F# on linux?
- timrobinson 16y ago
  http://www.mono-project.com/Release_Notes_Mono_2.10#Languages http://www.mono-project.com/Release_Notes_Mono_2.10#Language...
- viraptor 16y ago
  Via mono. If that's true... that would be an interesting news for the project (not because of the bug itself of course...).
  - timrobinson 16y ago
    If the LSE were using F# we'd have heard a lot more about it already. For instance, from Microsoft.
    - viraptor 16y ago
      Maybe not if they were using F# on mono. Their PR would have to work very hard to put this in a way that doesn't say "LSE is using our language on a foreign platform using a completely alternative implementation, which has nothing to do with us apart some ECMA specs - and they're still successful without MS involvement. Yes, we just confirmed that you can trust your highly critical .NET environment to the alternative versions of runtime."

alecco 16y ago

What if all these systems were open sourced? Including all the load tests.

viraptor 16y ago
At this size, some obfuscation might be good. If you find a bug and crash some browser, it's not a big deal. If you find some problem in LSE that can be triggered from outside, you're dealing with realworld $M transactions. Also, unless you have a huge farm of servers it would be useless to you - and basically anyone really.
rbanffy 16y ago
> What if all these systems were open sourced? LSE considers its software to be a competitive advantage.
- drstrangevibes 16y ago
  .....er sure
  - rbanffy 16y ago
    When they make it work, it could be ;-)

ig1 16y ago

The problem is that the LSE can't hire top tier talent. There are plenty of people who are experts in building hugely concurrent and fail-safe exchange software, and they pretty much all work for investment banks. The LSE can't afford to compete with $1000/day salaries so they end up hiring second-tier developers. And unfortunately second-tier developers are a lot lot worse than first-tier developers.

laktek 16y ago
I don't think it's about talent, but more of a problem of proper planning and execution. It seems things had been rushed to meet the tight demands, causing the chaos. When LSE acquired MilleniumIT, to redo the platform, they claimed to have developed world fastest trading platform[1] and had enough talent under their belt developing trading systems around the globe for last decade. [1] http://www.wallstreetandtech.com/exchanges/showArticle.jhtml?articleID=220000655 http://www.wallstreetandtech.com/exchanges/showArticle.jhtml...
- ig1 16y ago
  None of the exchanges have the required talent. Being the best exchange technology company is like being the best team in a minor league. If you're good at extreme low latency stuff you're going to be working for a bank. At a bank being great at low-latency can mean tens of millions of dollars of profit for your trading desk and a huge bonus for your personally. At an exchange being great a low-latency means ... well not much, your exchange gets a bit more flow and you get a pat on the back.
  - deleted 16y ago
    [deleted]
  - chollida1 16y ago
    > At an exchange being great a low-latency means ... well not much, your exchange gets a bit more flow and you get a pat on the back. Exchanges pay out bigger bonus' than you'd imagine. 7 figures for a programmer is high but no where out of the ordinary. Says a guy who has worked for an exchange.

alecco 16y ago

Sri Lanka? http://uk.reuters.com/article/2009/09/16/uk-lse-it-sb-idUKTRE58F4TC20090916 http://uk.reuters.com/article/2009/09/16/uk-lse-it-sb-idUKTR... (via /B @ftalphaville comments)

timrobinson 16y ago

"Continuous trading has now resumed in all order driven trading services."