9 ms·

Bloomberg Terminals Suffer Widespread Failures

132 points by svtrent 12y ago

bbgtoreuters 12y ago
Decades of crusty code - C++ wrapped around C wrapped around Fortran, home brewed versions of the STL, nih syndrome across the tooling. Actually surprised outages not more common. Still, given their interview process is basically parlour trick questions about pointers and char arrays, it is quite telling!
- bachback 12y ago
  Bloomberg started out as a network system independent of the Internet. Michael Bloomberg was a network engineer. I'm not sure they ever really switched to proper TCP/IP. This company sure looks like a dinosaur these days. However its hard to imagine a company being able to replace even a chunk of their infrastructure. They are tightly integrated into all institutions and there is no other service which offers a complete coverage of all the world's financial assets.
  - andy_ppp 12y ago
    Yes I think to replace them someone needs to create a marketplace for different aspects of Bloomberg and you buy access to specific services and data as you need it. Would be an amazing business that, supposing you could get buy in from a load of other firms.
    - whatok 12y ago
      Really don't see an ala carte solution being worth it for most firms unless they're really trying to cut down on cost. Integrating and maintaining separate services alone is going to require additional developers. Cost associated with that coupled with quality of service and reliability (outside of this situation) of Bloomberg's current offering makes it a very tough sale. Also, guys who have been using terminals for years are really not going to want to switch.
      - andy_ppp 12y ago
        I'm more thinking "if someone was to compete with Bloomberg how would they even come close to what they offer" may not be practical in reality!
    - bvm 12y ago
      interesting view - what makes Bloomberg brilliant (aside from the support) is that i have, for example, an array of my portfolios that i can access from every function. So if you could create a service that took all of the information that was relevant to you (portfolios, news feeds, models etc) and normalize it to make it interoperable with a whole host of 3rd party systems, you could be on to something!
  - apaprocki 12y ago
    Yes, we use proper TCP/IP :)
    - tacostakohashi 12y ago
      ...among other things.
    - kokey 12y ago
      The issue was probably something like a broadcast storm because of defective fault detection on some redundant network kit. That won't stop people blaming it on the fact that the stack isn't built on the latest hipster frameworks.
      - noir_lord 12y ago
        While I see your point do we have to throw the word "hipster" at everything new?. I'm not a huge fan of Javascript and it's proliferation of frameworks but some of that stuff is objectively both good and useful.
        PhantomGremlin 12y ago
        It's human nature to want to label things. IIRC a while ago I was watching one of those "house hunter" type shows and a husband on there was really annoying. I found a discussion online and people were calling him a "hipster douche". So, perhaps hipster by itself isn't so bad. :) Back a long time ago the applicable pejorative was "yuppie". I haven't heard that one recently. I suspect that many people reading here would have to look it up in Wikipedia. Edit: BTW at the time I was partial to calling people "yuppie scum". Sigh. It's so hard to keep up with the changing putdowns! :)
        santaclaus 12y ago
        Using the term 'hipster' to refer to the newest and shiniest web framework always felt like a bit of a misnomer to me. A solid portion of hipsterdom is rooted in recycling of the obscure and of the vintage (vinyl, the popularity of the lumber sexual look, etc). It seems like the 'hipster' thing to do would be to write a web framework in PL/I or cobol or something.
- 7sigma 12y ago
  yep, lets not forget 700mb + executables...
  - andy_ppp 12y ago
    That equate from what I hear to 128gb+ memory usage!
  - tacostakohashi 12y ago
    Only 700mb? Maybe back in the perkin elmer days, these days they're up to 4gb, constrained only by the various linkers, file size and address spaces available...
    - bonaldi 12y ago
      Server-side is very big - in fact, a long time ago the part called simply "the Big" got too big and now there are multiple Bigs.
      - mahyarm 12y ago
        And this is one reason why I don't like C++. I only hear about executables getting too big in C++ projects.
      - psawaya 12y ago
        When I was there (2008, intern), they were in the process of moving server code from Bigs into separate services (BAS's, if I recall).
- lectrick 12y ago
  I swear I should start a consultancy in the control of software complexity because I am seeing this sort of thing everywhere, and the downages and security issues will only continue (or get worse, as we continue to rely more on software)
- ketralnis 12y ago
  It's crusty old code because it works and it's tried and true. If you have a backend that works, why mess with it? Sure, C++/C/Fortran/some home grown template library is what it's built on. What would they gain by rewriting it some node framework of the week or whatever other shiny distraction? If they did do that and that caused downtime, there would be the same naysayers here complaining about that instead.
anonu 12y ago
Is this the first crack in the Hoover Dam that is Bloomberg Terminals? Probably not - but the widespread reliance on the $10k+ per year terminal in the financial services industry means that this thing is ripe for disruption. The key isn't to build a terminal replacement though. fintech startups need to chip away at little bits and pieces - and need to inter-operate as much as possible. Only then will the bbg dominance be reined in.
- anmilo 12y ago
  one of the reasons i'm generally Ok with Slack's valuation is that there is a ton of room for a messaging service to offer a service that is secure and auditable at the level that the SEC/FINRA and Banks require. Like the article mentions, if you talk to the people using BBG, it's really the chat functionality that provides the main value for literally everyone, at least in my experience (former front office dev)
  - Karunamon 12y ago
    I'm unfamiliar with these terminals (have kind of always wanted to play with one), but at first blush, a "secure and auditable" messaging system does not seem that difficult to build. Is it a legitimate compliance or practical implementation problem, or is it just the network effect?
    - whatok 12y ago
      Network effect.
      - schintan 12y ago
        Its not just the network effect..Bloomberg's messaging system is tightly integrated with the other functionality of the terminal..for example one can execute a trade via a message or get live streaming quotes.
    - easytiger 12y ago
      its also easy to keep it secure when it doesnt use the internet
chollida1 12y ago
The outage isn't that bad. I've talked to 5 other firms in Canada this morning and haven't heard of a single one that is down. Bloomberg's tickering plant is still working and their emsx(trading system) is up and running, albeit a bit slow right now. I think it was on a stackoverflow podcast that Joel or Jeff said that it makes you wonder about people who complain about down time if they've ever built something of their own. Google's been down, ditto for facebook, twitter and Azure. If you build a big service, something will eventually fail, Bloomberg is no different, if anything they are much better than Reuters or CapIQ for availability. I saw this quote from another comment here: > but the widespread reliance on the $10k+ per year terminal in the financial services industry means that this thing is ripe for disruption. I completely disagree, Bloomberg isn't some slow lethargic big company. They iterate relentlessly, their help desk is the best I've ever had the pleasure of dealing with. If I've got a problem with the trading system, their first level help will get me to a technical specialist in that area in under 30 seconds. If they can't solve it then I'll talk to a developer in 30 minutes. Bloomberg is the leader in teh financial space, not only because their tech is the best in teh business, but because they listen to their customer better than their competitors, they release more frequently, and when they want to enter an area( say risk management) they go out and buy someone who is already a leader in that field. In short they never stop running, if anything they are increasing the distance between them and their "peers". As one example, they just posted a message to every Bloomberg terminal user telling us that the issue was an internal networking issue that has been solved. EDIT to answer this question: > How do they protect the ticker tape thing from attacks, like say, a DDOS? 2 things: 1) This question really doesn't make sense. How do you DDOS a one way line? They send you quotes for the tickers that you request. You can't really DDOS them, unless you do something like request tickers over and over again in a tight loop, in which case their api will rate limit you. 2) its a closed system. Only terminal users can access it and only through the bloomberg terminal or their api, so they control all the access points.
- brador 12y ago
  How do they protect the ticker tape thing from attacks, like say, a DDOS?
  - 12y ago
akramhussein 12y ago
Random anecdote: Was told by a trader a few years back that one trading desk had both Bloomberg and Thomson Reuters systems next to one another and once they were able to capitalise on Bloomberg being a few seconds faster to make a net gain.
- RayVR 12y ago
  This is really not possible. If there's any truth to this story at all you have forgotten the real version.
- zhte415 12y ago
  Having a Reuters and Bloomberg terminal on the desk next to each other is commonplace, I did have, plus a Datastream. As for a few seconds delay, that's probably how they're interacting with them, not how they could use them. Both have both very different UIs and very different Excel plugins (assuming someone has both on their desk, from experience, they probably use the Excel plugins a lot/mission critical functionality).
deleted 12y ago
[deleted]
innguest 12y ago
Amazing how propaganda machine NY Times is fast to cover this story. That crash was totally not strategic at all. /s
- ceejayoz 12y ago
  What weird conspiracy theory are you flogging? I can't even make sense of the accusation here.
octo_t 12y ago
https://news.ycombinator.com/item?id=9393096 https://news.ycombinator.com/item?id=9393096 previous discussion from earlier today.
apaprocki 12y ago
"Service has been fully restored. We experienced a combination of hardware and software failures in the network, which caused an excessive volume of network traffic. This led to customer disconnections as a result of the machines being overwhelmed. We discovered the root cause quickly, isolated the faulty hardware, and restarted the software. We are reviewing our multiple redundant systems, which failed to prevent this disruption."
RockyMcNuts 12y ago
Probably just a large sugary beverage having its revenge on Mike Bloomberg. http://www.businessinsider.com/reason-for-the-global-bloomberg-outage-2015-4 http://www.businessinsider.com/reason-for-the-global-bloombe... But if you want to cripple the financial system, Bloomberg's a good single point of failure to attack. That and Seamless, which also had a hiccup today.
mbrzusto 12y ago
<onion> bloomberg network unable to report crash of major financial information service </onion>
keithpeter 12y ago
“What I miss is the instant Bloomberg chats, which I rate higher than trading or data feeds. The fact is, Bloomberg connects 100 percent of the Street, and all that human intelligence is what makes markets hum.” I enjoyed the article and I have squirrelled this quote from a trader away for future use - the need to meet and exchange information is why we used to have stock exchanges of course.
- jsprogrammer 12y ago
  Except this information is apparently being exchanged in a closed, proprietary market.
memossy 12y ago
Bloomberg starts at $1,750 a month At that level of cost there is an expectation of almost perfect up time and multiple redundancy - the only reason people pay that much is they believe they can get more value than that out of it..
- hueving 12y ago
  Cost isn't necessarily tied to uptime expectations. It's tied to the value people get from it. Bloomberg is so far ahead of it's competitors that this minor event probably won't lose them any business.
- jschwartzi 12y ago
  It's rare enough that it got a news report. I'd say that qualifies as "almost perfect."
Scoundreller 12y ago
Moderately off-topic (and don't say: "Today? Nowhere."): If I wanted to play around with a Bloomberg for a few hours in North America, just to see what the fuss is about, how could I go about doing this?
- lmg643 12y ago
  New York Public library (business/finance library) on Madison Ave between 33-34th (NYC) has 2-3 terminals for open public use. Usually a short wait but you can use them.
- semi-extrinsic 12y ago
  Since we're moderately off-topic: I once made a gnuplot script that emulates the look of a Bloomberg terminal. Super hacky, but it works: http://folk.ntnu.no/asmunder/bloom.gnu http://folk.ntnu.no/asmunder/bloom.gnu and you need this file as well http://folk.ntnu.no/asmunder/fader.png http://folk.ntnu.no/asmunder/fader.png
- bhatbha 12y ago
  The Graduate School of Business Library at Stanford used to have a couple of terminals, in case you are in the bay area.
PhantomGremlin 12y ago
LOL. Maybe it was a can of coke??? But reports from inside the company suggested that a spilled can of Coke in one of the server rooms had been responsible for knocking out systems across two continents. http://www.dailymail.co.uk/news/article-3043927/Did-spilt-COKE-cause-City-computer-blackout-brought-world-s-financial-markets-standstill.html http://www.dailymail.co.uk/news/article-3043927/Did-spilt-CO...
- el_duderino 12y ago
  I honestly can't believe some (semi reputable) news sites are spewing this bullshit. Fuckin please.

132 points by svtrent 12y ago

Decades of crusty code - C++ wrapped around C wrapped around Fortran, home brewed versions of the STL, nih syndrome across the tooling. Actually surprised outages not more common. Still, given their interview process is basically parlour trick questions about pointers and char arrays, it is quite telling!

bachback 12y ago
Bloomberg started out as a network system independent of the Internet. Michael Bloomberg was a network engineer. I'm not sure they ever really switched to proper TCP/IP. This company sure looks like a dinosaur these days. However its hard to imagine a company being able to replace even a chunk of their infrastructure. They are tightly integrated into all institutions and there is no other service which offers a complete coverage of all the world's financial assets.
- andy_ppp 12y ago
  Yes I think to replace them someone needs to create a marketplace for different aspects of Bloomberg and you buy access to specific services and data as you need it. Would be an amazing business that, supposing you could get buy in from a load of other firms.
  - whatok 12y ago
    Really don't see an ala carte solution being worth it for most firms unless they're really trying to cut down on cost. Integrating and maintaining separate services alone is going to require additional developers. Cost associated with that coupled with quality of service and reliability (outside of this situation) of Bloomberg's current offering makes it a very tough sale. Also, guys who have been using terminals for years are really not going to want to switch.
    - andy_ppp 12y ago
      I'm more thinking "if someone was to compete with Bloomberg how would they even come close to what they offer" may not be practical in reality!
  - bvm 12y ago
    interesting view - what makes Bloomberg brilliant (aside from the support) is that i have, for example, an array of my portfolios that i can access from every function. So if you could create a service that took all of the information that was relevant to you (portfolios, news feeds, models etc) and normalize it to make it interoperable with a whole host of 3rd party systems, you could be on to something!
- apaprocki 12y ago
  Yes, we use proper TCP/IP :)
  - tacostakohashi 12y ago
    ...among other things.
  - kokey 12y ago
    The issue was probably something like a broadcast storm because of defective fault detection on some redundant network kit. That won't stop people blaming it on the fact that the stack isn't built on the latest hipster frameworks.
    - noir_lord 12y ago
      While I see your point do we have to throw the word "hipster" at everything new?. I'm not a huge fan of Javascript and it's proliferation of frameworks but some of that stuff is objectively both good and useful.
      - PhantomGremlin 12y ago
        It's human nature to want to label things. IIRC a while ago I was watching one of those "house hunter" type shows and a husband on there was really annoying. I found a discussion online and people were calling him a "hipster douche". So, perhaps hipster by itself isn't so bad. :) Back a long time ago the applicable pejorative was "yuppie". I haven't heard that one recently. I suspect that many people reading here would have to look it up in Wikipedia. Edit: BTW at the time I was partial to calling people "yuppie scum". Sigh. It's so hard to keep up with the changing putdowns! :)
      - santaclaus 12y ago
        Using the term 'hipster' to refer to the newest and shiniest web framework always felt like a bit of a misnomer to me. A solid portion of hipsterdom is rooted in recycling of the obscure and of the vintage (vinyl, the popularity of the lumber sexual look, etc). It seems like the 'hipster' thing to do would be to write a web framework in PL/I or cobol or something.
7sigma 12y ago
yep, lets not forget 700mb + executables...
- andy_ppp 12y ago
  That equate from what I hear to 128gb+ memory usage!
- tacostakohashi 12y ago
  Only 700mb? Maybe back in the perkin elmer days, these days they're up to 4gb, constrained only by the various linkers, file size and address spaces available...
  - bonaldi 12y ago
    Server-side is very big - in fact, a long time ago the part called simply "the Big" got too big and now there are multiple Bigs.
    - mahyarm 12y ago
      And this is one reason why I don't like C++. I only hear about executables getting too big in C++ projects.
    - psawaya 12y ago
      When I was there (2008, intern), they were in the process of moving server code from Bigs into separate services (BAS's, if I recall).
lectrick 12y ago
I swear I should start a consultancy in the control of software complexity because I am seeing this sort of thing everywhere, and the downages and security issues will only continue (or get worse, as we continue to rely more on software)
ketralnis 12y ago
It's crusty old code because it works and it's tried and true. If you have a backend that works, why mess with it? Sure, C++/C/Fortran/some home grown template library is what it's built on. What would they gain by rewriting it some node framework of the week or whatever other shiny distraction? If they did do that and that caused downtime, there would be the same naysayers here complaining about that instead.

anonu 12y ago

Is this the first crack in the Hoover Dam that is Bloomberg Terminals? Probably not - but the widespread reliance on the $10k+ per year terminal in the financial services industry means that this thing is ripe for disruption. The key isn't to build a terminal replacement though. fintech startups need to chip away at little bits and pieces - and need to inter-operate as much as possible. Only then will the bbg dominance be reined in.

anmilo 12y ago
one of the reasons i'm generally Ok with Slack's valuation is that there is a ton of room for a messaging service to offer a service that is secure and auditable at the level that the SEC/FINRA and Banks require. Like the article mentions, if you talk to the people using BBG, it's really the chat functionality that provides the main value for literally everyone, at least in my experience (former front office dev)
- Karunamon 12y ago
  I'm unfamiliar with these terminals (have kind of always wanted to play with one), but at first blush, a "secure and auditable" messaging system does not seem that difficult to build. Is it a legitimate compliance or practical implementation problem, or is it just the network effect?
  - whatok 12y ago
    Network effect.
    - schintan 12y ago
      Its not just the network effect..Bloomberg's messaging system is tightly integrated with the other functionality of the terminal..for example one can execute a trade via a message or get live streaming quotes.
  - easytiger 12y ago
    its also easy to keep it secure when it doesnt use the internet

chollida1 12y ago

The outage isn't that bad. I've talked to 5 other firms in Canada this morning and haven't heard of a single one that is down. Bloomberg's tickering plant is still working and their emsx(trading system) is up and running, albeit a bit slow right now. I think it was on a stackoverflow podcast that Joel or Jeff said that it makes you wonder about people who complain about down time if they've ever built something of their own. Google's been down, ditto for facebook, twitter and Azure. If you build a big service, something will eventually fail, Bloomberg is no different, if anything they are much better than Reuters or CapIQ for availability. I saw this quote from another comment here: > but the widespread reliance on the $10k+ per year terminal in the financial services industry means that this thing is ripe for disruption. I completely disagree, Bloomberg isn't some slow lethargic big company. They iterate relentlessly, their help desk is the best I've ever had the pleasure of dealing with. If I've got a problem with the trading system, their first level help will get me to a technical specialist in that area in under 30 seconds. If they can't solve it then I'll talk to a developer in 30 minutes. Bloomberg is the leader in teh financial space, not only because their tech is the best in teh business, but because they listen to their customer better than their competitors, they release more frequently, and when they want to enter an area( say risk management) they go out and buy someone who is already a leader in that field. In short they never stop running, if anything they are increasing the distance between them and their "peers". As one example, they just posted a message to every Bloomberg terminal user telling us that the issue was an internal networking issue that has been solved. EDIT to answer this question: > How do they protect the ticker tape thing from attacks, like say, a DDOS? 2 things: 1) This question really doesn't make sense. How do you DDOS a one way line? They send you quotes for the tickers that you request. You can't really DDOS them, unless you do something like request tickers over and over again in a tight loop, in which case their api will rate limit you. 2) its a closed system. Only terminal users can access it and only through the bloomberg terminal or their api, so they control all the access points.

brador 12y ago
How do they protect the ticker tape thing from attacks, like say, a DDOS?
- 12y ago

akramhussein 12y ago

Random anecdote: Was told by a trader a few years back that one trading desk had both Bloomberg and Thomson Reuters systems next to one another and once they were able to capitalise on Bloomberg being a few seconds faster to make a net gain.

RayVR 12y ago
This is really not possible. If there's any truth to this story at all you have forgotten the real version.
zhte415 12y ago
Having a Reuters and Bloomberg terminal on the desk next to each other is commonplace, I did have, plus a Datastream. As for a few seconds delay, that's probably how they're interacting with them, not how they could use them. Both have both very different UIs and very different Excel plugins (assuming someone has both on their desk, from experience, they probably use the Excel plugins a lot/mission critical functionality).

deleted 12y ago

[deleted]

innguest 12y ago

Amazing how propaganda machine NY Times is fast to cover this story. That crash was totally not strategic at all. /s

ceejayoz 12y ago
What weird conspiracy theory are you flogging? I can't even make sense of the accusation here.

octo_t 12y ago

https://news.ycombinator.com/item?id=9393096 https://news.ycombinator.com/item?id=9393096 previous discussion from earlier today.

apaprocki 12y ago

"Service has been fully restored. We experienced a combination of hardware and software failures in the network, which caused an excessive volume of network traffic. This led to customer disconnections as a result of the machines being overwhelmed. We discovered the root cause quickly, isolated the faulty hardware, and restarted the software. We are reviewing our multiple redundant systems, which failed to prevent this disruption."

RockyMcNuts 12y ago

Probably just a large sugary beverage having its revenge on Mike Bloomberg. http://www.businessinsider.com/reason-for-the-global-bloomberg-outage-2015-4 http://www.businessinsider.com/reason-for-the-global-bloombe... But if you want to cripple the financial system, Bloomberg's a good single point of failure to attack. That and Seamless, which also had a hiccup today.

mbrzusto 12y ago

<onion> bloomberg network unable to report crash of major financial information service </onion>

keithpeter 12y ago

“What I miss is the instant Bloomberg chats, which I rate higher than trading or data feeds. The fact is, Bloomberg connects 100 percent of the Street, and all that human intelligence is what makes markets hum.” I enjoyed the article and I have squirrelled this quote from a trader away for future use - the need to meet and exchange information is why we used to have stock exchanges of course.

jsprogrammer 12y ago
Except this information is apparently being exchanged in a closed, proprietary market.

memossy 12y ago

Bloomberg starts at $1,750 a month At that level of cost there is an expectation of almost perfect up time and multiple redundancy - the only reason people pay that much is they believe they can get more value than that out of it..

hueving 12y ago
Cost isn't necessarily tied to uptime expectations. It's tied to the value people get from it. Bloomberg is so far ahead of it's competitors that this minor event probably won't lose them any business.
jschwartzi 12y ago
It's rare enough that it got a news report. I'd say that qualifies as "almost perfect."

Scoundreller 12y ago

Moderately off-topic (and don't say: "Today? Nowhere."): If I wanted to play around with a Bloomberg for a few hours in North America, just to see what the fuss is about, how could I go about doing this?

lmg643 12y ago
New York Public library (business/finance library) on Madison Ave between 33-34th (NYC) has 2-3 terminals for open public use. Usually a short wait but you can use them.
semi-extrinsic 12y ago
Since we're moderately off-topic: I once made a gnuplot script that emulates the look of a Bloomberg terminal. Super hacky, but it works: http://folk.ntnu.no/asmunder/bloom.gnu http://folk.ntnu.no/asmunder/bloom.gnu and you need this file as well http://folk.ntnu.no/asmunder/fader.png http://folk.ntnu.no/asmunder/fader.png
bhatbha 12y ago
The Graduate School of Business Library at Stanford used to have a couple of terminals, in case you are in the bay area.

PhantomGremlin 12y ago

LOL. Maybe it was a can of coke??? But reports from inside the company suggested that a spilled can of Coke in one of the server rooms had been responsible for knocking out systems across two continents. http://www.dailymail.co.uk/news/article-3043927/Did-spilt-COKE-cause-City-computer-blackout-brought-world-s-financial-markets-standstill.html http://www.dailymail.co.uk/news/article-3043927/Did-spilt-CO...

el_duderino 12y ago
I honestly can't believe some (semi reputable) news sites are spewing this bullshit. Fuckin please.