6 ms·

The outage isn't that bad. I've talked to 5 other firms in Canada this morning and haven't heard of a single one that is down. Bloomberg's tickering plant is

The outage isn't that bad. I've talked to 5 other firms in Canada this morning and haven't heard of a single one that is down. Bloomberg's tickering plant is still working and their emsx(trading system) is up and running, albeit a bit slow right now. I think it was on a stackoverflow podcast that Joel or Jeff said that it makes you wonder about people who complain about down time if they've ever built something of their own. Google's been down, ditto for facebook, twitter and Azure. If you build a big service, something will eventually fail, Bloomberg is no different, if anything they are much better than Reuters or CapIQ for availability. I saw this quote from another comment here: > but the widespread reliance on the $10k+ per year terminal in the financial services industry means that this thing is ripe for disruption. I completely disagree, Bloomberg isn't some slow lethargic big company. They iterate relentlessly, their help desk is the best I've ever had the pleasure of dealing with. If I've got a problem with the trading system, their first level help will get me to a technical specialist in that area in under 30 seconds. If they can't solve it then I'll talk to a developer in 30 minutes. Bloomberg is the leader in teh financial space, not only because their tech is the best in teh business, but because they listen to their customer better than their competitors, they release more frequently, and when they want to enter an area( say risk management) they go out and buy someone who is already a leader in that field. In short they never stop running, if anything they are increasing the distance between them and their "peers". As one example, they just posted a message to every Bloomberg terminal user telling us that the issue was an internal networking issue that has been solved. EDIT to answer this question: > How do they protect the ticker tape thing from attacks, like say, a DDOS? 2 things: 1) This question really doesn't make sense. How do you DDOS a one way line? They send you quotes for the tickers that you request. You can't really DDOS them, unless you do something like request tickers over and over again in a tight loop, in which case their api will rate limit you. 2) its a closed system. Only terminal users can access it and only through the bloomberg terminal or their api, so they control all the access points.

brador 12y ago
How do they protect the ticker tape thing from attacks, like say, a DDOS?
- chinathrow 12y ago
  If the data is mission critical to your company, you consume the feed via private leased lines or interconnects.
  - deleted 12y ago
    [deleted]
- gaius 12y ago
  The same way that ALL important systems should be protected: by not connecting them to the Internet in the first place.
ztnewman 12y ago
It was pretty widespread in the (European) morning
- bvm 12y ago
  seemed completely and utterly down in London across a good sample of clients.
lmorris84 12y ago
Presumably many institutions would have other sources of pricing data too - I know this is the case in many sell-sides, not sure about everywhere else. I've worked in a few sell-sides and you see bloomberg screens all over the place, but usually alongside front office trading systems like Fidessa which are powered by direct feeds a lot of the time. Probably a much bigger issue for everyone who doesn't have exchange connectivity, but you'd also hope they have some sort of fallback for exactly this scenario. I.e. reuters or something.
- osullivj 12y ago
  Yes, many banks would failover to Reuters IDN_SELECTFEED
hackuser 12y ago
> Google's been down, ditto for facebook, twitter and Azure The first three, at least, are free services and availability isn't a priority for many of their customers. Bloomberg users have different requirements. Boeing 747's are complex systems; it's not acceptable if one fails occasionally.
- cheriot 12y ago
  > Boeing 747's are complex systems; it's not acceptable if one fails occasionally. Yet they have failed. Reality isn't always acceptable.
- aeturnum 12y ago
  > Boeing 747's are complex systems; it's not acceptable if one fails occasionally. You're right that 747s should not just fall out of the sky. However, parts of 747s fail constantly. In a machine that complex, they are guaranteed to have some downtime. A well manufactured 747 will fail cleanly and safely and recover smoothly.
- ryguytilidie 12y ago
  I think comparing the acceptability of failure of a stock information system and a commercial aircraft is a bit unreasonable.
- phaemon 12y ago
  Google Apps is not a free service. The pricing for business is $5 or $10 per user per month. It's worth bearing in mind that when people complain about the service, they are often paying for it.
lmg643 12y ago
The outage affected Europe and Asia during their trading day, which is a very big deal, wouldn't have expected Canada to see anything since they are 5-6 hours behind. Bloomberg runs some of the most reliable technology in the world, and is one of the best at running distributed architecture, so the fact that a problem like this wasn't more contained is surprising, if not shocking.
ketralnis 12y ago
> it makes you wonder about people who complain about down time HN in particular seems to have a "_____ is down" on the front page every few days. It's always filled with comments like this guy[1] saying how their architecture is clearly broken or how any fifteen year old could write something better. [1] https://news.ycombinator.com/item?id=9394071 https://news.ycombinator.com/item?id=9394071
- rgbrenner 12y ago
  hn was never built to be reliable, and has always had reliability issues.. it used to (still is?) run from a single server.
genericacct 12y ago
WRT 2) DOS does not require a public network; you can war dial access points or attack infrastructure to achieve the same effect. Any insight on what happened today? I'm surprised there's no failover mechanism that tries to route traffic some other way in case of a communication failure. Unfortunately BBG requires a bit more scrutiny because when their network goes down entire countries cannot finance themselves -- they had to postpone some debt sales in the UK today I read.

brador 12y ago

How do they protect the ticker tape thing from attacks, like say, a DDOS?

chinathrow 12y ago
If the data is mission critical to your company, you consume the feed via private leased lines or interconnects.
- deleted 12y ago
  [deleted]
gaius 12y ago
The same way that ALL important systems should be protected: by not connecting them to the Internet in the first place.

ztnewman 12y ago

It was pretty widespread in the (European) morning

bvm 12y ago
seemed completely and utterly down in London across a good sample of clients.

lmorris84 12y ago

Presumably many institutions would have other sources of pricing data too - I know this is the case in many sell-sides, not sure about everywhere else. I've worked in a few sell-sides and you see bloomberg screens all over the place, but usually alongside front office trading systems like Fidessa which are powered by direct feeds a lot of the time. Probably a much bigger issue for everyone who doesn't have exchange connectivity, but you'd also hope they have some sort of fallback for exactly this scenario. I.e. reuters or something.

osullivj 12y ago
Yes, many banks would failover to Reuters IDN_SELECTFEED

hackuser 12y ago

> Google's been down, ditto for facebook, twitter and Azure The first three, at least, are free services and availability isn't a priority for many of their customers. Bloomberg users have different requirements. Boeing 747's are complex systems; it's not acceptable if one fails occasionally.

cheriot 12y ago
> Boeing 747's are complex systems; it's not acceptable if one fails occasionally. Yet they have failed. Reality isn't always acceptable.
aeturnum 12y ago
> Boeing 747's are complex systems; it's not acceptable if one fails occasionally. You're right that 747s should not just fall out of the sky. However, parts of 747s fail constantly. In a machine that complex, they are guaranteed to have some downtime. A well manufactured 747 will fail cleanly and safely and recover smoothly.
ryguytilidie 12y ago
I think comparing the acceptability of failure of a stock information system and a commercial aircraft is a bit unreasonable.
phaemon 12y ago
Google Apps is not a free service. The pricing for business is $5 or $10 per user per month. It's worth bearing in mind that when people complain about the service, they are often paying for it.

lmg643 12y ago

The outage affected Europe and Asia during their trading day, which is a very big deal, wouldn't have expected Canada to see anything since they are 5-6 hours behind. Bloomberg runs some of the most reliable technology in the world, and is one of the best at running distributed architecture, so the fact that a problem like this wasn't more contained is surprising, if not shocking.

ketralnis 12y ago

> it makes you wonder about people who complain about down time HN in particular seems to have a "_____ is down" on the front page every few days. It's always filled with comments like this guy[1] saying how their architecture is clearly broken or how any fifteen year old could write something better. [1] https://news.ycombinator.com/item?id=9394071 https://news.ycombinator.com/item?id=9394071

rgbrenner 12y ago
hn was never built to be reliable, and has always had reliability issues.. it used to (still is?) run from a single server.

genericacct 12y ago

WRT 2) DOS does not require a public network; you can war dial access points or attack infrastructure to achieve the same effect. Any insight on what happened today? I'm surprised there's no failover mechanism that tries to route traffic some other way in case of a communication failure. Unfortunately BBG requires a bit more scrutiny because when their network goes down entire countries cannot finance themselves -- they had to postpone some debt sales in the UK today I read.