8 ms·

Inside Equinix's NY4 data center where Wall Street trades

79 points by dodders 10y ago

chollida1 10y ago
If you are at all into this sort of thing( data centers) and you ever get a chance to go, I strongly recommend it. The amount of tech in this building is just staggering. As the article hints at, everyone who is everyone in finance has a spot there. This has an effect of the more people who are colocated there, the bigger the draw it becomes for the remaining few who aren't currently colocated there. In the interconnected world of finance, data center cross connects become very important. Its not unusual for a prop trading firm to have 30 different cross connects to vendors, exchanges and sell side firms. So I guess its not surprising that there have been companies formed with the sole intention of connecting companies that already exist in teh data center. Radianz and Lucera are mentioned in the article but there are usually a half a dozen in each of the main data centers in the new york and chicago surrounding area. as usual Nanex does a decent job of describing the reason why there are so many data centers and they all are atleast 10 miles apart. http://www.nanex.net/aqck2/3532.html http://www.nanex.net/aqck2/3532.html Another article from today that talks about the new data center. https://medium.com/@RobinWigg/wave-a-final-goodbye-to-this-5e8a03d2d653#.ou51wdi0m https://medium.com/@RobinWigg/wave-a-final-goodbye-to-this-5...
- noir_lord 10y ago
  The Medium piece is interesting, still not sure where I stand on HFT, the technology is endlessly fascinating I'm just not sure about the ends to which it is put.
- rayshan 10y ago
  How does one go about visiting such a facility? Do you have to be a customer or can anyone just request a tour?
godgod 10y ago
The stock market is rigged. It's all about who can get the stock data the fastest. Must watch: https://www.youtube.com/watch?v=XpvS05WvHbk https://www.youtube.com/watch?v=XpvS05WvHbk
sna1l 10y ago
I visited a Equinix data center in San Jose and it was an incredible experience. The redundancies and failsafes they have in place are mind blowing. No two doors to the same room can be open at the same time. All doors use handprint readers. They have a rack of big batteries that fill up a huge room that go on automatically in case of power failure. This will last about an hour until a 10000 HP generator on the roof starts and powers the data center for a week.
- jorts 10y ago
  Typically the operators will have an agreement so that they can get the generators refueled should the outage last even longer than what fuel they have on premise, so they theoretically could run indefinitely on generator power.
  - barkingcat 10y ago
    An outage requiring generator power for longer than 1 week might not allow transportation of fuel. Just want to bring that up in case people really did think you can run "indefinitely" on generator power if the roads, bridges, and gas supply network is out, or if there's an emergency state of war or terrorism. At the 1wk+ territory, it will start to look something like Katrina, where the roads were all blocked by flooding waters and fuel resupply missions required the protection of the National Guard. I mean you can charter helicopters to bring in fuel, but I think there are more urgent things to power than a datacentre in those cases, for example, hospitals and shelters for emergency humanitarian aid.
    - chinathrow 10y ago
      Never underestimate what people would do to keep the stock market running...
      - stephengillie 10y ago
        These days, I'd hope they can failover to standby hosts in "IL#" or "VA#" or another datacenter in a nearby region. The DR/BC space has become pretty robust, and with the right vendor, a datacenter loss can be recovered from simply by powering on fully-replicated VMs in another datacenter.
elecengin 10y ago
I've been to NY4! It really is less magical then these articles make it seem. Yes - the security is a bit imposing. For anyone familiar with datacenters, though, it is pretty run of the mill except for it's tremendous scale. Worth shouting out to Lucera - I have met the CEO and he is a really thoughtful technologist. They are doing some awesome work bringing AWS-style computing to the low latency space. Some interesting details on their deployment: http://www.enterprisetech.com/2014/02/20/amazon-cant-catch-lucera-financial-cloud/ http://www.enterprisetech.com/2014/02/20/amazon-cant-catch-l...
- chasb 10y ago
  If high-frequency trading were limited or abolished, would Lucera's advantage over AWS disappear?
  - puppetmaster 10y ago
    Not really. Even in a world where HFT is limited, cross connections provide consistent connections between trading entities. While AWS provides over-the-Internet or "black boxed" networking on a best efforts basis, Lucera's connections are point to point.
chinathrow 10y ago
What is it with these new auto-play video distractions lately on bloomberg.com? Does that really improve time spent on the site?
atemerev 10y ago
Together with LD4 in London and TY3 in Tokyo, I think, more than 50% of worlds trading activity goes through these 3 buildings.
atemerev 10y ago
(The easiest way to cause worldwide financial panic is to bring a GPS spoofer near NY4. Most of HFT operations synchronize time via GPS receivers).
- arca_vorago 10y ago
  Who needs a spoofer when you can make a jammer or emp? (don't do this, I have a friend of a friend story of a guy who did this in uni and got visited by men in black suits, the professor almost got fired)
- minimax 10y ago
  Any GPS clock you are going to find in NY4 is going to be sufficiently sophisticated to check the GPS signal against its internal oscillator in order to check for validity. It can fall back to the internal oscillator if the GPS time looks bad. The other reason this is unlikely to be a problem is that network time protocols break all the time (oops someone did a maintenance and broke sync with the master clock). Lots of real world trading code has sanity checks on the time.
  - atemerev 10y ago
    This is why I mentioned a spoofer, not jammer :)
    - minimax 10y ago
      It's the same problem. Assume my GPS clock is synced to actual GPS. You turn on your spoofer and now my clock sees that the GPS source is running faster or slower than it should be so it falls back to the internal oscillator.
      - atemerev 10y ago
        What if I am careful not to mess up and introduce only microsecond/millisecond level corrections? :)
- 10y ago
deleted 10y ago
[deleted]
hbcondo714 10y ago
What tremendous overhead Equinix is taking on with their facilities and clientele. The Bloomberg author mentions their annual filing[1] which discloses that they have a net income loss and 'substantial debt'. [1] https://www.last10k.com/sec-filings/eqix https://www.last10k.com/sec-filings/eqix
- nickpsecurity 10y ago
  The numbers I saw were positive. It was over $100mil in 2015 with projected $10 in 2016. Revenues are huge. Im a bit tired so might have missed the negatives but looked good at a glance.
jtchang 10y ago
Equinix is really a top tier provider. I have some space down in San Jose SV5 and it's pretty awesome. If anyone is interested in a colo swap let me know. I have a few units free and would love to barter it for some space maybe a bit closer to SF. Also just for kicks I requested a quote for the SV5 center a few months back. They are roughly 3.2k / month for a full rack [1]: [1] http://i.imgur.com/ZPDm3gi.png http://i.imgur.com/ZPDm3gi.png
- nickpsecurity 10y ago
  That seems like a bargain for the level of assurance they provide. Two such locations with fail-over is median cost of a single, IT worker.
- bogomipz 10y ago
  What makes them a "top tier provider"? They provide power and cooling and rack real estate. They don't have a backbone. There's plenty of carrier neutral facilities that provide the same for cheaper. They all provide similar SLAs these days.
meesterdude 10y ago
Wow, I remember this project! I was involved in the Electrical commissioning for the first phase of the buildout. I Remember the 2 hour car rides at 5AM to get there. One night I had to stay up overnight doing a UPS load-test there, was cool walking around with nobody in sight, boring otherwise.
bogomipz 10y ago
Equinix loves the flash interior decorating but it doesn't do anything for your uptime or latency. Datacenters really aren't that interesting, maybe the "meet me room" have some neat looking network gear but by and large DC are visually monotonous. Question - aside from having a high concentration of wall street firms colocated there what is interesting about this? What innovations are they making? The funny thing about all the security - the man traps, the palm reader, the alarms systems there is still usually some security guard at the front desk who is often napping. Not that I blame them but its kind of laughable.

79 points by dodders 10y ago

If you are at all into this sort of thing( data centers) and you ever get a chance to go, I strongly recommend it. The amount of tech in this building is just staggering. As the article hints at, everyone who is everyone in finance has a spot there. This has an effect of the more people who are colocated there, the bigger the draw it becomes for the remaining few who aren't currently colocated there. In the interconnected world of finance, data center cross connects become very important. Its not unusual for a prop trading firm to have 30 different cross connects to vendors, exchanges and sell side firms. So I guess its not surprising that there have been companies formed with the sole intention of connecting companies that already exist in teh data center. Radianz and Lucera are mentioned in the article but there are usually a half a dozen in each of the main data centers in the new york and chicago surrounding area. as usual Nanex does a decent job of describing the reason why there are so many data centers and they all are atleast 10 miles apart. http://www.nanex.net/aqck2/3532.html http://www.nanex.net/aqck2/3532.html Another article from today that talks about the new data center. https://medium.com/@RobinWigg/wave-a-final-goodbye-to-this-5e8a03d2d653#.ou51wdi0m https://medium.com/@RobinWigg/wave-a-final-goodbye-to-this-5...

noir_lord 10y ago
The Medium piece is interesting, still not sure where I stand on HFT, the technology is endlessly fascinating I'm just not sure about the ends to which it is put.
rayshan 10y ago
How does one go about visiting such a facility? Do you have to be a customer or can anyone just request a tour?

godgod 10y ago

The stock market is rigged. It's all about who can get the stock data the fastest. Must watch: https://www.youtube.com/watch?v=XpvS05WvHbk https://www.youtube.com/watch?v=XpvS05WvHbk

sna1l 10y ago

I visited a Equinix data center in San Jose and it was an incredible experience. The redundancies and failsafes they have in place are mind blowing. No two doors to the same room can be open at the same time. All doors use handprint readers. They have a rack of big batteries that fill up a huge room that go on automatically in case of power failure. This will last about an hour until a 10000 HP generator on the roof starts and powers the data center for a week.

jorts 10y ago
Typically the operators will have an agreement so that they can get the generators refueled should the outage last even longer than what fuel they have on premise, so they theoretically could run indefinitely on generator power.
- barkingcat 10y ago
  An outage requiring generator power for longer than 1 week might not allow transportation of fuel. Just want to bring that up in case people really did think you can run "indefinitely" on generator power if the roads, bridges, and gas supply network is out, or if there's an emergency state of war or terrorism. At the 1wk+ territory, it will start to look something like Katrina, where the roads were all blocked by flooding waters and fuel resupply missions required the protection of the National Guard. I mean you can charter helicopters to bring in fuel, but I think there are more urgent things to power than a datacentre in those cases, for example, hospitals and shelters for emergency humanitarian aid.
  - chinathrow 10y ago
    Never underestimate what people would do to keep the stock market running...
    - stephengillie 10y ago
      These days, I'd hope they can failover to standby hosts in "IL#" or "VA#" or another datacenter in a nearby region. The DR/BC space has become pretty robust, and with the right vendor, a datacenter loss can be recovered from simply by powering on fully-replicated VMs in another datacenter.

elecengin 10y ago

I've been to NY4! It really is less magical then these articles make it seem. Yes - the security is a bit imposing. For anyone familiar with datacenters, though, it is pretty run of the mill except for it's tremendous scale. Worth shouting out to Lucera - I have met the CEO and he is a really thoughtful technologist. They are doing some awesome work bringing AWS-style computing to the low latency space. Some interesting details on their deployment: http://www.enterprisetech.com/2014/02/20/amazon-cant-catch-lucera-financial-cloud/ http://www.enterprisetech.com/2014/02/20/amazon-cant-catch-l...

chasb 10y ago
If high-frequency trading were limited or abolished, would Lucera's advantage over AWS disappear?
- puppetmaster 10y ago
  Not really. Even in a world where HFT is limited, cross connections provide consistent connections between trading entities. While AWS provides over-the-Internet or "black boxed" networking on a best efforts basis, Lucera's connections are point to point.

chinathrow 10y ago

What is it with these new auto-play video distractions lately on bloomberg.com? Does that really improve time spent on the site?

atemerev 10y ago

Together with LD4 in London and TY3 in Tokyo, I think, more than 50% of worlds trading activity goes through these 3 buildings.

atemerev 10y ago

(The easiest way to cause worldwide financial panic is to bring a GPS spoofer near NY4. Most of HFT operations synchronize time via GPS receivers).

arca_vorago 10y ago
Who needs a spoofer when you can make a jammer or emp? (don't do this, I have a friend of a friend story of a guy who did this in uni and got visited by men in black suits, the professor almost got fired)
minimax 10y ago
Any GPS clock you are going to find in NY4 is going to be sufficiently sophisticated to check the GPS signal against its internal oscillator in order to check for validity. It can fall back to the internal oscillator if the GPS time looks bad. The other reason this is unlikely to be a problem is that network time protocols break all the time (oops someone did a maintenance and broke sync with the master clock). Lots of real world trading code has sanity checks on the time.
- atemerev 10y ago
  This is why I mentioned a spoofer, not jammer :)
  - minimax 10y ago
    It's the same problem. Assume my GPS clock is synced to actual GPS. You turn on your spoofer and now my clock sees that the GPS source is running faster or slower than it should be so it falls back to the internal oscillator.
    - atemerev 10y ago
      What if I am careful not to mess up and introduce only microsecond/millisecond level corrections? :)
10y ago

deleted 10y ago

[deleted]

hbcondo714 10y ago

What tremendous overhead Equinix is taking on with their facilities and clientele. The Bloomberg author mentions their annual filing[1] which discloses that they have a net income loss and 'substantial debt'. [1] https://www.last10k.com/sec-filings/eqix https://www.last10k.com/sec-filings/eqix

nickpsecurity 10y ago
The numbers I saw were positive. It was over $100mil in 2015 with projected $10 in 2016. Revenues are huge. Im a bit tired so might have missed the negatives but looked good at a glance.

jtchang 10y ago

Equinix is really a top tier provider. I have some space down in San Jose SV5 and it's pretty awesome. If anyone is interested in a colo swap let me know. I have a few units free and would love to barter it for some space maybe a bit closer to SF. Also just for kicks I requested a quote for the SV5 center a few months back. They are roughly 3.2k / month for a full rack [1]: [1] http://i.imgur.com/ZPDm3gi.png http://i.imgur.com/ZPDm3gi.png

nickpsecurity 10y ago
That seems like a bargain for the level of assurance they provide. Two such locations with fail-over is median cost of a single, IT worker.
bogomipz 10y ago
What makes them a "top tier provider"? They provide power and cooling and rack real estate. They don't have a backbone. There's plenty of carrier neutral facilities that provide the same for cheaper. They all provide similar SLAs these days.

meesterdude 10y ago

Wow, I remember this project! I was involved in the Electrical commissioning for the first phase of the buildout. I Remember the 2 hour car rides at 5AM to get there. One night I had to stay up overnight doing a UPS load-test there, was cool walking around with nobody in sight, boring otherwise.

bogomipz 10y ago

Equinix loves the flash interior decorating but it doesn't do anything for your uptime or latency. Datacenters really aren't that interesting, maybe the "meet me room" have some neat looking network gear but by and large DC are visually monotonous. Question - aside from having a high concentration of wall street firms colocated there what is interesting about this? What innovations are they making? The funny thing about all the security - the man traps, the palm reader, the alarms systems there is still usually some security guard at the front desk who is often napping. Not that I blame them but its kind of laughable.