6 ms·

Stack Exchange’s Colocation Move: Lessons Learned

126 points by revorad 14y ago

grogers 14y ago
Is it really with getting Colo space for just one rack of servers? To me this seems to be well below the threshold where owning your own hardware makes sense.
- KyleBrandt 14y ago
  Probably depends on what direction you think you will go in long term. Companies tend to gain knowledge and expertise over time so it less common to to shift hosting models (except early on). At just one rack, if you want to own hardware you would usually just rent a single cabinet (Can be pretty cheap depending on location). The number of servers in a rack is variable depending on the power. We got 208v / 30A , so we can pretty much fill up a rack with redundant power. In our main facility, we currently have 4 racks in use which are not full yet and a 5th already provisioned for growth.
- druiid 14y ago
  I would disagree. With modern processors and memory you can cram a LOT into one cabinet. We run with Dell C8000's and R620/R720's and they are 'wow' level fast. Given that you can cram dual processors and 256GB of ram * 8 into 4U with the new C8000's you basically can have around 2TB of memory and 16 physical cores in 4U... that's a lot of hardware. Addendum: I should say though that be aware the C8000 servers are hungry beasts for power. I believe they are nearly 9A at full draw.
- borlak 14y ago
  I worked at a place where the hardware in one rack easily exceeded 150k. We started saving money vs. a cloud solution in about 6 months. You can pack a lot of firepower in a virtualized environment.
  - InclinedPlane 14y ago
    When you own the hardware you can also get a tax break on hardware depreciation.
jedberg 14y ago
I can definitely relate to the last part: > It might make us weirdoes, but when cabling looks this neat it is a sexy and sleek piece of art. I always tell people that server cabling is as much art is it science. When I was early in my career, I had some great mentors in this respect, and now when I see a well cabled rack, it really speaks to me. Here is the reddit server rack just before we tore it down: http://imgur.com/DlaX4 http://imgur.com/DlaX4
- KyleBrandt 14y ago
  Next post will go in details, but we jave a lot cables. A web server for example: - 2 Ethernet for Redis/Web/Other traffic - 2 Ethernet for DB traffic - 1 Ethernet for our KVM - 1 Ethernet for our drac - 2 Power cables for A/B Power Getting all the neatly into 1U cable arms was ... fun :-P
  - skorgu 14y ago
    Drac and kvm?
    - KyleBrandt 14y ago
      The DRACs have some nice features: Allows you to mount external media, see the previous boot screen etc. However, when physically at a facility, DRACs are not very handy. They also don't allow you to easily switch back forth over multiple servers. The two also in part act as a back up for each other.
      - skorgu 14y ago
        We've always used a crash cart if the dracs didn't cut it. Which KVM did you go with? All the IP KVMs I've used were pretty horrid to use.
- barredo 14y ago
  And here's an entire subreddit http://www.reddit.com/r/cableporn/top/ http://www.reddit.com/r/cableporn/top/
Corrado 14y ago
I think the biggest lesson learned here is to not co-locate in the first place. Almost everything they encountered could have been avoided by moving to a "cloud" provider (AWS, RackSpace, etc.) While I get a kick out of a well wired rack, I think its a waste of time to do work you shouldn't be doing in the first place. And how much time did it take to do all the experimentation with different rack combinations? Purchasing the components, installing them, and then sending back when they don't fit. Those things don't matter and don't move your company forward, unless you're in the rack management business (like Rackspace or AWS).
- druiid 14y ago
  I didn't see that as a lesson learned. You have effort involved in moving 'sites' regardless of if you host your own equipment or 'rent'. If you're on the 'cloud' you have to potentially learn about a new providers API, test to make sure that the server blocks you rent are of the same level and capabilities as what your old provider had (meaning performance testing, etc.). Basically you have pain regardless of what you do, one just abstracts a very specific piece of the equation away from you. Frankly I see it as they have the talent in-house to manage their own equipment, and thus they can realize a tremendous cost savings for their monthly bills by not running everything on top of AWS... and no matter which way you shake it, five racks of equipment on a cloud service is going to be more expensive.
- kmontrose 14y ago
  You give up a lot moving to a cloud solution. http://blog.serverfault.com/2011/03/23/performance-tuning-intel-nics/ http://blog.serverfault.com/2011/03/23/performance-tuning-in... is my go to "we couldn't do things like this in the cloud" example. Not to say the cloud is never the right choice, it's just never made sense for us.
  - runarb 14y ago
    On the other hand, many are glad they don’t have to handle the performing tuning of NICs them self…
hhw 14y ago
On lesson 2, power is always the biggest concern when it comes to colo so it may not necessarily be that the facility is attempting to nickel and dime you (although that is often also the case), but that they need to restrict power density to a certain level so they can ensure they have enough power for the rest of the facility and so that they can provide adequate cooling. Pricing is also highly dependent on how much available space there is in a facility; the best deals are to be had when a facility is brand new and needs to fill the space. Whereas a facility that's almost full may stick to full list price and be content to have you walk if you don't want to pay those rates. On lesson 6, when the hosting provider does own the building, chances are there will be less bandwidth options available within the building unless the provider specifically offers bandwidth neutral services. May not be a concern if you're happy just using the provider's own bandwidth mix, but if you're pushing a significant amount of traffic, or need higher quality transit, you are much better off shopping around for different bandwidth options.
- KyleBrandt 14y ago
  On 2: Many of the providers we looked at did say it was cooling. They would give higher power to single racks, but then they would make us take up more floor space. So really floor space is often how the express their cooling capacity. On 6: That is a very good point. We ended up going to the Google building which has a lot of transit options. Transit if definitely a factor
jrochkind1 14y ago
> Our biggest error with this was not making one person ultimately responsible for the physical design. Choices need to be made and not everyone’s ideas can be reconciled with each other and the constraints of reality. For a holistic design, eventually someone has to reconcile reality with what everyone wants or you end up with a bunch of individually well thought out pieces that don’t fit together (as well as a bit of frustration.) That doesn't just apply to server room design. It applies to just about ANY design. Including software.
ChuckMcM 14y ago
At Blekko we ended up moving 600 servers from one co-location facility over to a neighboring town about 5 miles away. It is a lot of work, and its a lot of co-ordination. When I was at Google I thought they were just being arrogant by designing their own racks, building them and shipping them in pre-built "chunks" to the data center. I was very wrong about that, it saves a huge amount of time. I look forward to a time when a colo facility says something like "We can lease you 50 OpenCompute 2.0 slots for $3,000 a month" and know that I can just populate the hardware, plug my switches into the structured wiring solution and be done. Not sure if Colos will last that long though :-)

126 points by revorad 14y ago

Is it really with getting Colo space for just one rack of servers? To me this seems to be well below the threshold where owning your own hardware makes sense.

KyleBrandt 14y ago
Probably depends on what direction you think you will go in long term. Companies tend to gain knowledge and expertise over time so it less common to to shift hosting models (except early on). At just one rack, if you want to own hardware you would usually just rent a single cabinet (Can be pretty cheap depending on location). The number of servers in a rack is variable depending on the power. We got 208v / 30A , so we can pretty much fill up a rack with redundant power. In our main facility, we currently have 4 racks in use which are not full yet and a 5th already provisioned for growth.
druiid 14y ago
I would disagree. With modern processors and memory you can cram a LOT into one cabinet. We run with Dell C8000's and R620/R720's and they are 'wow' level fast. Given that you can cram dual processors and 256GB of ram * 8 into 4U with the new C8000's you basically can have around 2TB of memory and 16 physical cores in 4U... that's a lot of hardware. Addendum: I should say though that be aware the C8000 servers are hungry beasts for power. I believe they are nearly 9A at full draw.
borlak 14y ago
I worked at a place where the hardware in one rack easily exceeded 150k. We started saving money vs. a cloud solution in about 6 months. You can pack a lot of firepower in a virtualized environment.
- InclinedPlane 14y ago
  When you own the hardware you can also get a tax break on hardware depreciation.

jedberg 14y ago

I can definitely relate to the last part: > It might make us weirdoes, but when cabling looks this neat it is a sexy and sleek piece of art. I always tell people that server cabling is as much art is it science. When I was early in my career, I had some great mentors in this respect, and now when I see a well cabled rack, it really speaks to me. Here is the reddit server rack just before we tore it down: http://imgur.com/DlaX4 http://imgur.com/DlaX4

KyleBrandt 14y ago
Next post will go in details, but we jave a lot cables. A web server for example: - 2 Ethernet for Redis/Web/Other traffic - 2 Ethernet for DB traffic - 1 Ethernet for our KVM - 1 Ethernet for our drac - 2 Power cables for A/B Power Getting all the neatly into 1U cable arms was ... fun :-P
- skorgu 14y ago
  Drac and kvm?
  - KyleBrandt 14y ago
    The DRACs have some nice features: Allows you to mount external media, see the previous boot screen etc. However, when physically at a facility, DRACs are not very handy. They also don't allow you to easily switch back forth over multiple servers. The two also in part act as a back up for each other.
    - skorgu 14y ago
      We've always used a crash cart if the dracs didn't cut it. Which KVM did you go with? All the IP KVMs I've used were pretty horrid to use.
barredo 14y ago
And here's an entire subreddit http://www.reddit.com/r/cableporn/top/ http://www.reddit.com/r/cableporn/top/

Corrado 14y ago

I think the biggest lesson learned here is to not co-locate in the first place. Almost everything they encountered could have been avoided by moving to a "cloud" provider (AWS, RackSpace, etc.) While I get a kick out of a well wired rack, I think its a waste of time to do work you shouldn't be doing in the first place. And how much time did it take to do all the experimentation with different rack combinations? Purchasing the components, installing them, and then sending back when they don't fit. Those things don't matter and don't move your company forward, unless you're in the rack management business (like Rackspace or AWS).

druiid 14y ago
I didn't see that as a lesson learned. You have effort involved in moving 'sites' regardless of if you host your own equipment or 'rent'. If you're on the 'cloud' you have to potentially learn about a new providers API, test to make sure that the server blocks you rent are of the same level and capabilities as what your old provider had (meaning performance testing, etc.). Basically you have pain regardless of what you do, one just abstracts a very specific piece of the equation away from you. Frankly I see it as they have the talent in-house to manage their own equipment, and thus they can realize a tremendous cost savings for their monthly bills by not running everything on top of AWS... and no matter which way you shake it, five racks of equipment on a cloud service is going to be more expensive.
kmontrose 14y ago
You give up a lot moving to a cloud solution. http://blog.serverfault.com/2011/03/23/performance-tuning-intel-nics/ http://blog.serverfault.com/2011/03/23/performance-tuning-in... is my go to "we couldn't do things like this in the cloud" example. Not to say the cloud is never the right choice, it's just never made sense for us.
- runarb 14y ago
  On the other hand, many are glad they don’t have to handle the performing tuning of NICs them self…

hhw 14y ago

On lesson 2, power is always the biggest concern when it comes to colo so it may not necessarily be that the facility is attempting to nickel and dime you (although that is often also the case), but that they need to restrict power density to a certain level so they can ensure they have enough power for the rest of the facility and so that they can provide adequate cooling. Pricing is also highly dependent on how much available space there is in a facility; the best deals are to be had when a facility is brand new and needs to fill the space. Whereas a facility that's almost full may stick to full list price and be content to have you walk if you don't want to pay those rates. On lesson 6, when the hosting provider does own the building, chances are there will be less bandwidth options available within the building unless the provider specifically offers bandwidth neutral services. May not be a concern if you're happy just using the provider's own bandwidth mix, but if you're pushing a significant amount of traffic, or need higher quality transit, you are much better off shopping around for different bandwidth options.

KyleBrandt 14y ago
On 2: Many of the providers we looked at did say it was cooling. They would give higher power to single racks, but then they would make us take up more floor space. So really floor space is often how the express their cooling capacity. On 6: That is a very good point. We ended up going to the Google building which has a lot of transit options. Transit if definitely a factor

jrochkind1 14y ago

> Our biggest error with this was not making one person ultimately responsible for the physical design. Choices need to be made and not everyone’s ideas can be reconciled with each other and the constraints of reality. For a holistic design, eventually someone has to reconcile reality with what everyone wants or you end up with a bunch of individually well thought out pieces that don’t fit together (as well as a bit of frustration.) That doesn't just apply to server room design. It applies to just about ANY design. Including software.

ChuckMcM 14y ago

At Blekko we ended up moving 600 servers from one co-location facility over to a neighboring town about 5 miles away. It is a lot of work, and its a lot of co-ordination. When I was at Google I thought they were just being arrogant by designing their own racks, building them and shipping them in pre-built "chunks" to the data center. I was very wrong about that, it saves a huge amount of time. I look forward to a time when a colo facility says something like "We can lease you 50 OpenCompute 2.0 slots for $3,000 a month" and know that I can just populate the hardware, plug my switches into the structured wiring solution and be done. Not sure if Colos will last that long though :-)