10 ms·

Going Colo

201 points by wyclif 14y ago

kintamanimatt 14y ago
The article says that you're are not allowed to use the full rated capacity of a circuit. Can anyone tell me why? This seems weird to me. Also, is his concern about earthquakes really warranted? I'd have thought that a DC would have been able to withstand more or less anything thrown its way.
- tptacek 14y ago
  The concern is probably mostly about the power grid. Power problems have caused outages at large colo centers in the last 5 years.
  - kintamanimatt 14y ago
    Why wouldn't they just rate a 20A circuit as 16A if that's all that's usable? What's the benefit to rating it as a 20A circuit if 4A are unusable? I feel like I'm missing something.
    - kika 14y ago
      Because it looks better in the pricelist? 20A for $500 is better than 15A for $500.
    - tedunangst 14y ago
      Peak draw is less than typical draw. If the fuse blew at 16A, you'd only be able to use 12A. So they give you some headroom. But if everybody draws at 20A all the time, the building melts down. Soft quotas vs hard quotas.
    - Daniel_Newby 14y ago
      1. As mmt mentions in another comment, that's the way the U.S. safety codes rate circuit capacity. 2. A 20 A circuit breaker is allowed to trip at 20.000001 A. Derating the continuous load provides the safety margin for reliable operation, even using inexpensive circuit breakers.
    - manuscreationis 14y ago
      Believe it or not, this is standard electrical engineering. The breakers in your home breaker box operate the exact same way. A 20A breaker will trip once you exceed a certain threshold, which I believe is 16 or 17. They're rated to go to 20 by the factory, but to prevent problems with overloading and causing fires/explosions/etc, they purposefully break at a lower threshold. So this isn't just a colo price scam thing, this is the way the world works.
- duskwuff 14y ago
  Most electrical equipment, including computers, draw significantly more power as they're starting up than they do at steady-state. If everyone's server were pulling the maximum rated capacity at steady-state, you'd probably trip a breaker when everything started up (for instance, if power had just been restored after a failure, or if several machines were power cycled simultaneously).
  - cagenut 14y ago
    Its this. You need the headroom for power-on/reboots. Blade chassis are smart enough to stagger starting up each blade for this reason. 1Us don't know about each other.
    - kika 14y ago
      1U also "know" as long as you use proper PDUs (Power Distribution Units, basically power strips with some tiny brain). PDU's will also measure the amps for you and you call poll them for numbers and you can powercycle servers remotely. They're usually worth every penny.
      - deleted 14y ago
        [deleted]
- nodesocket 14y ago
  Metered PDU's are awesome, they have a nice LED display on the front showing the current used amps. We use Tripp lite, cheaper than APC, and hold up solid.
  - kika 14y ago
    I have nothing good to say about our Tripplites. After 2 years in service their brains start to hang rapidly.
    - brandon 14y ago
      I can corroborate this for Tripplite switched PDUs, but their dumb metered PDUs always held up for me.
- wmf 14y ago
  http://en.wikipedia.org/wiki/National_Electrical_Code#Actual_vs_maximum_current_rating http://en.wikipedia.org/wiki/National_Electrical_Code#Actual... I'm not sure why this came about in the first place, though. AFAIK in Europe breakers are rated at 100%.
- mmt 14y ago
  The article says that you're are not allowed to use the full rated capacity of a circuit. Can anyone tell me why? This seems weird to me. I think this is an NFPA/NEC[1] standard. You could look at it another way: circuits are breakered such that you can temporarily peak 25% above their normal capacity without tripping the breaker. I don't know why the standard isn't more conservative, except to guess that it's because typical household/office uses are likely to have low continuous and high intermittent loads (e.g. vacuum cleaner), making it a good trade-off. [1] USA (and Canada?) only. Anyone know how it is in other countries?
  - mindslight 14y ago
    Yes, this is a requirement of the NEC. Branch circuits need to be rated at 125% of the continuous load. In general, wires are sized not only on their overcurrent protection, but also (overbuilt) for the planned load. Unplanned "noncontinuous" loads get a free pass as the only thing that really keeps them in check is the overcurrent protection. But kika's downthread comment is most on the mark - of course data centers would rather advertise the less relevant "circuit rating" as the number is higher. The load calculations are the same either way. (btw speaking of the NEC, http://bulk.resource.org/codes.gov/ http://bulk.resource.org/codes.gov/ is a fantastic resource)
- rdl 14y ago
  There is also power factor, as well as surge capacity.
nodesocket 14y ago
Surprised the author was able to get 100mbps unmetered for such low prices. Would guess the bandwidth is not multi-homed and single carrier. Usually you pay for a standard 100mbps drop, but then 95th percentile based on the number of purchased megabits in your contract.
- throwaway54-762 14y ago
  Did you read the location? "Hurricane Electric" should ring a few warning bells.
  - throwaway54-762 14y ago
    Why the downvotes? HE has really really low-quality bandwidth (highly variable latency) and the datacenter is likely single-homed, so when HE goes down you'll be cut off. This is a valid criticism of HE.
- kika 14y ago
  HE in Fremont is a very special place. I have about 10 racks there. They're cheap and bandwidth is free, the support is awesome, but it's a ghetto, to put it simply. Or a hippie camp. The racks are telecom, not computer ones (which means that they're shallow and narrow, 30" long server prevents the back door from closing). The power is not redundant and only 15A per rack tops (extra amps are ridiculously expensive). But for a frugal startup it's perfect - cheap, good connectivity, awesome support, easy access. Not very reliable, right, but you don't have many users, don't ya? :-)
  - deleted 14y ago
    [deleted]
  - smgoller 14y ago
    They just opened Phase 3 in Fremont2. All the racks are 36" and there is tons of available space. I agree that power is expensive, but the bandwidth cost can't really be beat.
- 14y ago
bschlinker 14y ago
Interesting that the author chose to use a Kill-a-Watt for power consumption monitoring.. seems like a better idea to use the planning applications available from the major vendors to determine usage depending on workload, etc. Dell and HP both have a calculator where you set the system (RAM, CPU, # HDs, etc.) and the load and it will provide you with an approximate power use, along with a peak load.
- ConstantineXVI 14y ago
  The best way to know how much your system consumes under load would be to monitor it's usage under load, no? Seems foolish to take the chance their calculator is off vs. taking the time to measure it yourself.
- gilrain 14y ago
  IIRC from earlier blog posts, Maciej built at least a few of his servers by hand. I imagine he got sucked into the "beautiful details" he mentions needing to resist in this one. (I think he's also mentioned he's having them built for him, now, but I believe they're still not from major vendors.)
- ajdecon 14y ago
  The calculator is useful for approximations, but I've been bitten a few times. The only real way to know, as always, is to measure it. The Dell/HP salesmen are worse, if you try to talk about power, especially if you do HPC. They will often flat-out refuse to believe that you're going to take 100 servers, stuff GPUs and all the RAM you can find in them, and run them at 100% for years on end.
jwegan 14y ago
Another big question for colos is redundant power. Preferably they have two separate lines all the way to the utility and test their generators on a regular basis.
rosser 14y ago
I'm also currently working with Luke, of prgmr.com, to set up a new home for my colo-ed server. (In fact, I'm getting the same 10U, 4 amp package.) He and his team have been fantastic. I can't recommend them enough.
- dholowiski 14y ago
  Honestly, I found that page (http://prgmr.com/san-jose-co-location.html http://prgmr.com/san-jose-co-location.html) fascinating. Way better than the linked article.
  - rdl 14y ago
    EDIT: I was wrong (based on misreading website) They are single-homed to Cogent (at least that's how it appears; I didn't check looking glass), which is fine for a hobby server, but not really good for a startup.
    - lsc 14y ago
      nope. I'm single homed to egihosting, who is mostly he.net but also globalcrossing and nlayer. http://www.datacentermap.com/as/47066_upstreams.html http://www.datacentermap.com/as/47066_upstreams.html (47066 is me) http://www.datacentermap.com/as/18779_upstreams.html http://www.datacentermap.com/as/18779_upstreams.html (18779 is egihosting) Now, that's not a whole heck of a lot better from my point of view; but egihosting (they are another co-lo provider you might want to check out) is multihomed... so if you are the sort that thinks outsourcing means you don't have to worry about it, I have outsourced my multihomed bandwidth to egihosting. The references to Cogent were referring my last attempt to get a second upstream. (the deal appears to have fallen through? we will see. I will have to write about it. It was a long story, in which a mistake I made 7 years ago comes back to bite me in the ass.) I mean, yeah, the network isn't awesome, but eh, it is reasonable, I think, for the cost. I think that if you have physical hardware, you are better off spending the money on a second location (and for the difference between my level and 'premium' co-location, well, you can host with me, you can duplicate your setup with someone else in my price range, and probably have some money left over. Two cheap ones are usually more reliable than one good one (especially as this is way more of a 'market for lemons' than auto buying. It's really hard to get true information about the reliability of various providers.)
justauser 14y ago
As mentioned in the blog, "So many factors to consider!" Do your homework early and upfront even if planning to colo just an Atom or Soekris box. I can tell you that moving many multiples of racked equipment via semi-truck from one datacenter to another is not an enjoyable experience. Ping - Power - Pipe : Ask everything you need. If a datacenter is unwilling to share or discuss who they are, what they do and what they offer then walk away. Colocation is a buyer beware environment. Here's a VERY ROUGH list/braindump of questions that AT THE VERY LEAST you need to know before placing equipment with the colocation provider/datacenter. Tier I,II,III, IV datacenter? Who owns the datacenter? Investors? What's your ASN number(s)? Carrier neutral facility? How many connectivity providers are present in the data center? Does data center run BGP over these providers? IP Transit providers - Bandwidth Pricing? Overage pricing? Straight $mbps/month? Who manages your network? Ip allotment? Assignment procedure? High density for micro vms(Heroku style)? 200 VMs per host meaning 200IPs per vm. Swip/rwhois updates on assignments? Cross-connects? Cross connect pricing? Charges? Local loop(last mile connectivity) charges? BGP support? Is our traffic on a private/separate v-lan from your other customers? Infrastructure? -Redundancy in detail -Brands -Networking -Connection -Power. Maintenance programs? A/B feeds from two separator breakers or breaker panels or better yet two Utility Power Service Provider if they are truly A/B feeds. A/B Diesel-Generators? Additional fuel storage on site? Is a PDU included? If so, how many outlets and is it accessible from the web or telnet? What type of hand off/drops to your rack will you be provided (Ethernet)? Are there raised floors? Are there any fire suppression systems in place? Physical/Video Security? Biometrics? Screening of employees? Insurance requirements to colocate equipment with you? On-site visit requirements? 24x7 staffing? Remote Hands - Cost and Abilities? What's included? Shipping/Receiving of hardware? Sas II 70 or SSAE 16 certification? Audit available for review? -Common practices -Staffing -Security -Emergencies - Escalations -Contacts Outage alerts? Notifications through Twitter or similar offsite status page? RFOs for past outages? What's been your corrective action? Power density per rack? 80% rule or true metered? Can you rack the equipment for us? Who provides cabling? Types of rails for servers? Billing-95th or other? Rack size? Partitioning available? 4u/10u/Quarter/Half? Cages? Lockable or baker's racks? Shared with others? Finally, ask for a listing of ALL fees.
- wmf 14y ago
whimsy 14y ago
Hey, you spelled prgmr.com wrong. (You got it right in the <a> tag though.)
brandon 14y ago
It's really cool that Maciej is so open about his setup. This is especially true given how opinionated people are about these sorts of operations. For example: I'm hoping the Hurricane Electric facility he mentions is Fremont 2. The Fremont 1 facility has been plagued with power and packet loss issues over the last year (ask your friends with Linodes in Fremont all about it ;) It also seems a shame to put both of the deployments so close together. Proximity to your machines is a non-issue with a decent colo provider: his example of driving to Vegas every time a drive fails assumes that the provider would be unable or unwilling to sell him 20 minutes of remote hands. Spend a little extra money up front for out-of-band access to your infrastructure and leverage the provider for the physical stuff!
- wyclif 14y ago
  Is Maciej still hellbanned from HN?
  - tptacek 14y ago
    No, he was only banned for a few hours, but he (understandably) stopped posting here afterwards.
    - joshu 14y ago
      He was banned for voting from months or perhaps years. Edit: "for" -> "from"
      - hboon 14y ago
        Why was that? and the hell bane.
      - tptacek 14y ago
        What? Huh? Really? Why? What could that possibly have accomplished? Never mind. Who cares.
      - didip 14y ago
        I missed out on the whole story, what happened?
Chirael 14y ago
Nice write-up - thanks for this!
halayli 14y ago
This is very similar to what I ended up doing. I bought a server, and hosted it at he.net.
krobertson 14y ago
HE Fremont is a total ghetto. 15A per full rack, essentially no cooling. Its just a massive open warehouse. Fremont 2 is much better than Fremont 1, but still had one power outage when I was there. Connectivity wise, they are pretty good. Sometimes may be a little more DDoS prone though. I've been in several Sacramento datacenters and currently have a half rack in in Webion in Rancho Cordova. Ragingwire is freakin expensive. The pricing with prgmr.com looked pretty good though. SJ gets better bandwidth pricing, and probably about just as crappy on power. If looking for a serious deployment, generally have to look outside CA. Great connectivity, but power is what will kill you.
- getsat 14y ago
  Linode's Fremont datacenter seems to have power outages every three months or so. I believe they're using HE's facilities as well.
- rdl 14y ago
  I pretty much stopped looking at his advice when he said he went to HE. If you create selection criteria where HE passes, the problem is with your criteria.
- lsc 14y ago
  if you are using serious power and are willing to go super high density (e.g. you are willing to pay the premium for blades) BAIS in santa clara is not a bad deal; north of 10Kw usable for under two grand a month. Santa clara power prices are dramatically better than san jose power prices. I'm trying to move my own stuff in the direction of santa clara. Of course, at the low end? I mean, if you are the $200/month customer that is thinking about buying from me? that sort of thing probably makes less difference. The funny thing about he.net is that if you work out the cost per watt, they are more expensive than most places after negotiation. (on the other hand, he.net gives you the price off the top; most other places go for 30-50% over what the real price is.)
  - rdl 14y ago
    I really like Layer42 in Santa Clara (and in Mountain View); I can probably get you a deal on power/bandwidth there (cofounder of my startup has a cage, etc.)
nsxwolf 14y ago
So, what's the consensus regarding colo vs. the cloud? I've enjoyed walking through rows of colo'd equipment, both as a security guard and as a technology professional. There's a lot of romance to it. It also seems that we've entered an era where you're a fool if you colo a lean startup. Is there a best practice, book, really good blog post, etc that helps one determine when it makes sense to colo vs just spinning up more Heroku dynos?
- rdl 14y ago
  IO heavy applications still don't belong in the cloud. They may make more sense in dedicated servers vs. coloing your own hardware, though.
  - lsc 14y ago
    >IO heavy applications still don't belong in the cloud. I agree. >They may make more sense in dedicated servers vs. coloing your own hardware, though. depends on what skills you have already, and then your time to money exchange rate. If you use a lot of compute power, you can save a lot of money with a relatively small time investment, if you already have the skills.
    - rdl 14y ago
      Even though I like messing with hardware, if you've only got a few racks of equipment, some kind of managed hosting often makes sense for alternate sites just due to cost of sending staff there to mess with things. Maybe use colo at your primary site, but managed hosting for a DR site somewhere.
      - lsc 14y ago
        I think "the cloud" is perfect for DR. have a script that spins up your site in the cloud, test it one day a month, and pay for it one day a month. You go down for a day? you pay for one day in the cloud. Of course, this does eat the same amount of labor as building for the cloud /and/ building for co-lo, so you still have the time cost.
troyk 14y ago
If your looking to Sacramento for seismic or other reasons, the place for scrappy startups is istreet solutions. I'm local to Sac and have had all my servers parked their since 2003 with only a few hiccups. If you inquire ask for Mark (the owner), ask for the "Troy Deal" (I am his first customer, leased a cage of an otherwise empty datacenter in 2003, spent way to many creepy nights behind windows servers and a kvm switch before discovering the serenity of headless linux). Not that he'll give the same deal as his first customer, but it doesn't hurt to ask!
mdemare 14y ago
10 terabytes of fanfic? I thought that you could fit the Library of Congress in less. 10TB is 20M novels of 100K words, unzipped.
- astine 14y ago
  "I thought that you could fit the Library of Congress in less." Nope. I've worked there, they've got multiple petabytes of data, though a major portion of that is audio and video. Even the books aren't stored as txt files.
deleted 14y ago
[deleted]
pcowans 14y ago
Just a couple of observations based on experience: - You'll most likely run out of power well before you run out of physical rack space. - You'll (hopefully) spend less time in the data centre than you think, especially if you have good lights out access, remote power bar access etc. If you're locating in a large city you can often get a better deal somewhere in the suburbs rather than downtown with very little downside (at least that seems to be the case here in London). - Interconnects between racks can be awkward to get set up, and you can't necessarily get the rack next door when you want more space. - Redundant everything will save you a lot of hassle, but gets quite expensive. In particular redundant power will save you from downtime when the data centre decide they need to take one of the power feeds down for maintenance (it happens), and setting up redundant (external) networking has the same effect. Having a second network for admin access prevents you from being locked out in some failure cases. Hot swappable drives are useful for systems which are awkward to take offline (e.g. database servers), but not massively useful elsewhere if you have good redundancy at the server level. - Data centre rack floors are about the most unpleasant place to work you can imagine :-) - If you can get a handle for how competent the data centre are at mitigating DoS attacks it's a useful datapoint. If they aren't on the ball an attack against anyone hosted in the same place will take you offline too. having said that, I'm not entirely sure there's a fool-proof way to determine this ahead of time. - You can generally borrow a cart with a keyboard and monitor on it to wheel up to your rack, so no need to have those permanently installed. - Keep a stash of network cables, cable ties etc. at the bottom of your rack - it's annoying to show up only to find you've forgotten that kind of thing. - You may need to register authorised employees before showing up, so make sure you do that before taking your new intern to help rack a server. Expect them to ask for ID before letting you in. - Data centres (at least some of them) will take deliveries on your behalf, so there's normally no need to get things shipped to your office then drive them out yourself.
papsosouid 14y ago
Somewhat on-topicish, does anyone have any recommendations for colocation in Toronto? The only recommendations I've gotten for a reliable provider has been for peer1, but they are insanely expensive and both their support and sales staff are quite the opposite of helpful. For reference (since peer1 tends to make up "whatever we think we can get away with" pricing), I was quoted $60/Mb for 95th percentile or $0.85/GB (which puts 1TB of data transfer at nearly $1000!).
- barkingcat 14y ago
  Give 151 Front a try. http://www.151-front-street.com/ http://www.151-front-street.com/
caseyf 14y ago
We moved from a single dedicated server to a colo 5 years ago and we've never looked back. It's saved us a lot of money. We started with 2 lonely servers in 24U and now we've got 9 servers, a couple routers, a couple switches, and some other stuff in there. Here are my tips for people that are considering a colo. These are really all about working comfortably within a confined space (a half rack or smaller) * 10U is not that much. Don't buy space that you aren't going to need but compare prices and find out about options for expansion. Space isn't expensive but you might have trouble buying space that isn't bundled with power. * Keep your wiring organized. Don't say "I'll fix that later". * This goes hand in hand with the previous: If everything has redundant power, are you going to run out of outlets before you fill your space? Consider installing a metered PDU (or PDUs) before you add any machines * You probably don't need 2U servers. The 1Us that I currently buy have 2 CPUs, 10 x 2.5" drives (also handy for SSD) and redundant power: http://www.supermicro.com/products/system/1U/1027/SYS-1027R-N3RF.cfm http://www.supermicro.com/products/system/1U/1027/SYS-1027R-... Also, like justauser said, get a complete price list. Sucks to grow and find out that you are going to get hosed on cross connects or extra power or something like that.
sargun 14y ago
So, I have a lot of opinions about datacenters. Bandwidth - So, this seriously matters, but you shouldn't spend money on it. The biggest things about bandwidth, and connectivity are: reliability, latency, and capacity (in order of most important to least important) Reliability: This is things like, packet loss, how often your connectivity is impaired, etc... This will have the largest Ux impact. Most bandwidth providers will have a pretty good story about this. Typically, the delta of reliability between a $15/mo provider and a $150/mo provider doesn't actually provide any realistic differences. Don't waste your money. Latency: Although, most of the applications we'll be working on won't be sensitive to connectivity latency, it does create an upper bound of the speed of TCP data transfer. (Look at the big fat, long pipe problem). All of us have big fat, long pipes to our houses phones now. You can't do much for this than locate your datacenter well. The other thing i, you can ensure that your transit provider has good peering relationships, and transit. I would recommend looking at the list of peers through a looking glass (bgp.he.net is a particularly good looking one). Capacity: This is the question of how much bandwidth can I soak up at any given point. You shouldn't ever have to think about this problem. You're probably not going to have to scale your bandwidth faster than what your upstream already has. Nonetheless, if you're running a bandwidth intensive site, you should consider this, because you could potentially choke your upstream. They will more than likely just go ahead and blackhole you in order to preserve the rest of their revenue stream.
dredmorbius 14y ago
If you're looking at Sacramento, consider the flood risk. The entire city sits in a flood plain, and is protected by floodwalls, levies, dikes, and flood gates. The last major flooding of major structures that I'm aware of was 1956 (mostly affecting areas north and east of Sacramento State), but given changing weather patterns and grave concerns over the delta levee system, nothing is certain. Most of the major datacenters are clustered along US50 (which follows the American River), with others downtown and in West Sacramento, also within vulnerable zones. Probabilities of a major flooding event exceed those of a quake. Though of course the Fremont/Hayward datacenters are direction on the Hayward fault.

201 points by wyclif 14y ago

The article says that you're are not allowed to use the full rated capacity of a circuit. Can anyone tell me why? This seems weird to me. Also, is his concern about earthquakes really warranted? I'd have thought that a DC would have been able to withstand more or less anything thrown its way.

tptacek 14y ago
The concern is probably mostly about the power grid. Power problems have caused outages at large colo centers in the last 5 years.
- kintamanimatt 14y ago
  Why wouldn't they just rate a 20A circuit as 16A if that's all that's usable? What's the benefit to rating it as a 20A circuit if 4A are unusable? I feel like I'm missing something.
  - kika 14y ago
    Because it looks better in the pricelist? 20A for $500 is better than 15A for $500.
  - tedunangst 14y ago
    Peak draw is less than typical draw. If the fuse blew at 16A, you'd only be able to use 12A. So they give you some headroom. But if everybody draws at 20A all the time, the building melts down. Soft quotas vs hard quotas.
  - Daniel_Newby 14y ago
    1. As mmt mentions in another comment, that's the way the U.S. safety codes rate circuit capacity. 2. A 20 A circuit breaker is allowed to trip at 20.000001 A. Derating the continuous load provides the safety margin for reliable operation, even using inexpensive circuit breakers.
  - manuscreationis 14y ago
    Believe it or not, this is standard electrical engineering. The breakers in your home breaker box operate the exact same way. A 20A breaker will trip once you exceed a certain threshold, which I believe is 16 or 17. They're rated to go to 20 by the factory, but to prevent problems with overloading and causing fires/explosions/etc, they purposefully break at a lower threshold. So this isn't just a colo price scam thing, this is the way the world works.
duskwuff 14y ago
Most electrical equipment, including computers, draw significantly more power as they're starting up than they do at steady-state. If everyone's server were pulling the maximum rated capacity at steady-state, you'd probably trip a breaker when everything started up (for instance, if power had just been restored after a failure, or if several machines were power cycled simultaneously).
- cagenut 14y ago
  Its this. You need the headroom for power-on/reboots. Blade chassis are smart enough to stagger starting up each blade for this reason. 1Us don't know about each other.
  - kika 14y ago
    1U also "know" as long as you use proper PDUs (Power Distribution Units, basically power strips with some tiny brain). PDU's will also measure the amps for you and you call poll them for numbers and you can powercycle servers remotely. They're usually worth every penny.
    - deleted 14y ago
      [deleted]
nodesocket 14y ago
Metered PDU's are awesome, they have a nice LED display on the front showing the current used amps. We use Tripp lite, cheaper than APC, and hold up solid.
- kika 14y ago
  I have nothing good to say about our Tripplites. After 2 years in service their brains start to hang rapidly.
  - brandon 14y ago
    I can corroborate this for Tripplite switched PDUs, but their dumb metered PDUs always held up for me.
wmf 14y ago
http://en.wikipedia.org/wiki/National_Electrical_Code#Actual_vs_maximum_current_rating http://en.wikipedia.org/wiki/National_Electrical_Code#Actual... I'm not sure why this came about in the first place, though. AFAIK in Europe breakers are rated at 100%.
mmt 14y ago
The article says that you're are not allowed to use the full rated capacity of a circuit. Can anyone tell me why? This seems weird to me. I think this is an NFPA/NEC[1] standard. You could look at it another way: circuits are breakered such that you can temporarily peak 25% above their normal capacity without tripping the breaker. I don't know why the standard isn't more conservative, except to guess that it's because typical household/office uses are likely to have low continuous and high intermittent loads (e.g. vacuum cleaner), making it a good trade-off. [1] USA (and Canada?) only. Anyone know how it is in other countries?
- mindslight 14y ago
  Yes, this is a requirement of the NEC. Branch circuits need to be rated at 125% of the continuous load. In general, wires are sized not only on their overcurrent protection, but also (overbuilt) for the planned load. Unplanned "noncontinuous" loads get a free pass as the only thing that really keeps them in check is the overcurrent protection. But kika's downthread comment is most on the mark - of course data centers would rather advertise the less relevant "circuit rating" as the number is higher. The load calculations are the same either way. (btw speaking of the NEC, http://bulk.resource.org/codes.gov/ http://bulk.resource.org/codes.gov/ is a fantastic resource)
rdl 14y ago
There is also power factor, as well as surge capacity.

nodesocket 14y ago

Surprised the author was able to get 100mbps unmetered for such low prices. Would guess the bandwidth is not multi-homed and single carrier. Usually you pay for a standard 100mbps drop, but then 95th percentile based on the number of purchased megabits in your contract.

throwaway54-762 14y ago
Did you read the location? "Hurricane Electric" should ring a few warning bells.
- throwaway54-762 14y ago
  Why the downvotes? HE has really really low-quality bandwidth (highly variable latency) and the datacenter is likely single-homed, so when HE goes down you'll be cut off. This is a valid criticism of HE.
kika 14y ago
HE in Fremont is a very special place. I have about 10 racks there. They're cheap and bandwidth is free, the support is awesome, but it's a ghetto, to put it simply. Or a hippie camp. The racks are telecom, not computer ones (which means that they're shallow and narrow, 30" long server prevents the back door from closing). The power is not redundant and only 15A per rack tops (extra amps are ridiculously expensive). But for a frugal startup it's perfect - cheap, good connectivity, awesome support, easy access. Not very reliable, right, but you don't have many users, don't ya? :-)
- deleted 14y ago
  [deleted]
- smgoller 14y ago
  They just opened Phase 3 in Fremont2. All the racks are 36" and there is tons of available space. I agree that power is expensive, but the bandwidth cost can't really be beat.
14y ago

bschlinker 14y ago

Interesting that the author chose to use a Kill-a-Watt for power consumption monitoring.. seems like a better idea to use the planning applications available from the major vendors to determine usage depending on workload, etc. Dell and HP both have a calculator where you set the system (RAM, CPU, # HDs, etc.) and the load and it will provide you with an approximate power use, along with a peak load.

ConstantineXVI 14y ago
The best way to know how much your system consumes under load would be to monitor it's usage under load, no? Seems foolish to take the chance their calculator is off vs. taking the time to measure it yourself.
gilrain 14y ago
IIRC from earlier blog posts, Maciej built at least a few of his servers by hand. I imagine he got sucked into the "beautiful details" he mentions needing to resist in this one. (I think he's also mentioned he's having them built for him, now, but I believe they're still not from major vendors.)
ajdecon 14y ago
The calculator is useful for approximations, but I've been bitten a few times. The only real way to know, as always, is to measure it. The Dell/HP salesmen are worse, if you try to talk about power, especially if you do HPC. They will often flat-out refuse to believe that you're going to take 100 servers, stuff GPUs and all the RAM you can find in them, and run them at 100% for years on end.

jwegan 14y ago

Another big question for colos is redundant power. Preferably they have two separate lines all the way to the utility and test their generators on a regular basis.

rosser 14y ago

I'm also currently working with Luke, of prgmr.com, to set up a new home for my colo-ed server. (In fact, I'm getting the same 10U, 4 amp package.) He and his team have been fantastic. I can't recommend them enough.

dholowiski 14y ago
Honestly, I found that page (http://prgmr.com/san-jose-co-location.html http://prgmr.com/san-jose-co-location.html) fascinating. Way better than the linked article.
- rdl 14y ago
  EDIT: I was wrong (based on misreading website) They are single-homed to Cogent (at least that's how it appears; I didn't check looking glass), which is fine for a hobby server, but not really good for a startup.
  - lsc 14y ago
    nope. I'm single homed to egihosting, who is mostly he.net but also globalcrossing and nlayer. http://www.datacentermap.com/as/47066_upstreams.html http://www.datacentermap.com/as/47066_upstreams.html (47066 is me) http://www.datacentermap.com/as/18779_upstreams.html http://www.datacentermap.com/as/18779_upstreams.html (18779 is egihosting) Now, that's not a whole heck of a lot better from my point of view; but egihosting (they are another co-lo provider you might want to check out) is multihomed... so if you are the sort that thinks outsourcing means you don't have to worry about it, I have outsourced my multihomed bandwidth to egihosting. The references to Cogent were referring my last attempt to get a second upstream. (the deal appears to have fallen through? we will see. I will have to write about it. It was a long story, in which a mistake I made 7 years ago comes back to bite me in the ass.) I mean, yeah, the network isn't awesome, but eh, it is reasonable, I think, for the cost. I think that if you have physical hardware, you are better off spending the money on a second location (and for the difference between my level and 'premium' co-location, well, you can host with me, you can duplicate your setup with someone else in my price range, and probably have some money left over. Two cheap ones are usually more reliable than one good one (especially as this is way more of a 'market for lemons' than auto buying. It's really hard to get true information about the reliability of various providers.)

justauser 14y ago

As mentioned in the blog, "So many factors to consider!" Do your homework early and upfront even if planning to colo just an Atom or Soekris box. I can tell you that moving many multiples of racked equipment via semi-truck from one datacenter to another is not an enjoyable experience. Ping - Power - Pipe : Ask everything you need. If a datacenter is unwilling to share or discuss who they are, what they do and what they offer then walk away. Colocation is a buyer beware environment. Here's a VERY ROUGH list/braindump of questions that AT THE VERY LEAST you need to know before placing equipment with the colocation provider/datacenter. Tier I,II,III, IV datacenter? Who owns the datacenter? Investors? What's your ASN number(s)? Carrier neutral facility? How many connectivity providers are present in the data center? Does data center run BGP over these providers? IP Transit providers - Bandwidth Pricing? Overage pricing? Straight $mbps/month? Who manages your network? Ip allotment? Assignment procedure? High density for micro vms(Heroku style)? 200 VMs per host meaning 200IPs per vm. Swip/rwhois updates on assignments? Cross-connects? Cross connect pricing? Charges? Local loop(last mile connectivity) charges? BGP support? Is our traffic on a private/separate v-lan from your other customers? Infrastructure? -Redundancy in detail -Brands -Networking -Connection -Power. Maintenance programs? A/B feeds from two separator breakers or breaker panels or better yet two Utility Power Service Provider if they are truly A/B feeds. A/B Diesel-Generators? Additional fuel storage on site? Is a PDU included? If so, how many outlets and is it accessible from the web or telnet? What type of hand off/drops to your rack will you be provided (Ethernet)? Are there raised floors? Are there any fire suppression systems in place? Physical/Video Security? Biometrics? Screening of employees? Insurance requirements to colocate equipment with you? On-site visit requirements? 24x7 staffing? Remote Hands - Cost and Abilities? What's included? Shipping/Receiving of hardware? Sas II 70 or SSAE 16 certification? Audit available for review? -Common practices -Staffing -Security -Emergencies - Escalations -Contacts Outage alerts? Notifications through Twitter or similar offsite status page? RFOs for past outages? What's been your corrective action? Power density per rack? 80% rule or true metered? Can you rack the equipment for us? Who provides cabling? Types of rails for servers? Billing-95th or other? Rack size? Partitioning available? 4u/10u/Quarter/Half? Cages? Lockable or baker's racks? Shared with others? Finally, ask for a listing of ALL fees.

wmf 14y ago

whimsy 14y ago

Hey, you spelled prgmr.com wrong. (You got it right in the <a> tag though.)

brandon 14y ago

It's really cool that Maciej is so open about his setup. This is especially true given how opinionated people are about these sorts of operations. For example: I'm hoping the Hurricane Electric facility he mentions is Fremont 2. The Fremont 1 facility has been plagued with power and packet loss issues over the last year (ask your friends with Linodes in Fremont all about it ;) It also seems a shame to put both of the deployments so close together. Proximity to your machines is a non-issue with a decent colo provider: his example of driving to Vegas every time a drive fails assumes that the provider would be unable or unwilling to sell him 20 minutes of remote hands. Spend a little extra money up front for out-of-band access to your infrastructure and leverage the provider for the physical stuff!

wyclif 14y ago
Is Maciej still hellbanned from HN?
- tptacek 14y ago
  No, he was only banned for a few hours, but he (understandably) stopped posting here afterwards.
  - joshu 14y ago
    He was banned for voting from months or perhaps years. Edit: "for" -> "from"
    - hboon 14y ago
      Why was that? and the hell bane.
    - tptacek 14y ago
      What? Huh? Really? Why? What could that possibly have accomplished? Never mind. Who cares.
    - didip 14y ago
      I missed out on the whole story, what happened?

Chirael 14y ago

Nice write-up - thanks for this!

halayli 14y ago

This is very similar to what I ended up doing. I bought a server, and hosted it at he.net.

krobertson 14y ago

HE Fremont is a total ghetto. 15A per full rack, essentially no cooling. Its just a massive open warehouse. Fremont 2 is much better than Fremont 1, but still had one power outage when I was there. Connectivity wise, they are pretty good. Sometimes may be a little more DDoS prone though. I've been in several Sacramento datacenters and currently have a half rack in in Webion in Rancho Cordova. Ragingwire is freakin expensive. The pricing with prgmr.com looked pretty good though. SJ gets better bandwidth pricing, and probably about just as crappy on power. If looking for a serious deployment, generally have to look outside CA. Great connectivity, but power is what will kill you.

getsat 14y ago
Linode's Fremont datacenter seems to have power outages every three months or so. I believe they're using HE's facilities as well.
rdl 14y ago
I pretty much stopped looking at his advice when he said he went to HE. If you create selection criteria where HE passes, the problem is with your criteria.
lsc 14y ago
if you are using serious power and are willing to go super high density (e.g. you are willing to pay the premium for blades) BAIS in santa clara is not a bad deal; north of 10Kw usable for under two grand a month. Santa clara power prices are dramatically better than san jose power prices. I'm trying to move my own stuff in the direction of santa clara. Of course, at the low end? I mean, if you are the $200/month customer that is thinking about buying from me? that sort of thing probably makes less difference. The funny thing about he.net is that if you work out the cost per watt, they are more expensive than most places after negotiation. (on the other hand, he.net gives you the price off the top; most other places go for 30-50% over what the real price is.)
- rdl 14y ago
  I really like Layer42 in Santa Clara (and in Mountain View); I can probably get you a deal on power/bandwidth there (cofounder of my startup has a cage, etc.)

nsxwolf 14y ago

So, what's the consensus regarding colo vs. the cloud? I've enjoyed walking through rows of colo'd equipment, both as a security guard and as a technology professional. There's a lot of romance to it. It also seems that we've entered an era where you're a fool if you colo a lean startup. Is there a best practice, book, really good blog post, etc that helps one determine when it makes sense to colo vs just spinning up more Heroku dynos?

rdl 14y ago
IO heavy applications still don't belong in the cloud. They may make more sense in dedicated servers vs. coloing your own hardware, though.
- lsc 14y ago
  >IO heavy applications still don't belong in the cloud. I agree. >They may make more sense in dedicated servers vs. coloing your own hardware, though. depends on what skills you have already, and then your time to money exchange rate. If you use a lot of compute power, you can save a lot of money with a relatively small time investment, if you already have the skills.
  - rdl 14y ago
    Even though I like messing with hardware, if you've only got a few racks of equipment, some kind of managed hosting often makes sense for alternate sites just due to cost of sending staff there to mess with things. Maybe use colo at your primary site, but managed hosting for a DR site somewhere.
    - lsc 14y ago
      I think "the cloud" is perfect for DR. have a script that spins up your site in the cloud, test it one day a month, and pay for it one day a month. You go down for a day? you pay for one day in the cloud. Of course, this does eat the same amount of labor as building for the cloud /and/ building for co-lo, so you still have the time cost.

troyk 14y ago

If your looking to Sacramento for seismic or other reasons, the place for scrappy startups is istreet solutions. I'm local to Sac and have had all my servers parked their since 2003 with only a few hiccups. If you inquire ask for Mark (the owner), ask for the "Troy Deal" (I am his first customer, leased a cage of an otherwise empty datacenter in 2003, spent way to many creepy nights behind windows servers and a kvm switch before discovering the serenity of headless linux). Not that he'll give the same deal as his first customer, but it doesn't hurt to ask!

mdemare 14y ago

10 terabytes of fanfic? I thought that you could fit the Library of Congress in less. 10TB is 20M novels of 100K words, unzipped.

astine 14y ago
"I thought that you could fit the Library of Congress in less." Nope. I've worked there, they've got multiple petabytes of data, though a major portion of that is audio and video. Even the books aren't stored as txt files.

deleted 14y ago

[deleted]

pcowans 14y ago

Just a couple of observations based on experience: - You'll most likely run out of power well before you run out of physical rack space. - You'll (hopefully) spend less time in the data centre than you think, especially if you have good lights out access, remote power bar access etc. If you're locating in a large city you can often get a better deal somewhere in the suburbs rather than downtown with very little downside (at least that seems to be the case here in London). - Interconnects between racks can be awkward to get set up, and you can't necessarily get the rack next door when you want more space. - Redundant everything will save you a lot of hassle, but gets quite expensive. In particular redundant power will save you from downtime when the data centre decide they need to take one of the power feeds down for maintenance (it happens), and setting up redundant (external) networking has the same effect. Having a second network for admin access prevents you from being locked out in some failure cases. Hot swappable drives are useful for systems which are awkward to take offline (e.g. database servers), but not massively useful elsewhere if you have good redundancy at the server level. - Data centre rack floors are about the most unpleasant place to work you can imagine :-) - If you can get a handle for how competent the data centre are at mitigating DoS attacks it's a useful datapoint. If they aren't on the ball an attack against anyone hosted in the same place will take you offline too. having said that, I'm not entirely sure there's a fool-proof way to determine this ahead of time. - You can generally borrow a cart with a keyboard and monitor on it to wheel up to your rack, so no need to have those permanently installed. - Keep a stash of network cables, cable ties etc. at the bottom of your rack - it's annoying to show up only to find you've forgotten that kind of thing. - You may need to register authorised employees before showing up, so make sure you do that before taking your new intern to help rack a server. Expect them to ask for ID before letting you in. - Data centres (at least some of them) will take deliveries on your behalf, so there's normally no need to get things shipped to your office then drive them out yourself.

papsosouid 14y ago

Somewhat on-topicish, does anyone have any recommendations for colocation in Toronto? The only recommendations I've gotten for a reliable provider has been for peer1, but they are insanely expensive and both their support and sales staff are quite the opposite of helpful. For reference (since peer1 tends to make up "whatever we think we can get away with" pricing), I was quoted $60/Mb for 95th percentile or $0.85/GB (which puts 1TB of data transfer at nearly $1000!).

barkingcat 14y ago
Give 151 Front a try. http://www.151-front-street.com/ http://www.151-front-street.com/

caseyf 14y ago

We moved from a single dedicated server to a colo 5 years ago and we've never looked back. It's saved us a lot of money. We started with 2 lonely servers in 24U and now we've got 9 servers, a couple routers, a couple switches, and some other stuff in there. Here are my tips for people that are considering a colo. These are really all about working comfortably within a confined space (a half rack or smaller) * 10U is not that much. Don't buy space that you aren't going to need but compare prices and find out about options for expansion. Space isn't expensive but you might have trouble buying space that isn't bundled with power. * Keep your wiring organized. Don't say "I'll fix that later". * This goes hand in hand with the previous: If everything has redundant power, are you going to run out of outlets before you fill your space? Consider installing a metered PDU (or PDUs) before you add any machines * You probably don't need 2U servers. The 1Us that I currently buy have 2 CPUs, 10 x 2.5" drives (also handy for SSD) and redundant power: http://www.supermicro.com/products/system/1U/1027/SYS-1027R-N3RF.cfm http://www.supermicro.com/products/system/1U/1027/SYS-1027R-... Also, like justauser said, get a complete price list. Sucks to grow and find out that you are going to get hosed on cross connects or extra power or something like that.

sargun 14y ago

So, I have a lot of opinions about datacenters. Bandwidth - So, this seriously matters, but you shouldn't spend money on it. The biggest things about bandwidth, and connectivity are: reliability, latency, and capacity (in order of most important to least important) Reliability: This is things like, packet loss, how often your connectivity is impaired, etc... This will have the largest Ux impact. Most bandwidth providers will have a pretty good story about this. Typically, the delta of reliability between a $15/mo provider and a $150/mo provider doesn't actually provide any realistic differences. Don't waste your money. Latency: Although, most of the applications we'll be working on won't be sensitive to connectivity latency, it does create an upper bound of the speed of TCP data transfer. (Look at the big fat, long pipe problem). All of us have big fat, long pipes to our houses phones now. You can't do much for this than locate your datacenter well. The other thing i, you can ensure that your transit provider has good peering relationships, and transit. I would recommend looking at the list of peers through a looking glass (bgp.he.net is a particularly good looking one). Capacity: This is the question of how much bandwidth can I soak up at any given point. You shouldn't ever have to think about this problem. You're probably not going to have to scale your bandwidth faster than what your upstream already has. Nonetheless, if you're running a bandwidth intensive site, you should consider this, because you could potentially choke your upstream. They will more than likely just go ahead and blackhole you in order to preserve the rest of their revenue stream.

dredmorbius 14y ago

If you're looking at Sacramento, consider the flood risk. The entire city sits in a flood plain, and is protected by floodwalls, levies, dikes, and flood gates. The last major flooding of major structures that I'm aware of was 1956 (mostly affecting areas north and east of Sacramento State), but given changing weather patterns and grave concerns over the delta levee system, nothing is certain. Most of the major datacenters are clustered along US50 (which follows the American River), with others downtown and in West Sacramento, also within vulnerable zones. Probabilities of a major flooding event exceed those of a quake. Though of course the Fremont/Hayward datacenters are direction on the Hayward fault.