4 ms·

I think the biggest lesson learned here is to not co-locate in the first place. Almost everything they encountered could have been avoided by moving to a "clou

by Corrado 14y ago

I think the biggest lesson learned here is to not co-locate in the first place. Almost everything they encountered could have been avoided by moving to a "cloud" provider (AWS, RackSpace, etc.) While I get a kick out of a well wired rack, I think its a waste of time to do work you shouldn't be doing in the first place. And how much time did it take to do all the experimentation with different rack combinations? Purchasing the components, installing them, and then sending back when they don't fit. Those things don't matter and don't move your company forward, unless you're in the rack management business (like Rackspace or AWS).

druiid 14y ago
I didn't see that as a lesson learned. You have effort involved in moving 'sites' regardless of if you host your own equipment or 'rent'. If you're on the 'cloud' you have to potentially learn about a new providers API, test to make sure that the server blocks you rent are of the same level and capabilities as what your old provider had (meaning performance testing, etc.). Basically you have pain regardless of what you do, one just abstracts a very specific piece of the equation away from you. Frankly I see it as they have the talent in-house to manage their own equipment, and thus they can realize a tremendous cost savings for their monthly bills by not running everything on top of AWS... and no matter which way you shake it, five racks of equipment on a cloud service is going to be more expensive.
kmontrose 14y ago
You give up a lot moving to a cloud solution. http://blog.serverfault.com/2011/03/23/performance-tuning-intel-nics/ http://blog.serverfault.com/2011/03/23/performance-tuning-in... is my go to "we couldn't do things like this in the cloud" example. Not to say the cloud is never the right choice, it's just never made sense for us.
- runarb 14y ago
  On the other hand, many are glad they don’t have to handle the performing tuning of NICs them self…
  - druiid 14y ago
    But... I also highly doubt most cloud providers are bothering to tune NIC's. They are probably just throwing KVM/Xen up on some standard hardware (whatever is cheapest with highest memory density) and opening the machine up for access. There are going to be parts they tune as a default stack, but that far? I somehow doubt it.
  - deleted 14y ago
    [deleted]
- druiid 14y ago
  Yep, totally agree. There's this great 'OMG THE CLOUD' here on HN, but really as I'm sure you guys did, you have to sit back, actually do a cost-benefit analysis (even if it's two guys on a whiteboard and not a full report) and decide what is best for the way the company/product are structured.
- druiid 14y ago
  Thanks for posting this. With some new hardware we purchased we'll be doing the same move Broadcom -> Intel, so I think I'll actually be bookmarking this to come back to later!
- MichaelGG 14y ago
  Really? Last time the subject was mentioned, the StackOverflow people basically said "if you don't buy your own servers then you don't love computers and we love computers so..." - there wasn't an actual cost comparison, maybe I missed it. It seemed to be more just giddiness at buying servers and wiring stuff up. Which seemed odd, seeing as last I heard, SO doesn't even run their own routing. That said, buying servers, especially a gen or two older and refurb, can be a major cost win, so it wouldn't surprise me. Running the actual racks, meh, I'd rather have the colo people handle that unless there's so many racks that it's simply not feasible. For a couple dozen servers, I can't see the point in running it all yourself; it's pretty straightforward plugging stuff in, ain't it?
- deleted 14y ago
  [deleted]
- Corrado 14y ago
  Yes, you do give up a lot for the cloud and your example is a great one. However, those types of problems are fairly rare and can generally be designed around. I think S.O. would be a great service to move off of real hardware. They don't seem to be doing things where dedicated hardware would give you an advantage (big data, video transcoding, 3D graphics, etc.) so why go through the trouble of managing it yourself. Plus, if the system were designed correctly, you get fail-over for "free".
AznHisoka 14y ago
It's easy to say this when you're hacking on a side project and just need the smallest instance possible, or just a medium instance. But when you're scaling for millions of users, then those costs add up.. really really quick. AWS, especially nickel and dime you for the smallest of things, such as total # of I/O requests. Plus I find, the whole myth about able to scale faster.. somewhat misleading. I use EngineYard, and you need to ALLOCATE a certain amt of space. Sure you can increase it in the future, but you'd need to shut down the instance, and create a new one with the extra space... not exactly fast nor convenient. And it's not like you can't add an extra hard drive if you hosted your own servers...
- druiid 14y ago
  I think there is always going to be some amount of overhead with physical or cloud services for scaling. I can say though that one thing you can do with physical servers which is pretty neat is shut down entire physical nodes when you don't need them to save on resources. Your scaling software has to know how to do this, but IPMI is fun stuff like that.
- Corrado 14y ago
  I agree that the cloud can make the best case for hacking side projects. It's quick to deploy and quick to destroy. However, scaling your system depends on how you architect the system. Building in the cloud is not the same as building in your colo cabinet. Your example of moving your instance to a bigger one is a classic example of doing it wrong (so to speak). In the cloud you don't scale up, you scale out; many machines doing the work. And bringing those machines on-line takes minutes, that is certainly faster than shutting down and rebuilding. I've played with EngineYard & Heroku and while I really like their services, they tend to re-enforce the "old world" way of doing something. Break out and play with "raw" AWS and I think you'll find a lot to like.
rdl 14y ago
There's something between "cloud" and "build your own servers and rack them yourselves" -- I think managed colo/dedicated hosting is probably the sweet spot for most companies which don't need the burstiness of cloud servers and aren't in the dedicated datacenter business themselves (or with unique hardware, performance, security, etc. needs). e.g. SoftLayer dedicated servers, or a quality local/smaller/industry-specific provider. IIRC hackernews runs on a SoftLayer dedicated server.

by Corrado 14y ago

druiid 14y ago

I didn't see that as a lesson learned. You have effort involved in moving 'sites' regardless of if you host your own equipment or 'rent'. If you're on the 'cloud' you have to potentially learn about a new providers API, test to make sure that the server blocks you rent are of the same level and capabilities as what your old provider had (meaning performance testing, etc.). Basically you have pain regardless of what you do, one just abstracts a very specific piece of the equation away from you. Frankly I see it as they have the talent in-house to manage their own equipment, and thus they can realize a tremendous cost savings for their monthly bills by not running everything on top of AWS... and no matter which way you shake it, five racks of equipment on a cloud service is going to be more expensive.

kmontrose 14y ago

You give up a lot moving to a cloud solution. http://blog.serverfault.com/2011/03/23/performance-tuning-intel-nics/ http://blog.serverfault.com/2011/03/23/performance-tuning-in... is my go to "we couldn't do things like this in the cloud" example. Not to say the cloud is never the right choice, it's just never made sense for us.

runarb 14y ago
On the other hand, many are glad they don’t have to handle the performing tuning of NICs them self…
- druiid 14y ago
  But... I also highly doubt most cloud providers are bothering to tune NIC's. They are probably just throwing KVM/Xen up on some standard hardware (whatever is cheapest with highest memory density) and opening the machine up for access. There are going to be parts they tune as a default stack, but that far? I somehow doubt it.
- deleted 14y ago
  [deleted]
druiid 14y ago
Yep, totally agree. There's this great 'OMG THE CLOUD' here on HN, but really as I'm sure you guys did, you have to sit back, actually do a cost-benefit analysis (even if it's two guys on a whiteboard and not a full report) and decide what is best for the way the company/product are structured.
druiid 14y ago
Thanks for posting this. With some new hardware we purchased we'll be doing the same move Broadcom -> Intel, so I think I'll actually be bookmarking this to come back to later!
MichaelGG 14y ago
Really? Last time the subject was mentioned, the StackOverflow people basically said "if you don't buy your own servers then you don't love computers and we love computers so..." - there wasn't an actual cost comparison, maybe I missed it. It seemed to be more just giddiness at buying servers and wiring stuff up. Which seemed odd, seeing as last I heard, SO doesn't even run their own routing. That said, buying servers, especially a gen or two older and refurb, can be a major cost win, so it wouldn't surprise me. Running the actual racks, meh, I'd rather have the colo people handle that unless there's so many racks that it's simply not feasible. For a couple dozen servers, I can't see the point in running it all yourself; it's pretty straightforward plugging stuff in, ain't it?
deleted 14y ago
[deleted]
Corrado 14y ago
Yes, you do give up a lot for the cloud and your example is a great one. However, those types of problems are fairly rare and can generally be designed around. I think S.O. would be a great service to move off of real hardware. They don't seem to be doing things where dedicated hardware would give you an advantage (big data, video transcoding, 3D graphics, etc.) so why go through the trouble of managing it yourself. Plus, if the system were designed correctly, you get fail-over for "free".

AznHisoka 14y ago

It's easy to say this when you're hacking on a side project and just need the smallest instance possible, or just a medium instance. But when you're scaling for millions of users, then those costs add up.. really really quick. AWS, especially nickel and dime you for the smallest of things, such as total # of I/O requests. Plus I find, the whole myth about able to scale faster.. somewhat misleading. I use EngineYard, and you need to ALLOCATE a certain amt of space. Sure you can increase it in the future, but you'd need to shut down the instance, and create a new one with the extra space... not exactly fast nor convenient. And it's not like you can't add an extra hard drive if you hosted your own servers...

druiid 14y ago
I think there is always going to be some amount of overhead with physical or cloud services for scaling. I can say though that one thing you can do with physical servers which is pretty neat is shut down entire physical nodes when you don't need them to save on resources. Your scaling software has to know how to do this, but IPMI is fun stuff like that.
Corrado 14y ago
I agree that the cloud can make the best case for hacking side projects. It's quick to deploy and quick to destroy. However, scaling your system depends on how you architect the system. Building in the cloud is not the same as building in your colo cabinet. Your example of moving your instance to a bigger one is a classic example of doing it wrong (so to speak). In the cloud you don't scale up, you scale out; many machines doing the work. And bringing those machines on-line takes minutes, that is certainly faster than shutting down and rebuilding. I've played with EngineYard & Heroku and while I really like their services, they tend to re-enforce the "old world" way of doing something. Break out and play with "raw" AWS and I think you'll find a lot to like.

rdl 14y ago

There's something between "cloud" and "build your own servers and rack them yourselves" -- I think managed colo/dedicated hosting is probably the sweet spot for most companies which don't need the burstiness of cloud servers and aren't in the dedicated datacenter business themselves (or with unique hardware, performance, security, etc. needs). e.g. SoftLayer dedicated servers, or a quality local/smaller/industry-specific provider. IIRC hackernews runs on a SoftLayer dedicated server.