5 ms·

Maybe this is because of my Ops background, as I was a sysadmin for close to a decade, but I don't understand the aversion to buying actual hardware. When you'

by donw 15y ago

Maybe this is because of my Ops background, as I was a sysadmin for close to a decade, but I don't understand the aversion to buying actual hardware. When you're starting out, you go with something on the cloud -- EC2, Rackspace, Softlayer, Linode, or whatever cheeses your hamburger. That's just common sense. There's no reason for a fresh web or mobile startup to have tens of thousands of dollars of server inventory, bought for the anticipated rush of people that want to get in on the 'social synchronized tuba-thumping' craze. But when you hit the point where you're spending $10k a month on cloud hosting and your team is having to put out performance fires thanks to the shared hardware, why not just buy servers and throw them in a datacenter? Amortized over a year, you'll spend a lot less on capital (network and hardware), and a lot more on labor, as you'll need at least one more team member to handle the on-call rotation load. Apologies if this is rambling -- it's about 7:30am here, and I'm a bit low on sleep. :)

coderdude 15y ago
I can't remember where I saw this but the discussion was centered around rendering with systems like PRMan, Mental Ray etc. The quote went something like "in the 1st world, people cost more than hardware." The cost of the people needed to keep your hardware running smoothly may end up being more than the cost of the hardware and occasional putting out of fires. Though I'm willing to bet you could get away with having one guy handle the firefighting if he doesn't mind being woken up at 3am. In the end I'm a big fan of owning the hardware and dealing with the issues yourself.
- donw 15y ago
  Good point about people being expensive, doubly so in SF or Palo Alto. That said, a small ops team (three people) can manage thousands of machines without much difficulty, and you can probably share two of those with the dev team as Dev-Ops guys, with one grizzled old sysadmin to handle the nastier problems. I think that a lot of companies shy away from buying hardware for two reasons. One, because they haven't sat down and looked at the long-term ROI, and two, because for the past five years or so, every piece of IT media has talked about how stupid buying hardware is, and how we're all just going to go live in the cloud. Sounds quite a bit like the NoSQL rhetoric, actually. The reality is that at a certain scale, it's cheaper from a strategic and/or monetary standpoint to have in-house hardware. Don't get me wrong -- there's definitely a big home in the ecosystem for cloud hosting. Probably less than ten percent of businesses need to buy their own hardware. But the ones that do, need dedicated hardware in a bad way.
  - gaustin 15y ago
    <<That said, a small ops team (three people) can manage thousands of machines without much difficulty, and you can probably share two of those with the dev team as Dev-Ops guys, with one grizzled old sysadmin to handle the nastier problems.>> I worked for a company that contracted with a guy whose entire business was based on this model. He was the hardcore sysadmin and would juggle tons of servers and administrative config for you. He trained the devs to handle all of the mundane ops stuff, with him as backup. Really great system. It was really empowering for us devs, too.
  - ShawnJG 15y ago
    I think as soon as you are starting for real-given the founder are hackers and will be doing most of the coding, a hardware solution amortized over a few years is the best solution.
robotadam 15y ago
I agree with you; a lot of the reason it took us a while to get out of EC2 was my lack of comfort. If I had pushed that earlier many things would have been easier. There are so many great things about EC2, but it pushes the need to think about scaling earlier in many cases, which means that you're thinking about scaling when you still need to be concentrating 100% on product/market fit. We were very fortunate that the first product we built fit the market well, but if we had been wrong the delay could have cost us.
LeafStorm 15y ago
Maybe datacenter companies could offer bare-metal cloud hosting. They rent out the servers and keep the hardware physically working, your devops team handles all the software aspects.
- megaman821 15y ago
  Rackspace offers this but it is about 4x more expensive than their cloud offerings, but they also offer you the option to have a mix of hosted and cloud.
- gtuhl 15y ago
  Softlayer offers this as well. You can spec out a box however you want up to pretty exotic configurations, get billed monthly, and can SSH into them in 3-4 hours. It is expensive, but not as expensive as something like heroku or aws for equivalent cores/RAM, and you get real hard drives.
  - stoph 15y ago
    We are using Softlayer and we found that on our server we were not bottlenecked by memory, CPU or disk I/O. However, we were bottlenecked by our bandwidth which sat around 5 MB/s. Perhaps we could upgrade to a higher-throughput network card or something (I don't maintain our servers), but we ended up just offloading static files to a CDN instead.
    - ericd 15y ago
      Hm, that seems strange. The default SL server configs don't have 1 gigabit cards, those cost 10-20/mo more - perhaps that's why?
    - gtuhl 15y ago
      That does seem odd. I'd file a ticket and see what they have to say about it. In certain datacenters (think might be dallas05?) you can even get a 10Gbps uplink brought to a box so they have pretty big pipes available.
dmk23 15y ago
Actually there is absolutely no need to wait till you are running $10K cloud bill to buy your own hardware. There are plenty of real cheap second-hand boxes you can buy from equipment recycling companies. The real killer with the cloud is bandwidth and storage. With my colo deal I am paying about 5-10% of what bandwidth would cost at Amazon. Not to mention I can put any kind of storage I like with the actual cost per Gb to buy outright being equal to monthly cost on EC2. Go figure! Cloud has its place to handle spikes of demand, but it is really not cost effective to scale.
bevenky 15y ago
Just another way of thinking this out. Cloud essentially creates an abstraction (self or hosted) into units which allow for seamless fail over the of the underlying hardware. E.g. if one computing unit fails things switch automatically. This is where a cloud is essentially different from a VPS I feel. As mentioned, after a certain point a public cloud maybe expensive, however, a similar private cloud could be created using openstack etc..
nobotz 15y ago
Nothing wrong with buying hardware yourself if you know what you are doing. I help run a dedicated hosting company in India, one of the biggest. Most of the hardware we get in our colocation facility from customers is atleast 2 generation old processors and almost always the wrong form factor and it has typically costed them atleast 2x of what it costs us to buy the same hardware due to our scale of acquisition. There are advantages of a Hosting company buying hardware for you and amortizing it. It is impossible for a small startup buying < 5-10 servers to achieve say a cost of Rs. 100-200 USD per month per server including amortization ( say 24 months ), bandwidth, power and remote hands per server. And I super agree with the fact that it is easy to run high bill on cloud hosting where you pay through your nose for everything except perhaps the base level machines for the privilege of being able to burst your load to 100x. Only about 2% of the customers on cloud actually require burst capacity beyond 2x. It is easy enough to achieve cost savings of 30% with always-on 2x capacity using standard dedicated servers from a dedicated server focused hosting company than buying capacity on the cloud and using maybe burst capacity 5% of the time.

by donw 15y ago

coderdude 15y ago

I can't remember where I saw this but the discussion was centered around rendering with systems like PRMan, Mental Ray etc. The quote went something like "in the 1st world, people cost more than hardware." The cost of the people needed to keep your hardware running smoothly may end up being more than the cost of the hardware and occasional putting out of fires. Though I'm willing to bet you could get away with having one guy handle the firefighting if he doesn't mind being woken up at 3am. In the end I'm a big fan of owning the hardware and dealing with the issues yourself.

donw 15y ago
Good point about people being expensive, doubly so in SF or Palo Alto. That said, a small ops team (three people) can manage thousands of machines without much difficulty, and you can probably share two of those with the dev team as Dev-Ops guys, with one grizzled old sysadmin to handle the nastier problems. I think that a lot of companies shy away from buying hardware for two reasons. One, because they haven't sat down and looked at the long-term ROI, and two, because for the past five years or so, every piece of IT media has talked about how stupid buying hardware is, and how we're all just going to go live in the cloud. Sounds quite a bit like the NoSQL rhetoric, actually. The reality is that at a certain scale, it's cheaper from a strategic and/or monetary standpoint to have in-house hardware. Don't get me wrong -- there's definitely a big home in the ecosystem for cloud hosting. Probably less than ten percent of businesses need to buy their own hardware. But the ones that do, need dedicated hardware in a bad way.
- gaustin 15y ago
  <<That said, a small ops team (three people) can manage thousands of machines without much difficulty, and you can probably share two of those with the dev team as Dev-Ops guys, with one grizzled old sysadmin to handle the nastier problems.>> I worked for a company that contracted with a guy whose entire business was based on this model. He was the hardcore sysadmin and would juggle tons of servers and administrative config for you. He trained the devs to handle all of the mundane ops stuff, with him as backup. Really great system. It was really empowering for us devs, too.
- ShawnJG 15y ago
  I think as soon as you are starting for real-given the founder are hackers and will be doing most of the coding, a hardware solution amortized over a few years is the best solution.

robotadam 15y ago

I agree with you; a lot of the reason it took us a while to get out of EC2 was my lack of comfort. If I had pushed that earlier many things would have been easier. There are so many great things about EC2, but it pushes the need to think about scaling earlier in many cases, which means that you're thinking about scaling when you still need to be concentrating 100% on product/market fit. We were very fortunate that the first product we built fit the market well, but if we had been wrong the delay could have cost us.

LeafStorm 15y ago

Maybe datacenter companies could offer bare-metal cloud hosting. They rent out the servers and keep the hardware physically working, your devops team handles all the software aspects.

megaman821 15y ago
Rackspace offers this but it is about 4x more expensive than their cloud offerings, but they also offer you the option to have a mix of hosted and cloud.
gtuhl 15y ago
Softlayer offers this as well. You can spec out a box however you want up to pretty exotic configurations, get billed monthly, and can SSH into them in 3-4 hours. It is expensive, but not as expensive as something like heroku or aws for equivalent cores/RAM, and you get real hard drives.
- stoph 15y ago
  We are using Softlayer and we found that on our server we were not bottlenecked by memory, CPU or disk I/O. However, we were bottlenecked by our bandwidth which sat around 5 MB/s. Perhaps we could upgrade to a higher-throughput network card or something (I don't maintain our servers), but we ended up just offloading static files to a CDN instead.
  - ericd 15y ago
    Hm, that seems strange. The default SL server configs don't have 1 gigabit cards, those cost 10-20/mo more - perhaps that's why?
  - gtuhl 15y ago
    That does seem odd. I'd file a ticket and see what they have to say about it. In certain datacenters (think might be dallas05?) you can even get a 10Gbps uplink brought to a box so they have pretty big pipes available.

dmk23 15y ago

Actually there is absolutely no need to wait till you are running $10K cloud bill to buy your own hardware. There are plenty of real cheap second-hand boxes you can buy from equipment recycling companies. The real killer with the cloud is bandwidth and storage. With my colo deal I am paying about 5-10% of what bandwidth would cost at Amazon. Not to mention I can put any kind of storage I like with the actual cost per Gb to buy outright being equal to monthly cost on EC2. Go figure! Cloud has its place to handle spikes of demand, but it is really not cost effective to scale.

bevenky 15y ago

Just another way of thinking this out. Cloud essentially creates an abstraction (self or hosted) into units which allow for seamless fail over the of the underlying hardware. E.g. if one computing unit fails things switch automatically. This is where a cloud is essentially different from a VPS I feel. As mentioned, after a certain point a public cloud maybe expensive, however, a similar private cloud could be created using openstack etc..

nobotz 15y ago

Nothing wrong with buying hardware yourself if you know what you are doing. I help run a dedicated hosting company in India, one of the biggest. Most of the hardware we get in our colocation facility from customers is atleast 2 generation old processors and almost always the wrong form factor and it has typically costed them atleast 2x of what it costs us to buy the same hardware due to our scale of acquisition. There are advantages of a Hosting company buying hardware for you and amortizing it. It is impossible for a small startup buying < 5-10 servers to achieve say a cost of Rs. 100-200 USD per month per server including amortization ( say 24 months ), bandwidth, power and remote hands per server. And I super agree with the fact that it is easy to run high bill on cloud hosting where you pay through your nose for everything except perhaps the base level machines for the privilege of being able to burst your load to 100x. Only about 2% of the customers on cloud actually require burst capacity beyond 2x. It is easy enough to achieve cost savings of 30% with always-on 2x capacity using standard dedicated servers from a dedicated server focused hosting company than buying capacity on the cloud and using maybe burst capacity 5% of the time.