8 ms·

Building Servers for Fun and Prof... OK, Maybe Just for Fun

86 points by bussetta 14y ago

randomchars 14y ago
> But not the kind that go in your home. No, that'd be as nuts as the now-discontinued Windows Home Server product. What's wrong with home servers? I've been wanting to build one for quite some time.
- ios84dev 14y ago
  I think he means rack mount home servers, those are pretty nuts because of the noise that most rack mount cases make. Figure in a 1U case you normally have at least 4 15K RPM fans going at all times, even if you put it into "quiet" mode its enough to hear from the next room normally.
  - timc3 14y ago
    My rack mount case is silent enough to record with a high quality condenser microphone in the same room. But it is 4U with 120mm slow fans. Its all about how you build them. Obviously a 1U with 40mm fans running fast could make a lot of noise.
- zdw 14y ago
  HP makes a "Microserver" that's designed for home/office use: http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-4237916-4237918-4237917-4248009.html?dnr=1 http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-42... Quiet, ECC RAM, and decent speed for most home-related tasks (backup, etc.)
  - stordoff 14y ago
    I can highly recommend these. I recently got one of these for £100 (after cashback), and it is a brilliant piece of kit. I've got 4x1.5TB HDDs, and a 250GB OS drive, in mine, running Windows Server 2008 R2, and the performance is pretty good. 50MBps+ over a gigabit network is what I'm getting, which is more than enough for me.
- burningion 14y ago
  In lieu of building a home server, I highly recommend the Synology line of NAS machines. They've got everything you could possibly want out of a home server, with a much faster configuration. Automatic backups, a built in Torrent server, Plex media server, and remote file access all in one container. Seriously a great product and value if you're looking for a home server.
  - randomchars 14y ago
    That looks really nice! Torrent server is one of the primary uses I would need this for so it's tempting.
  - evandena 14y ago
    I don't want to come across as a comment crapper, but I've been recommending FreeNAS for years now. ZFS is perfect for home NAS purposes.
    - burningion 14y ago
      You're not comment crapping at all, you're offering a great alternative to those who are time rich / cash poor, or who like configuring and want to learn about building servers. For anyone who's cash rich and time poor, the NAS products from Synology are great.
    - deleted 14y ago
      [deleted]
  - hkarthik 14y ago
    +1 I love my Synology NAS. Truly a great product.
mdgrech23 14y ago
Hardware is cheap, programmers are expensive. System admins even more so. For this reason I feel like cloud hosting providers are the way to go for bootstrappers or startups. Once you get big you can do things like FB and create your own datacenter. However the enjoyment and power that comes from having full control over your entire environment should never be over looked.
- Detrus 14y ago
  But it also costs programmer time to tweak software for weak hardware. Sometimes it makes sense to spend that time on setting up a few big iron servers yourself.
- Muzza 14y ago
  > System admins even more so. Really? Honest question.
  - timc3 14y ago
    Depends on your business. If your small business is writing code, having a sysadmin around and not 100% busy they are could be costly compared to hiring another programer. But looking at general rates, I would say no.
  - deleted 14y ago
    [deleted]
  - thaumaturgy 14y ago
    Good sysadmins, yes. Keep in mind that the ideal sysadmin is someone who knows the guts of BSD or Linux (typically, not both) inside and out; can troubleshoot the strangest of problems very quickly; is constantly measuring and improving performance; and can hack together shell scripts or Python or even C code as needed. Linux and BSD are hellaciously complex and prone to very strange behavior once you start taking them into high-performance land. If you're OK settling with, "can read syslog and look stuff up on the web", those guys are cheap.
    - 14y ago
tobias3 14y ago
I myself like to do this. But only for home servers. I would never do that for a "serious" application. There are several reasons, the most important one being that this is a perfect example where outsourcing the work is both cheaper (economies of scale), more reliable (parts are tested better) and less risky (companies which do that at a larger scale have better risk management). If one the SSDs dies he has to drive there and switch it out, instead of one person simply doing that for the whole data center. This is simply inefficient. And of course Amazon hosting is more expensive. It is more flexible; you can spin up instances at your whim. You pay for that. It would be better to compare it with standard dedicated server hosting.
- peterwwillis 14y ago
  Have you worked with "professional support" in a colo before? When a hard drive dies under vendor support, one guy from the vendor drives out there to replace it. Half the time they don't test it after it's replaced unless you request it. Sometimes the parts are duds. Sometimes they bring the wrong part, or it doesn't fix whatever was broken on the server. If it's the datacenter's remote hands, it can be hours, and on rare occasions days, before somebody starts working on your issue or even answers the phone or e-mail. Same issues come, and you have to have your own spare hardware for the remote hands to use. It's highly variant based on the datacenter and vendor(s) and other issues. There's no guarantee that outsourcing will be reliable. You have to find the best one possible and build a good relationship.
  - codinghorror 14y ago
    Having done this twice before, I strongly, STRONGLY recommend picking a colocation center that's less than an hour drive from someone who works for the company. Or you. Just in case, but that does come up from time to time and if it's many hours away (or worst of all an airplane ride) you will get hours/days to fix things as you noted.
    - peterwwillis 14y ago
      The shop I worked with, we had support on the machines, so Dell would send a tech out to replace a part for the servers that had valid support contracts. We still needed someone on the phone to remote-in to test the fix, of course. For more complicated problems, our main colo was about 45 mins away from almost all the SAs. We used remote hands for colos that were an airplane ride away, and had varying (read: sometimes really shitty) results.
vmind 14y ago
I much prefer the middle ground of dedicated servers over the hassle of colocation and hardware management. (16GB RAM quad core with 2TB raid1 from OVH for £65 a month are a good affordable level, and very quick and easy to spin up new ones).
- girasquid 14y ago
  If bang for your buck is a concern, you might be interested in this offering from Hetzner: http://www.hetzner.de/en/hosting/produkte_rootserver/ex4s http://www.hetzner.de/en/hosting/produkte_rootserver/ex4s - you get double the RAM and a bit more disk space for a little bit less. I only have a month's worth of experience to talk about it, but other than the fact that it's in Germany (timezones) it's been great.
  - vmind 14y ago
    Yep, have used Hetzner servers too for some things and would use again, but for our customers being 90% in the US, and OVH offer a canadian location (and we may need some EU servers in the future).
    - girasquid 14y ago
      Fair enough. I'm always curious about why people choose one provider over another - thanks for filling me in on your reasoning. :)
- henrikschroder 14y ago
  I was also gonna come in here and mention the middle ground. Amazon is great either if you don't want to commit to anything, or if you have very spiky load and sometimes need to spin up a lot of servers. Building your own servers is dirt cheap, but there's always the risk of hardware breaking, and you can't exactly scale up very quickly. But the middle ground, rented dedicated servers, can be a perfect compromise for a lot of companies. Scaling up and adding servers might take a day instead of instantly, and it costs more than owning servers, but you get all the benefits of managed hosting, broken hardware simply isn't something you need to worry about anymore. If a server goes wonky, just scrap it and rent a new one.
stephengillie 14y ago
What happens when a HDD starts to go out? Do you have to pay & trust colo personnel to replace it, or do you have to pay to have your server shipped back to you? I could see putting my own server into a colo if it were like a storage locker - carry/cart your server into the building, mount it in the rack you've rented, plug into the provided power and network connections, and lock the security door.
- patio11 14y ago
  Typically you'd pay for the first option. The term of art is "remote hands" service. There are options where you can install servers in cages. (Some industries/companies require very strict physical access control.) This is more for show than because it meaningfully increases your security against the attack "your colocation provider is secretly The Adversary."
  - peterwwillis 14y ago
    If you buy the hardware from a vendor and get their 24/7 support they'll send a tech to your cage to replace the drive. It's a little bit more expensive than the machine he built, but with the knowledge that someone will go fix it when it's broken. Worth the money, still ridiculously cheaper than AWS.
  - jaf656s 14y ago
    >This is more for show than because it meaningfully increases your security against the attack "your colocation provider is secretly The Adversary." It does help protect you against the "another colo customer" attack, whether an actual attack or accident.
- spindritf 14y ago
  There's always a third option: go to the collocation facility and change the disk yourself. You won't usually be charged for access during business hours. Attractiveness of that solution depends greatly on the distance to the DC.
- codinghorror 14y ago
gprasanth 14y ago
Network connectivity is one very big problem. What good is a blazing fast server with 2Mbps (disruptive!) link? This is the very reason a lot of people would hate their own servers. Cloud = connectivity + (computing power) Yay! for connectivity.
- codinghorror 14y ago
  Of course it depends on what you're doing. Running a porn site or image/video sharing service? You're going to go through a ton of bandwidth and that might end up being your bottleneck way more than performance.
  - dholowiski 14y ago
    > Of course it depends on what you're doing. This sums up every comment here. Every web app is a unique snowflake and what works for you (EC2/Dedicated, SSD, RAM bandwidth) is just as unique. Much more important than building your own server, is fully understanding your requirements, and deploying the appropriate solution. But it is still wicked fun to build a server.
jaequery 14y ago
i've known this secret for a while. i went from paying $4k+/month spanning over 10+ servers in amazon ec2, to just 2 dedicated servers < $1k/month in an unlimited 1gig colo. the performance difference i see is huge. i never realized how actually slow EC2 is. their issue is definitely in slow IO, which i think only their SSD instances can fix (which will cost me in the $8k+). i now have a wicked setup, Xenserver Cloud, SAN, all Highly Available, don't have to worry about bandwidth overcharges, and much much faster .
- codinghorror 14y ago
  Particularly now that we have 6 GBps SATA and cheap(ish) 512 GB SSD drives, the I/O differential can be enormous. I thought about putting four 512 GB SSDs drives in a Raid 10, which would give me striping performance levels without losing the mirror, but that seemed like a bit too much overkill. These servers have 4 front drive bays, only 2 are filled with the SSD mirror, so we could decide to drop in 2 more drives and rebuild the array if we need even more I/O perf.
btgeekboy 14y ago
I like the part where he builds new servers on a rug. What's the over/under for how long until parts start failing?
- codinghorror 14y ago
  I have built many dozens of computers over the last 20 years and I've never ever seen any static-related problems. I don't wear a static wrist strap, but I do always touch something metal on the case before touching the internals.
  - ersii 14y ago
    I've been saying this as well, but has realised I don't know what problems static electricity could or would cause - long-term nor short-term. I've stopped saying that I've never ever seen any static-related problems and I'm using the silly bracelet connected to something metal these days.
    - tlb 14y ago
      Static does cause damage other than immediate catastrophic failure. It can degrade silicon in subtle ways, resulting in random errors or crashes or even slowness due to retrying transactions. Wearing the wrist strap is cheap insurance against mysterious badness.
      - codinghorror 14y ago
        Yeah, but it's also a little bit voodoo. Unless we can point to specific instances of things going wrong, it's kind of imagineering a problem where one does not actually exist. That said I always advise touching something metal on the exterior case before touching anything in a computer, and that's how I have always done it.
        marshray 14y ago
        Static damage to a chip may not result in an immediate failure. It may manifest as lessened performance, unreliability, or a shortened lifespan. I've seen this demonstrated in training videos for the electronics industry. (I think EEVblog has a static demo up on YouTube). The semiconductor industry (i.e., the companies like Intel that made the parts on your circuit boards) spends many millions, if not billions, of $ a year on static-protective mitigations. They're pretty smart folks and also very cost-conscious. They would not be spending all this money if it were pure voodoo. However, parts mounted on a circuit board are much more resilient than loose chips. I, too, use the "touching metal" method but am very careful about it.
JoeCortopassi 14y ago
"...you don't need the redundancy, geographical backup, and flexibility that comes with cloud virtualization" This is perhaps the single most glossed over topic in the entire article. If I am a 1-5 person shop, maintaining a web app, virtualized hosting pays huge dividends in that I don't even notice if a hard drive or motherboard takes a dump. There are additional costs that come with the benefit of being abstracted away from hardware failure or geographic problems (building fire, power out, etc), and that's something that every business has to evaluate for itself.
- codinghorror 14y ago
  Well, you'd always colocate enough servers so that you can lose at least one machine without caring. E.g. HAProxy to 2 web tier machines on the back end. HAProxy will fail over to just one server no problem. (And yes, you can heartbeat so you have two cheap physical HAProxy machines, too. This gets into sub-blade territory, where the 1U server is internally two or more complete low-ish power servers with independent power supplies, etc. ) That's the whole premise behind FOSS, you don't need to worry about all the licenses, and the hardware is so cheap it is effectively free and getting freer by the second, so you throw a lot of cheap hardware at the problem. But agreed, "lot" in this case means at least two so one can fail and you don't need to care.
  - JoeCortopassi 14y ago
    I guess my bigger point though, would be that when using something like AWS, I don't have to think/spend time on a lot of implementation details. When using colocation services/hardware/failover, I'm just adding a bunch of little things to my daily tasks and responsibilities. Sometimes this is a big deal (like in a bootstrapped two man team) and other times it's not. Ultimately, I think it comes down to priority instead of possibility. If your company lives and dies on having reliable servers, you should probably roll your own. But if servers are 'just' a technical detail to your overall business model, then a cloud solution can be well worth the additional cost
    - BryantD 14y ago
abtinf 14y ago
That article is not just comparing apples to oranges; with that default aws setup, he is comparing apples to a herd of donkeys. The correct comparison is his server vs a single EC2 High-Memory Double Extra Large instance with a 3 year heavy-utilization reservation. This instance costs $3100 upfront plus $0.14/hour. The total 3 year cost for this server on AWS would be 3100 + (.14 * 24 * 365 * 3) = 6779.2, or about $188.31 per month. Sure, its more expensive. But AWS provides an insane of value on top of the server. Like instantly being able to provision additional capacity. I wouldn't be at all surprised if, on a full-loaded cost basis, it is extremely competitive with building his server. Heck, the employee salary expense of building your own server will easily drive the cost of the server well beyond the $3100 up front amazon fee. I love building hardware too (never had a computer I didn't build except for laptops). But my mind boggles at AWS value proposition
- codinghorror 14y ago
  Wait, so you can basically buy a server at AWS EC2 for a term of 3 years? That's what this heavy utilization reservation sounds an awful lot like.
  - deleted 14y ago
    [deleted]
  - recuter 14y ago
    Not only that, now that they have that marketplace going you can pawn off your contract unto somebody else if your needs change. I'm not sure how liquid it is yet. You could also equally go dumpster diving for unused virtual instances, say, if some startup goes under and liquidates.
  - twelve45 14y ago
    Yeah for the closest possible comparison in your scenario, you want to do a 3 year reserved with heavy utilization. When you reserve an instance, you are committing to a higher upfront in order to get a lower hourly for the reservation period. The low/medium/high utilization is sort of a knob that allows you to further control this upfront vs. hourly cost. With high utilization you have the highest upfront, but also the lowest hourly cost. If you are planning to run the server 24x7, this will also give you the lowest total cost.
rdw 14y ago
Where's he colocating those servers? Last time I dabbled with colocation, the bandwidth costs per-server were by far the dominant cost. I found it difficult to get anything at all for small quantities of servers. Looking at he.net at the moment, I see they have a deal for $1/Mbps. Presumably someone like Jeff Atwood can get twice as good a deal as that, so he'd pay around $500/month for bandwidth for those servers. Going by the cheat sheet (https://blog.cloudvertical.com/2012/10/aws-cost-cheat-sheet-2/ https://blog.cloudvertical.com/2012/10/aws-cost-cheat-sheet-...), that is within a factor of 2 of a yearly-reserved h1.4xlarge ($2263/month * 0.47 savings = $1199). It's almost equal to the three-year reserved machine ($2263 * 0.30 = $678). Edit: He probably only needs 1 Gbps for all 4 machines, driving his bandwidth costs down by a factor of 4, but we could start to take power/space/cooling costs into account at that point. So is he getting a better deal than that? I'd love to know where.
- aes256 14y ago
  Maybe I've been out of the game too long, but $1/Mbps sounds very low. Last time I looked into it (one or two years ago) you could only get bargain bucket bandwidth approaching those prices, with fairly crumby providers. I'm assuming Amazon are a cut above that.
  - rdw 14y ago
    Yeah, $1/Mbps sounds like a steal to me as well, which may be why it's a limited-time special offer. thatamurgy's Pinboard links lead to this hosting provider: http://prgmr.com/san-jose-co-location.html http://prgmr.com/san-jose-co-location.html The machines in the original post being ~250W, they'd need the $362.50/month plan. But it's only at 10Mbps, which would only work if the application has low network requirements. Maciej appears to have gotten a much better deal, "For $206, I got 10U, 4 amps, and 100 Mbps connectivity", perhaps in exchange for promoting it through his blog. Everything's easier when you're a celebrity!
- codinghorror 14y ago
  A lot of colocation centers offer "unlimited" bandwidth, though I strongly suspect that may depend on whether you're running a porn site / megaupload clone or not. :) Fortunately bandwidth is one thing that has gotten substantially cheaper over time: http://www.codinghorror.com/blog/2007/02/the-economics-of-bandwidth.html http://www.codinghorror.com/blog/2007/02/the-economics-of-ba...
dholowiski 14y ago
I love building servers, it is amazingly geeky fun. And no doubt colocation with your own servers will give you the best bang for your buck, with renting a dedicated server coming in second. But you do have to be careful about redundancy. If you can't do without a day or two down-time, you'll need 2 or more servers, because when it's a dedicated server it's all up to you (or to someone you hire/pay for) when something fails. Sadly I live a 5 hour drive from the nearest co-location facility, so I'm forced to rent a dedicated server.
- codinghorror 14y ago
  Yeah I would never advise colocating just one server. Two at minimum. We'll have five when all is said and done (still building the other four here).
thaumaturgy 14y ago
maciej from pinboard has written some useful stuff on this too: - The five stages of hosting: http://blog.pinboard.in/2012/01/the_five_stages_of_hosting/ http://blog.pinboard.in/2012/01/the_five_stages_of_hosting/ - Building servers: http://blog.pinboard.in/2012/05/a_cloud_of_my_own/ http://blog.pinboard.in/2012/05/a_cloud_of_my_own/ - Going colo: http://blog.pinboard.in/2012/06/going_colo/ http://blog.pinboard.in/2012/06/going_colo/ A bunch of people are probably going to respond to Jeff's article by saying things like, "But VPS hosting means you get other people to deal with problems for you", but in reality all that means is that you're at maciej's "monastery" or "dorm room" stage of hosting, and your needs haven't yet driven you to get "the apartment".
- codinghorror 14y ago
  Maciej is 105% awesome, I'm a huge fan, and I linked several of his articles in the actual text of my post already. Go read them all!
- mmariani 14y ago
  The five stages of hosting is priceless! I'd really like to go with #5, but I'll stick to #3 for now. :)
- wiredfool 14y ago
  I love maciej's footnote: """* What is it with these aggro facility names? Rather than Hurricane Electric or Raging Wire, I would much prefer to host with "Calm Multihomed Oasis" or "Granite Mountain" or " Cooling Breezes Pure Sine Wave Mount Bitmore". """
timc3 14y ago
Happy rackmount home server user here, running ESXi with a load of dev instances, staging servers, OpenIndiana for ZFS running decent amount of storage connected to my editing workstations, databases, plus looks after our family video and photos. I still use our company servers that are co-located but having things at home on the same network where you are developing is very compelling for me. Its very quiet and cool (latest i7 processor). But if you are hosting your own production boxes, I usually buy from HP or Dell and know that I rarely need to worry about those machines. Would hardly ever build a box to put into production unless it was something that was rather more specialized.
darkarmani 14y ago
The core of this article is that you get more performance for your money by building your own servers and racking them. I think we can all agree there. The problem with "hardware is cheap and programmers are expensive" is that your hardware will fail when you least expect it and have programmers sitting idle. Hardware is cheap, so have someone else assemble it and rack it. If you don't need it up all of the time, this is a great way to get a lot of performance -- IO and memory in particular.
halayli 14y ago
I always say that there got to be an ec2-like solution but with real hardware. Provisioning physical machines at the same convenience level as ec2 would be awesome. I ended up doing what Jeff did. Bought my own server and hosted it at HE for $75/month. It's xeon 5650 with 48GB Ram + 1TB disk for $2k. Assuming the machine will last 3 years, it's a $131/month. That's way cheaper than the closest that softlayer offers (https://www.softlayer.com/Sales/orderServer/41/2087/ https://www.softlayer.com/Sales/orderServer/41/2087/) Most of the times, EC2 is really about convenience and not cost.
- BryantD 14y ago
  Dell's Crowbar project: https://github.com/dellcloudedge/crowbar https://github.com/dellcloudedge/crowbar Ubuntu Orchestra: https://launchpad.net/orchestra https://launchpad.net/orchestra Or build your own around cobbler & puppet/chef, but you're sort of reinventing the wheel at that point. Still, sometimes that's fun.
  - donavanm 14y ago
    Also razor from emc/puppet. IMO a large scale cobbler config takes a bunch of work to get right. the provisioning and host management space is just starting to catch on to APIs and SOA. You're going to spend lots of time rolling your own apis, libs, services, and integration.
papsosouid 14y ago
Did nobody notice that his AWS figures are totally wrong? AWS is expensive as hell, but even still $1400/month for 3 instances immediately looked incredibly wrong to me. >The instance types included in the Web Application customer sample are 2 small (for the front end), and 1 large (for the database). Nope, the instance types are 2 small for web, 2 small for app, and 2 large for DB. That's fully double what he's claiming it is. And he's ignoring the 4 300GB EBS volumes that are in that $1400/month as well as the load balancer and 120GB of bandwidth. And that is entirely on-demand instances, if you are comparing to a colo setup, you should be using the much cheaper reserved instances.
- codinghorror 14y ago
  We're just comparing the Amazon Web Application template as provided in their calculator, with no changes. You're right that it is x2 for each EC2 server which I didn't notice until later, but that doesn't change the economics or performance story very much.
  - papsosouid 14y ago
    Yes, we're comparing the web application template as provided. Except you compared less than half of it instead of comparing it. I think "I claimed that AWS is twice as expensive as it actually is" certainly changes the economics much. Your 3 servers is not 6 servers and a load balancer. Reliability counts too, not just performance. Especially since most web apps are going to be 90% idle on the AWS hardware.
duggan 14y ago
Here is an incredibly lengthly argument which tries to be more balanced: http://rossduggan.ie/blog/infrastructure/cloud-vs-metal-infrastructure/ http://rossduggan.ie/blog/infrastructure/cloud-vs-metal-infr... Jeff is glossing over a lot. If you're already massively invested in hardware, in terms of people, processes and hardware, then you could argue that cloud architecture is less valuable in the general case, otherwise, it's usually no contest.
alberth 14y ago
Why is Jeff building a server when he can get the EXACT same server at Hetzner [1] for just 79 euro per month. That translates to literally the exact same 3 year cost as building the server. [1] http://www.hetzner.de/en/hosting/produktmatrix/rootserver-produktmatrix-ex http://www.hetzner.de/en/hosting/produktmatrix/rootserver-pr...

86 points by bussetta 14y ago

> But not the kind that go in your home. No, that'd be as nuts as the now-discontinued Windows Home Server product. What's wrong with home servers? I've been wanting to build one for quite some time.

ios84dev 14y ago
I think he means rack mount home servers, those are pretty nuts because of the noise that most rack mount cases make. Figure in a 1U case you normally have at least 4 15K RPM fans going at all times, even if you put it into "quiet" mode its enough to hear from the next room normally.
- timc3 14y ago
  My rack mount case is silent enough to record with a high quality condenser microphone in the same room. But it is 4U with 120mm slow fans. Its all about how you build them. Obviously a 1U with 40mm fans running fast could make a lot of noise.
zdw 14y ago
HP makes a "Microserver" that's designed for home/office use: http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-4237916-4237918-4237917-4248009.html?dnr=1 http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-42... Quiet, ECC RAM, and decent speed for most home-related tasks (backup, etc.)
- stordoff 14y ago
  I can highly recommend these. I recently got one of these for £100 (after cashback), and it is a brilliant piece of kit. I've got 4x1.5TB HDDs, and a 250GB OS drive, in mine, running Windows Server 2008 R2, and the performance is pretty good. 50MBps+ over a gigabit network is what I'm getting, which is more than enough for me.
burningion 14y ago
In lieu of building a home server, I highly recommend the Synology line of NAS machines. They've got everything you could possibly want out of a home server, with a much faster configuration. Automatic backups, a built in Torrent server, Plex media server, and remote file access all in one container. Seriously a great product and value if you're looking for a home server.
- randomchars 14y ago
  That looks really nice! Torrent server is one of the primary uses I would need this for so it's tempting.
- evandena 14y ago
  I don't want to come across as a comment crapper, but I've been recommending FreeNAS for years now. ZFS is perfect for home NAS purposes.
  - burningion 14y ago
    You're not comment crapping at all, you're offering a great alternative to those who are time rich / cash poor, or who like configuring and want to learn about building servers. For anyone who's cash rich and time poor, the NAS products from Synology are great.
  - deleted 14y ago
    [deleted]
- hkarthik 14y ago
  +1 I love my Synology NAS. Truly a great product.

mdgrech23 14y ago

Hardware is cheap, programmers are expensive. System admins even more so. For this reason I feel like cloud hosting providers are the way to go for bootstrappers or startups. Once you get big you can do things like FB and create your own datacenter. However the enjoyment and power that comes from having full control over your entire environment should never be over looked.

Detrus 14y ago
But it also costs programmer time to tweak software for weak hardware. Sometimes it makes sense to spend that time on setting up a few big iron servers yourself.
Muzza 14y ago
> System admins even more so. Really? Honest question.
- timc3 14y ago
  Depends on your business. If your small business is writing code, having a sysadmin around and not 100% busy they are could be costly compared to hiring another programer. But looking at general rates, I would say no.
- deleted 14y ago
  [deleted]
- thaumaturgy 14y ago
  Good sysadmins, yes. Keep in mind that the ideal sysadmin is someone who knows the guts of BSD or Linux (typically, not both) inside and out; can troubleshoot the strangest of problems very quickly; is constantly measuring and improving performance; and can hack together shell scripts or Python or even C code as needed. Linux and BSD are hellaciously complex and prone to very strange behavior once you start taking them into high-performance land. If you're OK settling with, "can read syslog and look stuff up on the web", those guys are cheap.
  - 14y ago

tobias3 14y ago

I myself like to do this. But only for home servers. I would never do that for a "serious" application. There are several reasons, the most important one being that this is a perfect example where outsourcing the work is both cheaper (economies of scale), more reliable (parts are tested better) and less risky (companies which do that at a larger scale have better risk management). If one the SSDs dies he has to drive there and switch it out, instead of one person simply doing that for the whole data center. This is simply inefficient. And of course Amazon hosting is more expensive. It is more flexible; you can spin up instances at your whim. You pay for that. It would be better to compare it with standard dedicated server hosting.

peterwwillis 14y ago
Have you worked with "professional support" in a colo before? When a hard drive dies under vendor support, one guy from the vendor drives out there to replace it. Half the time they don't test it after it's replaced unless you request it. Sometimes the parts are duds. Sometimes they bring the wrong part, or it doesn't fix whatever was broken on the server. If it's the datacenter's remote hands, it can be hours, and on rare occasions days, before somebody starts working on your issue or even answers the phone or e-mail. Same issues come, and you have to have your own spare hardware for the remote hands to use. It's highly variant based on the datacenter and vendor(s) and other issues. There's no guarantee that outsourcing will be reliable. You have to find the best one possible and build a good relationship.
- codinghorror 14y ago
  Having done this twice before, I strongly, STRONGLY recommend picking a colocation center that's less than an hour drive from someone who works for the company. Or you. Just in case, but that does come up from time to time and if it's many hours away (or worst of all an airplane ride) you will get hours/days to fix things as you noted.
  - peterwwillis 14y ago
    The shop I worked with, we had support on the machines, so Dell would send a tech out to replace a part for the servers that had valid support contracts. We still needed someone on the phone to remote-in to test the fix, of course. For more complicated problems, our main colo was about 45 mins away from almost all the SAs. We used remote hands for colos that were an airplane ride away, and had varying (read: sometimes really shitty) results.

vmind 14y ago

I much prefer the middle ground of dedicated servers over the hassle of colocation and hardware management. (16GB RAM quad core with 2TB raid1 from OVH for £65 a month are a good affordable level, and very quick and easy to spin up new ones).

girasquid 14y ago
If bang for your buck is a concern, you might be interested in this offering from Hetzner: http://www.hetzner.de/en/hosting/produkte_rootserver/ex4s http://www.hetzner.de/en/hosting/produkte_rootserver/ex4s - you get double the RAM and a bit more disk space for a little bit less. I only have a month's worth of experience to talk about it, but other than the fact that it's in Germany (timezones) it's been great.
- vmind 14y ago
  Yep, have used Hetzner servers too for some things and would use again, but for our customers being 90% in the US, and OVH offer a canadian location (and we may need some EU servers in the future).
  - girasquid 14y ago
    Fair enough. I'm always curious about why people choose one provider over another - thanks for filling me in on your reasoning. :)
henrikschroder 14y ago
I was also gonna come in here and mention the middle ground. Amazon is great either if you don't want to commit to anything, or if you have very spiky load and sometimes need to spin up a lot of servers. Building your own servers is dirt cheap, but there's always the risk of hardware breaking, and you can't exactly scale up very quickly. But the middle ground, rented dedicated servers, can be a perfect compromise for a lot of companies. Scaling up and adding servers might take a day instead of instantly, and it costs more than owning servers, but you get all the benefits of managed hosting, broken hardware simply isn't something you need to worry about anymore. If a server goes wonky, just scrap it and rent a new one.

stephengillie 14y ago

What happens when a HDD starts to go out? Do you have to pay & trust colo personnel to replace it, or do you have to pay to have your server shipped back to you? I could see putting my own server into a colo if it were like a storage locker - carry/cart your server into the building, mount it in the rack you've rented, plug into the provided power and network connections, and lock the security door.

patio11 14y ago
Typically you'd pay for the first option. The term of art is "remote hands" service. There are options where you can install servers in cages. (Some industries/companies require very strict physical access control.) This is more for show than because it meaningfully increases your security against the attack "your colocation provider is secretly The Adversary."
- peterwwillis 14y ago
  If you buy the hardware from a vendor and get their 24/7 support they'll send a tech to your cage to replace the drive. It's a little bit more expensive than the machine he built, but with the knowledge that someone will go fix it when it's broken. Worth the money, still ridiculously cheaper than AWS.
- jaf656s 14y ago
  >This is more for show than because it meaningfully increases your security against the attack "your colocation provider is secretly The Adversary." It does help protect you against the "another colo customer" attack, whether an actual attack or accident.
spindritf 14y ago
There's always a third option: go to the collocation facility and change the disk yourself. You won't usually be charged for access during business hours. Attractiveness of that solution depends greatly on the distance to the DC.
codinghorror 14y ago

gprasanth 14y ago

Network connectivity is one very big problem. What good is a blazing fast server with 2Mbps (disruptive!) link? This is the very reason a lot of people would hate their own servers. Cloud = connectivity + (computing power) Yay! for connectivity.

codinghorror 14y ago
Of course it depends on what you're doing. Running a porn site or image/video sharing service? You're going to go through a ton of bandwidth and that might end up being your bottleneck way more than performance.
- dholowiski 14y ago
  > Of course it depends on what you're doing. This sums up every comment here. Every web app is a unique snowflake and what works for you (EC2/Dedicated, SSD, RAM bandwidth) is just as unique. Much more important than building your own server, is fully understanding your requirements, and deploying the appropriate solution. But it is still wicked fun to build a server.

jaequery 14y ago

i've known this secret for a while. i went from paying $4k+/month spanning over 10+ servers in amazon ec2, to just 2 dedicated servers < $1k/month in an unlimited 1gig colo. the performance difference i see is huge. i never realized how actually slow EC2 is. their issue is definitely in slow IO, which i think only their SSD instances can fix (which will cost me in the $8k+). i now have a wicked setup, Xenserver Cloud, SAN, all Highly Available, don't have to worry about bandwidth overcharges, and much much faster .

codinghorror 14y ago
Particularly now that we have 6 GBps SATA and cheap(ish) 512 GB SSD drives, the I/O differential can be enormous. I thought about putting four 512 GB SSDs drives in a Raid 10, which would give me striping performance levels without losing the mirror, but that seemed like a bit too much overkill. These servers have 4 front drive bays, only 2 are filled with the SSD mirror, so we could decide to drop in 2 more drives and rebuild the array if we need even more I/O perf.

btgeekboy 14y ago

I like the part where he builds new servers on a rug. What's the over/under for how long until parts start failing?

codinghorror 14y ago
I have built many dozens of computers over the last 20 years and I've never ever seen any static-related problems. I don't wear a static wrist strap, but I do always touch something metal on the case before touching the internals.
- ersii 14y ago
  I've been saying this as well, but has realised I don't know what problems static electricity could or would cause - long-term nor short-term. I've stopped saying that I've never ever seen any static-related problems and I'm using the silly bracelet connected to something metal these days.
  - tlb 14y ago
    Static does cause damage other than immediate catastrophic failure. It can degrade silicon in subtle ways, resulting in random errors or crashes or even slowness due to retrying transactions. Wearing the wrist strap is cheap insurance against mysterious badness.
    - codinghorror 14y ago
      Yeah, but it's also a little bit voodoo. Unless we can point to specific instances of things going wrong, it's kind of imagineering a problem where one does not actually exist. That said I always advise touching something metal on the exterior case before touching anything in a computer, and that's how I have always done it.
      - marshray 14y ago
        Static damage to a chip may not result in an immediate failure. It may manifest as lessened performance, unreliability, or a shortened lifespan. I've seen this demonstrated in training videos for the electronics industry. (I think EEVblog has a static demo up on YouTube). The semiconductor industry (i.e., the companies like Intel that made the parts on your circuit boards) spends many millions, if not billions, of $ a year on static-protective mitigations. They're pretty smart folks and also very cost-conscious. They would not be spending all this money if it were pure voodoo. However, parts mounted on a circuit board are much more resilient than loose chips. I, too, use the "touching metal" method but am very careful about it.

JoeCortopassi 14y ago

"...you don't need the redundancy, geographical backup, and flexibility that comes with cloud virtualization" This is perhaps the single most glossed over topic in the entire article. If I am a 1-5 person shop, maintaining a web app, virtualized hosting pays huge dividends in that I don't even notice if a hard drive or motherboard takes a dump. There are additional costs that come with the benefit of being abstracted away from hardware failure or geographic problems (building fire, power out, etc), and that's something that every business has to evaluate for itself.

codinghorror 14y ago
Well, you'd always colocate enough servers so that you can lose at least one machine without caring. E.g. HAProxy to 2 web tier machines on the back end. HAProxy will fail over to just one server no problem. (And yes, you can heartbeat so you have two cheap physical HAProxy machines, too. This gets into sub-blade territory, where the 1U server is internally two or more complete low-ish power servers with independent power supplies, etc. ) That's the whole premise behind FOSS, you don't need to worry about all the licenses, and the hardware is so cheap it is effectively free and getting freer by the second, so you throw a lot of cheap hardware at the problem. But agreed, "lot" in this case means at least two so one can fail and you don't need to care.
- JoeCortopassi 14y ago
  I guess my bigger point though, would be that when using something like AWS, I don't have to think/spend time on a lot of implementation details. When using colocation services/hardware/failover, I'm just adding a bunch of little things to my daily tasks and responsibilities. Sometimes this is a big deal (like in a bootstrapped two man team) and other times it's not. Ultimately, I think it comes down to priority instead of possibility. If your company lives and dies on having reliable servers, you should probably roll your own. But if servers are 'just' a technical detail to your overall business model, then a cloud solution can be well worth the additional cost
  - BryantD 14y ago

abtinf 14y ago

That article is not just comparing apples to oranges; with that default aws setup, he is comparing apples to a herd of donkeys. The correct comparison is his server vs a single EC2 High-Memory Double Extra Large instance with a 3 year heavy-utilization reservation. This instance costs $3100 upfront plus $0.14/hour. The total 3 year cost for this server on AWS would be 3100 + (.14 * 24 * 365 * 3) = 6779.2, or about $188.31 per month. Sure, its more expensive. But AWS provides an insane of value on top of the server. Like instantly being able to provision additional capacity. I wouldn't be at all surprised if, on a full-loaded cost basis, it is extremely competitive with building his server. Heck, the employee salary expense of building your own server will easily drive the cost of the server well beyond the $3100 up front amazon fee. I love building hardware too (never had a computer I didn't build except for laptops). But my mind boggles at AWS value proposition

codinghorror 14y ago
Wait, so you can basically buy a server at AWS EC2 for a term of 3 years? That's what this heavy utilization reservation sounds an awful lot like.
- deleted 14y ago
  [deleted]
- recuter 14y ago
  Not only that, now that they have that marketplace going you can pawn off your contract unto somebody else if your needs change. I'm not sure how liquid it is yet. You could also equally go dumpster diving for unused virtual instances, say, if some startup goes under and liquidates.
- twelve45 14y ago
  Yeah for the closest possible comparison in your scenario, you want to do a 3 year reserved with heavy utilization. When you reserve an instance, you are committing to a higher upfront in order to get a lower hourly for the reservation period. The low/medium/high utilization is sort of a knob that allows you to further control this upfront vs. hourly cost. With high utilization you have the highest upfront, but also the lowest hourly cost. If you are planning to run the server 24x7, this will also give you the lowest total cost.

rdw 14y ago

Where's he colocating those servers? Last time I dabbled with colocation, the bandwidth costs per-server were by far the dominant cost. I found it difficult to get anything at all for small quantities of servers. Looking at he.net at the moment, I see they have a deal for $1/Mbps. Presumably someone like Jeff Atwood can get twice as good a deal as that, so he'd pay around $500/month for bandwidth for those servers. Going by the cheat sheet (https://blog.cloudvertical.com/2012/10/aws-cost-cheat-sheet-2/ https://blog.cloudvertical.com/2012/10/aws-cost-cheat-sheet-...), that is within a factor of 2 of a yearly-reserved h1.4xlarge ($2263/month * 0.47 savings = $1199). It's almost equal to the three-year reserved machine ($2263 * 0.30 = $678). Edit: He probably only needs 1 Gbps for all 4 machines, driving his bandwidth costs down by a factor of 4, but we could start to take power/space/cooling costs into account at that point. So is he getting a better deal than that? I'd love to know where.

aes256 14y ago
Maybe I've been out of the game too long, but $1/Mbps sounds very low. Last time I looked into it (one or two years ago) you could only get bargain bucket bandwidth approaching those prices, with fairly crumby providers. I'm assuming Amazon are a cut above that.
- rdw 14y ago
  Yeah, $1/Mbps sounds like a steal to me as well, which may be why it's a limited-time special offer. thatamurgy's Pinboard links lead to this hosting provider: http://prgmr.com/san-jose-co-location.html http://prgmr.com/san-jose-co-location.html The machines in the original post being ~250W, they'd need the $362.50/month plan. But it's only at 10Mbps, which would only work if the application has low network requirements. Maciej appears to have gotten a much better deal, "For $206, I got 10U, 4 amps, and 100 Mbps connectivity", perhaps in exchange for promoting it through his blog. Everything's easier when you're a celebrity!
codinghorror 14y ago
A lot of colocation centers offer "unlimited" bandwidth, though I strongly suspect that may depend on whether you're running a porn site / megaupload clone or not. :) Fortunately bandwidth is one thing that has gotten substantially cheaper over time: http://www.codinghorror.com/blog/2007/02/the-economics-of-bandwidth.html http://www.codinghorror.com/blog/2007/02/the-economics-of-ba...

dholowiski 14y ago

I love building servers, it is amazingly geeky fun. And no doubt colocation with your own servers will give you the best bang for your buck, with renting a dedicated server coming in second. But you do have to be careful about redundancy. If you can't do without a day or two down-time, you'll need 2 or more servers, because when it's a dedicated server it's all up to you (or to someone you hire/pay for) when something fails. Sadly I live a 5 hour drive from the nearest co-location facility, so I'm forced to rent a dedicated server.

codinghorror 14y ago
Yeah I would never advise colocating just one server. Two at minimum. We'll have five when all is said and done (still building the other four here).

thaumaturgy 14y ago

maciej from pinboard has written some useful stuff on this too: - The five stages of hosting: http://blog.pinboard.in/2012/01/the_five_stages_of_hosting/ http://blog.pinboard.in/2012/01/the_five_stages_of_hosting/ - Building servers: http://blog.pinboard.in/2012/05/a_cloud_of_my_own/ http://blog.pinboard.in/2012/05/a_cloud_of_my_own/ - Going colo: http://blog.pinboard.in/2012/06/going_colo/ http://blog.pinboard.in/2012/06/going_colo/ A bunch of people are probably going to respond to Jeff's article by saying things like, "But VPS hosting means you get other people to deal with problems for you", but in reality all that means is that you're at maciej's "monastery" or "dorm room" stage of hosting, and your needs haven't yet driven you to get "the apartment".

codinghorror 14y ago
Maciej is 105% awesome, I'm a huge fan, and I linked several of his articles in the actual text of my post already. Go read them all!
mmariani 14y ago
The five stages of hosting is priceless! I'd really like to go with #5, but I'll stick to #3 for now. :)
wiredfool 14y ago
I love maciej's footnote: """* What is it with these aggro facility names? Rather than Hurricane Electric or Raging Wire, I would much prefer to host with "Calm Multihomed Oasis" or "Granite Mountain" or " Cooling Breezes Pure Sine Wave Mount Bitmore". """

timc3 14y ago

Happy rackmount home server user here, running ESXi with a load of dev instances, staging servers, OpenIndiana for ZFS running decent amount of storage connected to my editing workstations, databases, plus looks after our family video and photos. I still use our company servers that are co-located but having things at home on the same network where you are developing is very compelling for me. Its very quiet and cool (latest i7 processor). But if you are hosting your own production boxes, I usually buy from HP or Dell and know that I rarely need to worry about those machines. Would hardly ever build a box to put into production unless it was something that was rather more specialized.

darkarmani 14y ago

The core of this article is that you get more performance for your money by building your own servers and racking them. I think we can all agree there. The problem with "hardware is cheap and programmers are expensive" is that your hardware will fail when you least expect it and have programmers sitting idle. Hardware is cheap, so have someone else assemble it and rack it. If you don't need it up all of the time, this is a great way to get a lot of performance -- IO and memory in particular.

halayli 14y ago

I always say that there got to be an ec2-like solution but with real hardware. Provisioning physical machines at the same convenience level as ec2 would be awesome. I ended up doing what Jeff did. Bought my own server and hosted it at HE for $75/month. It's xeon 5650 with 48GB Ram + 1TB disk for $2k. Assuming the machine will last 3 years, it's a $131/month. That's way cheaper than the closest that softlayer offers (https://www.softlayer.com/Sales/orderServer/41/2087/ https://www.softlayer.com/Sales/orderServer/41/2087/) Most of the times, EC2 is really about convenience and not cost.

BryantD 14y ago
Dell's Crowbar project: https://github.com/dellcloudedge/crowbar https://github.com/dellcloudedge/crowbar Ubuntu Orchestra: https://launchpad.net/orchestra https://launchpad.net/orchestra Or build your own around cobbler & puppet/chef, but you're sort of reinventing the wheel at that point. Still, sometimes that's fun.
- donavanm 14y ago
  Also razor from emc/puppet. IMO a large scale cobbler config takes a bunch of work to get right. the provisioning and host management space is just starting to catch on to APIs and SOA. You're going to spend lots of time rolling your own apis, libs, services, and integration.

papsosouid 14y ago

Did nobody notice that his AWS figures are totally wrong? AWS is expensive as hell, but even still $1400/month for 3 instances immediately looked incredibly wrong to me. >The instance types included in the Web Application customer sample are 2 small (for the front end), and 1 large (for the database). Nope, the instance types are 2 small for web, 2 small for app, and 2 large for DB. That's fully double what he's claiming it is. And he's ignoring the 4 300GB EBS volumes that are in that $1400/month as well as the load balancer and 120GB of bandwidth. And that is entirely on-demand instances, if you are comparing to a colo setup, you should be using the much cheaper reserved instances.

codinghorror 14y ago
We're just comparing the Amazon Web Application template as provided in their calculator, with no changes. You're right that it is x2 for each EC2 server which I didn't notice until later, but that doesn't change the economics or performance story very much.
- papsosouid 14y ago
  Yes, we're comparing the web application template as provided. Except you compared less than half of it instead of comparing it. I think "I claimed that AWS is twice as expensive as it actually is" certainly changes the economics much. Your 3 servers is not 6 servers and a load balancer. Reliability counts too, not just performance. Especially since most web apps are going to be 90% idle on the AWS hardware.

duggan 14y ago

Here is an incredibly lengthly argument which tries to be more balanced: http://rossduggan.ie/blog/infrastructure/cloud-vs-metal-infrastructure/ http://rossduggan.ie/blog/infrastructure/cloud-vs-metal-infr... Jeff is glossing over a lot. If you're already massively invested in hardware, in terms of people, processes and hardware, then you could argue that cloud architecture is less valuable in the general case, otherwise, it's usually no contest.

alberth 14y ago

Why is Jeff building a server when he can get the EXACT same server at Hetzner [1] for just 79 euro per month. That translates to literally the exact same 3 year cost as building the server. [1] http://www.hetzner.de/en/hosting/produktmatrix/rootserver-produktmatrix-ex http://www.hetzner.de/en/hosting/produktmatrix/rootserver-pr...