8 ms·

I'd say don't even consider colo unless you have a specific use case. Rented dedicated servers are cheap, let someone else take care of the hardware.

by xuki 5y ago

manigandham 5y ago
True, there's a wide spectrum in the middle from rented colo to managed servers.
bsenftner 5y ago
Yes, they are cheap. Running one's own server is also easy peasy; far too many think it is difficult, it is not. The most expensive part is the electricity.
- maccard 5y ago
  > The most expensive part is the electricity. No, the most expensive part is the persons time for managing it. I can rent a monstrous Dedicated Server for $400/month from OVH, but even with a UK salary, if I have to spend more than 1 day a month on it in any shape or form (and that includes the initial setup), it's cheaper to use "the cloud" or some form of a managed service.
  - vidarh 5y ago
    If you have to use more than one a day a month managing the server, you are doing something wrong. I used to manage multiple racks worth of servers on top of managing the 1k containers running on them, maintaining the (pre-kubernetes) orchestration software I had written to deploy containers to our servers, and still had time left over to spend the majority of my time on the architecture and project management of new projects for clients.
    - treis 5y ago
      I'd say if you're not spending one day a month you're doing something wrong. Namely, you're not testing your back ups and disaster recovery often enough.
      - brimble 5y ago
        You're right, but given the context of the thread, it's worth pointing out that's not extra work that one doesn't also need to do with cloud hosting. Ought to test your backups and disaster recovery regularly, either way.
      - vidarh 5y ago
        That's not managing the server. That's managing your application setup and not something that takes additional time for managed servers over cloud setups, because you still need all of those things for a cloud setup. It also should not take anywhere near a day per server per month - if it does, then in a disaster scenario it means you're unable to recover at a reasonable pace.
        treis 5y ago
        >That's not managing the server. That's managing your application setup and not something that takes additional time for managed servers over cloud setups, because you still need all of those things for a cloud setup. It is additional time because the cloud handles the whole class of "your hardware died" problems.
        vidarh 5y ago
        Most decent bare metal setups abstracts away the same class of "your hardware died" problems. E.g. my first rule on this is everything runs in containers whether I run in a cloud environment or on bare metal. The OS image is identical, and include tying into service discovery and a suitable orchestration mechanism (which can range from something trivially simple to, say, Kubernetes). Any modern server hardware has IPMI or something equivalent, which means you plug it in, configure the IPMI, configure network boot from a tftp/bootp or similar server holding your installation image, and from there on out you're deploying containers the same way as you would in a cloud environment, and back them up and arrange for failover the same way as in a cloud environment. I've set up more than one hybrid setup where you didn't need to know if your workload was running in AWS or Hetzner or somewhere else, so we could use AWS for elasticity and Hetzner to keep cost down. This is not a hard problem. And if people don't have the right skills in house, it's easy to outsource this (I for one used to make my living of automating setups like this and operating them on a retainer basis).
    - maccard 5y ago
      Broadly speaking, managing 100 "cloud" servers is roughly the same amount of work as managing 2. Moving from managing 2 cloud servers to 100 is trivial. Moving from 1 server to 2 is an architecture problem. That architecture problem sometimes comes up when it comes to scaling too. it's the cattle vs pets problem. The last thing you want to find out is that someone ssh'ed in and installed a package that's required for your service to run, _during_ vertical resizing, and that sort of thing is far more common in systems with 1 server instead of many. If you're running containers, you would likely have even less overhead (and save money) by running on ECS/DigitalOcean/Azure App Service, and if you're running a big old monolith, you likely need > 1 instance for some sort of redundancy anyway.
      - vidarh 5y ago
        > Broadly speaking, managing 100 "cloud" servers is roughly the same amount of work as managing 2 If this was the case, my billable hours when I was doing contracting would be about 1/10th of what they were. I lived very comfortably of troubleshooting for teams who had gotten themselves into a thorough mess with this attitude. In fact, I earned more from the teams who insisted on cloud setups because they rarely understood the operational issues with it, whereas teams who chose dedicated servers generally thought about operational concerns more. > Moving from 1 server to 2 is an architecture problem. Moving from 1 container to 2 is an architecture problem. Moving from 1 server to 2 running those containers is an architecture problem with well established known solutions. E.g. for starters you're assuming no containers. But putting the application in containers and leaving the host OS only for basic infrastructure setup is basic practice today if you're running your own servers. Once you've set up a network boot source (tftp etc.) to network boot of an installer with a suitable setup script that you can trigger via IPMI, and a directory service and a basic orchestrator (be it Kubernetes or something else) on your network, it doesn't matter much if you have 1 server or a 100 - they come up and you put containers on them, and they look no different than a cloud service to the devs. You're right that it's easier to take shortcuts if you have just a few servers, but it's just as easy to take shortcuts with just a few cloud instances - the number of pet containers I've seen over the years is terrifying. > The last thing you want to find out is that someone ssh'ed in and installed a package that's required for your service to run Which is why you don't provide ssh access to the host servers to anyone without an understanding of ops concerns, log everything, and do all updates via an automated setup of your preference, and why you regularly recycle the containers whether you run a cloud environment or dedicated servers. The reality is that cloud systems do not at all make you immune to this - I've done year long projects to regularise AWS setups that were full of undocumented manual changes to bring everything into a terraform config for example. Often they're worse, because there are a whole lot of unobvious places to look for extra bits and pieces. > and if you're running a big old monolith, you likely need > 1 instance for some sort of redundancy anyway. Nobody here suggested a monolith. Nobody is suggesting you forgo redundancy. The. The point of this is that often you can put clear upper limits on the computation you will need to do for either your system as a whole or for a given subsystem, and you can guarantee that you will never need more than one server for a given part of the system. E.g. a real example: I've worked on a system that did some processing of data about companies. We know this will always fit on a single system because the population growth of humanity is slower than the performance growth of a server and the total number of companies worldwide fits on a single system with several magnitudes to spare today, and the type of companies we were interested in is just a subset. At this point, if you architect a system like that on the basis of assuming you will need to resize, you will risk making choices that makes you far more likely to have to resize. E.g. all of the data I'm talking about can easily fit in RAM on a relatively moderate server now and forever, but the moment you start planning for partitioning the data you have added orders of magnitude of performance overhead for communication. Properly assessing which parts of a system needs to be able to scale is at the core of architecture, and architects (or devs; it's terrifying how many places lack anyone with architecture experience) who are just planning for infinite scalability for everything is huge red flag to me. It usually means they don't understand their system. For a startup, especially, this is an existential question - preparing for unnecessary scaling has killed many startups.
  - tiew9Vii 5y ago
    I hear that said a lot. I don’t think it stands. A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t. The cloud actually requires more investment for large organisations. Previously you might of only had a handful of sysadmins but now you have a large dedicated platform team doing devops type work building your own abstractions/paas on top of your cloud. The advantage the cloud has is flexibility. You don’t need to go through a lengthy process to acquire hardware. Likewise cloud services are disposable, no longer need something? Hit the delete button. I think the more PaaS like services such as Heroku, Lambda, Fargate, Google Cloud Run do better realise the less maintenance story but not cloud generally.
    - deleted 5y ago
      [deleted]
    - maccard 5y ago
      > I think the more PaaS like services such as Heroku, Lambda, Fargate, Google Cloud Run do better realise the less maintenance story but not cloud generally. Completely agree here. Running an EC2 Spot instance + ebs volumes for 730 hours a month is a total waste of money, but running RDS behind fargate and ECS with an ALB is likely to save you time and money by month 2.
    - stephbu 5y ago
      I broadly agree with the statement that owning an OS in general is toil that applies any of these IaaS/VPS/BM scenarios. Owning a BM server different toil - server parts fail, the network it attaches to needs control and it fails too, firmware needs updating more regularly than ever, DC space needs managing over time etc. For a small number of machines maybe this is NBD. For thousands of machines this is just grunt work which while automated, still needs change management and control - rebooting the whole fleet in the middle of the day definitely opens doors in your career. Doing everything you did in the DC in the Cloud is absolutely the worst way to adopt Cloud. Owning an OS is a non-goal, you’ve gotta climb to a higher abstraction - workloads, and quit caring about machines. This is where most companies fail.
    - 5y ago
  - whiplash451 5y ago
    I think that depends a lot on which stage your startup is at, what your level of funding is, etc. Definitely not as clear-cut as you seem to imply.
- koolba 5y ago
  > The most expensive part is the electricity. There’s more than a kWh rate for that though. There’s battery backup power, diesel backup, and failover hot switching. It’s not like just plugging in your phone to charge.
- m348e912 5y ago
  Easy or not, it's a job. Configuring, patching, upgrading, troubleshooting, securing, monitoring. It's only not expensive if your time is worth nothing.
  - ykevinator2 5y ago
  - greedo 5y ago
    And you still need to do this with an ec2 instance. Sure you can automate a lot of it via config mgmt/orchestration, but unless you're using serverless stuff on AWS, it's an awful lot like having your own VMs.
- stephbu 5y ago
  It’s pretty hard to generalize this without qualifiers. Electricity can be the most expensive problem, but it requires carefully planned control of the other factors - e.g large scale with highly automated servers, network, and meat-reducing control planes to become true. Otherwise factors such as people and under-utilization can especially dominate smaller and/or less efficient facilities. Thinking thru the factors, many seem obvious, but are often forgotten/ignored when comparing rental or IaaS costs.: Space is a fixed cost driven by market rates and maximum Server Capital Costs i.e. floor space. Failing to fill the room increases your cost/server efficiency. Pretty common to run out of thermal/power before you run out space, as equipment efficiency increases through the lifespan of the facility. Server and Power costs scale together, carry a minimum cost for keeping machines on, and vary based on utilization. Again if the servers aren’t doing work, your efficiency ratio will drop. Larger space typically have pre negotiated power commitments too - failing to consume that carries fiscal penalties. Servers unit costs are fairly cheap, storage not so much. Full utilization throughout capital/lease lifespan is the goal - anything less increases relative cost/server. Network costs scale with Server Costs, and vary again by utilization - minimum invest rules apply, all servers need at least one network port, as well as upstream Core/TOR/Miniswitch gear. The network gear lifespan is typically longer than servers, but shorter than facilities. It usually incurs annual support/maintenance charges too. Bandwidth charges are variable as expected. People costs scale with a step-function and numbers driven by minimum coverage requirements, task complexities, and level of human toil. Performing any task on a device by-hand is expensive in most markets - touches on tickets, change management, task time etc. Fully burdened S+R in Western cultures is typically ~2x the salary - a $80K employee probably costs around $150K by the time all the workplace costs, taxes, and benefits are paid. Network folk are typically premium resources compared to DC Ops. Sustainable 24x7 coverage looks like a staff of 3-4 people.
  - vidarh 5y ago
    The thing is, you can buy your way out of most of these considerations by either renting colo space or renting managed servers until/unless you're at a scale where there are savings to doing it yourself. This is a commodity service with margins a tiny fraction of AWS' margins. Buying services at those levels instead of doing hosting on prem still nets you 80%-90% of the savings vs. cloud. Sometimes it nets you greater savings because it makes it easier to move your workloads to cheaper locations (e.g. for a small company in London operating servers somewhere with low energy prices and low property prices is hard; colo space reachable from London in often 30%-50% above what it can be like many other places in Europe, and so you'll find managed servers from providers like Hetzner is often cheaper than on prem if your company is in a location like that)
- briffle 5y ago
  Been a sysadmin for many years. Your right, for a few computers. Once you start getting into more than a quarter rack, you also need to start worrying about cooling (which also is lots of electricity) and usually things like ensuring the electricity stays on. (UPS, generator, etc). Don't forget to monitor and service all this stuff regularly. Once your past a few racks of equipment, you have generator tests and service appointments, redundant AC, Redundant UPS, dual power to each rack, etc. Dual Internet connection, and a link to your other server room that you use for DR, etc. The costs and complexity quickly escalate after a server or two.
  - dhzhzjsbevs 5y ago
    In the bad old days we bought our own servers and rented rack space in a datacenter they took care of all the hardware management, we looked after the software. Mind you this was only about 4U worth.
  - BryanRamirez 5y ago
    (I created an account just to upvote and comment on this) Absolutely, nevermind managing OS upgrades and needing to use configuration management to mitigate against drift. It took a lot of effort to make sure that each server was not a special snowflake that could not be reliably reproduced. Also, dealing with vendor warranties, and being on hold with HP (or whoever), then assuring them you're running the latest firmware.. please for the love of god just replace the failed memory/disk/cpu. I found the sheer physicality of computing infrastructure to be a source of exhaustion and burnout. It's a big part of the reason why I'm a software developer now :)
  - secabeen 5y ago
    > Once your past a few racks of equipment, you have generator tests and service appointments, redundant AC, Redundant UPS, dual power to each rack, etc. Dual Internet connection, and a link to your other server room that you use for DR, etc. The costs and complexity quickly escalate after a server or two. This is not always the case. For computationally-focused workloads like the OP describes, without direct customer interaction, it may be reasonable to accept risk of downtime in the event of failure. If you are doing computations that take weeks to complete, and you checkpoint regularly, does it really matter if your computation finishes on Saturday morning or Monday morning? If not, you can probably accept 6 hours of downtime once every couple of years and eliminate all of the redundancy overhead described above. In HPC, the general rule of thumb is buy your hardware if you can be sure you'll run compute on it more than ~40% of the time.
stephbu 5y ago
To add to this statement - don’t make any investment unless you have a solid plan to fully utilize it for the full length of the purchase terms. This applies as much to AWS @ minute terms, as it does to a rental @ daily terms, as it does to co-lo capital investment @ yearly terms.
- marcosdumay 5y ago
  > unless you have a solid plan to fully utilize it for the full length of the purchase Hum... I'd say it's much more reasonable to look at the ROI. Making investments to supply peak demand or to hedge against rare risks is perfectly ok.
  - stephbu 5y ago
    Fair point, I guess the point is that unallocated resources - space/power/servers etc. can become huge stealth money sinks, eating budget every hour of every day. Being cognizant of the consumption economics before you stump up for resources is important, as is fitting the investment model to match those economics. Setting utilization/allocation targets are just one way of measuring if those models efficiently match. This is true for any service or resource consumed.

I'd say don't even consider colo unless you have a specific use case. Rented dedicated servers are cheap, let someone else take care of the hardware.

by xuki 5y ago

I'd say don't even consider colo unless you have a specific use case. Rented dedicated servers are cheap, let someone else take care of the hardware.

manigandham 5y ago

True, there's a wide spectrum in the middle from rented colo to managed servers.

bsenftner 5y ago

Yes, they are cheap. Running one's own server is also easy peasy; far too many think it is difficult, it is not. The most expensive part is the electricity.

maccard 5y ago
> The most expensive part is the electricity. No, the most expensive part is the persons time for managing it. I can rent a monstrous Dedicated Server for $400/month from OVH, but even with a UK salary, if I have to spend more than 1 day a month on it in any shape or form (and that includes the initial setup), it's cheaper to use "the cloud" or some form of a managed service.
- vidarh 5y ago
  If you have to use more than one a day a month managing the server, you are doing something wrong. I used to manage multiple racks worth of servers on top of managing the 1k containers running on them, maintaining the (pre-kubernetes) orchestration software I had written to deploy containers to our servers, and still had time left over to spend the majority of my time on the architecture and project management of new projects for clients.
  - treis 5y ago
    I'd say if you're not spending one day a month you're doing something wrong. Namely, you're not testing your back ups and disaster recovery often enough.
    - brimble 5y ago
      You're right, but given the context of the thread, it's worth pointing out that's not extra work that one doesn't also need to do with cloud hosting. Ought to test your backups and disaster recovery regularly, either way.
    - vidarh 5y ago
      That's not managing the server. That's managing your application setup and not something that takes additional time for managed servers over cloud setups, because you still need all of those things for a cloud setup. It also should not take anywhere near a day per server per month - if it does, then in a disaster scenario it means you're unable to recover at a reasonable pace.
      - treis 5y ago
        >That's not managing the server. That's managing your application setup and not something that takes additional time for managed servers over cloud setups, because you still need all of those things for a cloud setup. It is additional time because the cloud handles the whole class of "your hardware died" problems.
        vidarh 5y ago
        Most decent bare metal setups abstracts away the same class of "your hardware died" problems. E.g. my first rule on this is everything runs in containers whether I run in a cloud environment or on bare metal. The OS image is identical, and include tying into service discovery and a suitable orchestration mechanism (which can range from something trivially simple to, say, Kubernetes). Any modern server hardware has IPMI or something equivalent, which means you plug it in, configure the IPMI, configure network boot from a tftp/bootp or similar server holding your installation image, and from there on out you're deploying containers the same way as you would in a cloud environment, and back them up and arrange for failover the same way as in a cloud environment. I've set up more than one hybrid setup where you didn't need to know if your workload was running in AWS or Hetzner or somewhere else, so we could use AWS for elasticity and Hetzner to keep cost down. This is not a hard problem. And if people don't have the right skills in house, it's easy to outsource this (I for one used to make my living of automating setups like this and operating them on a retainer basis).
  - maccard 5y ago
    Broadly speaking, managing 100 "cloud" servers is roughly the same amount of work as managing 2. Moving from managing 2 cloud servers to 100 is trivial. Moving from 1 server to 2 is an architecture problem. That architecture problem sometimes comes up when it comes to scaling too. it's the cattle vs pets problem. The last thing you want to find out is that someone ssh'ed in and installed a package that's required for your service to run, _during_ vertical resizing, and that sort of thing is far more common in systems with 1 server instead of many. If you're running containers, you would likely have even less overhead (and save money) by running on ECS/DigitalOcean/Azure App Service, and if you're running a big old monolith, you likely need > 1 instance for some sort of redundancy anyway.
    - vidarh 5y ago
      > Broadly speaking, managing 100 "cloud" servers is roughly the same amount of work as managing 2 If this was the case, my billable hours when I was doing contracting would be about 1/10th of what they were. I lived very comfortably of troubleshooting for teams who had gotten themselves into a thorough mess with this attitude. In fact, I earned more from the teams who insisted on cloud setups because they rarely understood the operational issues with it, whereas teams who chose dedicated servers generally thought about operational concerns more. > Moving from 1 server to 2 is an architecture problem. Moving from 1 container to 2 is an architecture problem. Moving from 1 server to 2 running those containers is an architecture problem with well established known solutions. E.g. for starters you're assuming no containers. But putting the application in containers and leaving the host OS only for basic infrastructure setup is basic practice today if you're running your own servers. Once you've set up a network boot source (tftp etc.) to network boot of an installer with a suitable setup script that you can trigger via IPMI, and a directory service and a basic orchestrator (be it Kubernetes or something else) on your network, it doesn't matter much if you have 1 server or a 100 - they come up and you put containers on them, and they look no different than a cloud service to the devs. You're right that it's easier to take shortcuts if you have just a few servers, but it's just as easy to take shortcuts with just a few cloud instances - the number of pet containers I've seen over the years is terrifying. > The last thing you want to find out is that someone ssh'ed in and installed a package that's required for your service to run Which is why you don't provide ssh access to the host servers to anyone without an understanding of ops concerns, log everything, and do all updates via an automated setup of your preference, and why you regularly recycle the containers whether you run a cloud environment or dedicated servers. The reality is that cloud systems do not at all make you immune to this - I've done year long projects to regularise AWS setups that were full of undocumented manual changes to bring everything into a terraform config for example. Often they're worse, because there are a whole lot of unobvious places to look for extra bits and pieces. > and if you're running a big old monolith, you likely need > 1 instance for some sort of redundancy anyway. Nobody here suggested a monolith. Nobody is suggesting you forgo redundancy. The. The point of this is that often you can put clear upper limits on the computation you will need to do for either your system as a whole or for a given subsystem, and you can guarantee that you will never need more than one server for a given part of the system. E.g. a real example: I've worked on a system that did some processing of data about companies. We know this will always fit on a single system because the population growth of humanity is slower than the performance growth of a server and the total number of companies worldwide fits on a single system with several magnitudes to spare today, and the type of companies we were interested in is just a subset. At this point, if you architect a system like that on the basis of assuming you will need to resize, you will risk making choices that makes you far more likely to have to resize. E.g. all of the data I'm talking about can easily fit in RAM on a relatively moderate server now and forever, but the moment you start planning for partitioning the data you have added orders of magnitude of performance overhead for communication. Properly assessing which parts of a system needs to be able to scale is at the core of architecture, and architects (or devs; it's terrifying how many places lack anyone with architecture experience) who are just planning for infinite scalability for everything is huge red flag to me. It usually means they don't understand their system. For a startup, especially, this is an existential question - preparing for unnecessary scaling has killed many startups.
- tiew9Vii 5y ago
  I hear that said a lot. I don’t think it stands. A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t. The cloud actually requires more investment for large organisations. Previously you might of only had a handful of sysadmins but now you have a large dedicated platform team doing devops type work building your own abstractions/paas on top of your cloud. The advantage the cloud has is flexibility. You don’t need to go through a lengthy process to acquire hardware. Likewise cloud services are disposable, no longer need something? Hit the delete button. I think the more PaaS like services such as Heroku, Lambda, Fargate, Google Cloud Run do better realise the less maintenance story but not cloud generally.
  - deleted 5y ago
    [deleted]
  - maccard 5y ago
    > I think the more PaaS like services such as Heroku, Lambda, Fargate, Google Cloud Run do better realise the less maintenance story but not cloud generally. Completely agree here. Running an EC2 Spot instance + ebs volumes for 730 hours a month is a total waste of money, but running RDS behind fargate and ECS with an ALB is likely to save you time and money by month 2.
  - stephbu 5y ago
    I broadly agree with the statement that owning an OS in general is toil that applies any of these IaaS/VPS/BM scenarios. Owning a BM server different toil - server parts fail, the network it attaches to needs control and it fails too, firmware needs updating more regularly than ever, DC space needs managing over time etc. For a small number of machines maybe this is NBD. For thousands of machines this is just grunt work which while automated, still needs change management and control - rebooting the whole fleet in the middle of the day definitely opens doors in your career. Doing everything you did in the DC in the Cloud is absolutely the worst way to adopt Cloud. Owning an OS is a non-goal, you’ve gotta climb to a higher abstraction - workloads, and quit caring about machines. This is where most companies fail.
  - 5y ago
- whiplash451 5y ago
  I think that depends a lot on which stage your startup is at, what your level of funding is, etc. Definitely not as clear-cut as you seem to imply.
koolba 5y ago
> The most expensive part is the electricity. There’s more than a kWh rate for that though. There’s battery backup power, diesel backup, and failover hot switching. It’s not like just plugging in your phone to charge.
m348e912 5y ago
Easy or not, it's a job. Configuring, patching, upgrading, troubleshooting, securing, monitoring. It's only not expensive if your time is worth nothing.
- ykevinator2 5y ago
- greedo 5y ago
  And you still need to do this with an ec2 instance. Sure you can automate a lot of it via config mgmt/orchestration, but unless you're using serverless stuff on AWS, it's an awful lot like having your own VMs.
stephbu 5y ago
It’s pretty hard to generalize this without qualifiers. Electricity can be the most expensive problem, but it requires carefully planned control of the other factors - e.g large scale with highly automated servers, network, and meat-reducing control planes to become true. Otherwise factors such as people and under-utilization can especially dominate smaller and/or less efficient facilities. Thinking thru the factors, many seem obvious, but are often forgotten/ignored when comparing rental or IaaS costs.: Space is a fixed cost driven by market rates and maximum Server Capital Costs i.e. floor space. Failing to fill the room increases your cost/server efficiency. Pretty common to run out of thermal/power before you run out space, as equipment efficiency increases through the lifespan of the facility. Server and Power costs scale together, carry a minimum cost for keeping machines on, and vary based on utilization. Again if the servers aren’t doing work, your efficiency ratio will drop. Larger space typically have pre negotiated power commitments too - failing to consume that carries fiscal penalties. Servers unit costs are fairly cheap, storage not so much. Full utilization throughout capital/lease lifespan is the goal - anything less increases relative cost/server. Network costs scale with Server Costs, and vary again by utilization - minimum invest rules apply, all servers need at least one network port, as well as upstream Core/TOR/Miniswitch gear. The network gear lifespan is typically longer than servers, but shorter than facilities. It usually incurs annual support/maintenance charges too. Bandwidth charges are variable as expected. People costs scale with a step-function and numbers driven by minimum coverage requirements, task complexities, and level of human toil. Performing any task on a device by-hand is expensive in most markets - touches on tickets, change management, task time etc. Fully burdened S+R in Western cultures is typically ~2x the salary - a $80K employee probably costs around $150K by the time all the workplace costs, taxes, and benefits are paid. Network folk are typically premium resources compared to DC Ops. Sustainable 24x7 coverage looks like a staff of 3-4 people.
- vidarh 5y ago
  The thing is, you can buy your way out of most of these considerations by either renting colo space or renting managed servers until/unless you're at a scale where there are savings to doing it yourself. This is a commodity service with margins a tiny fraction of AWS' margins. Buying services at those levels instead of doing hosting on prem still nets you 80%-90% of the savings vs. cloud. Sometimes it nets you greater savings because it makes it easier to move your workloads to cheaper locations (e.g. for a small company in London operating servers somewhere with low energy prices and low property prices is hard; colo space reachable from London in often 30%-50% above what it can be like many other places in Europe, and so you'll find managed servers from providers like Hetzner is often cheaper than on prem if your company is in a location like that)
briffle 5y ago
Been a sysadmin for many years. Your right, for a few computers. Once you start getting into more than a quarter rack, you also need to start worrying about cooling (which also is lots of electricity) and usually things like ensuring the electricity stays on. (UPS, generator, etc). Don't forget to monitor and service all this stuff regularly. Once your past a few racks of equipment, you have generator tests and service appointments, redundant AC, Redundant UPS, dual power to each rack, etc. Dual Internet connection, and a link to your other server room that you use for DR, etc. The costs and complexity quickly escalate after a server or two.
- dhzhzjsbevs 5y ago
  In the bad old days we bought our own servers and rented rack space in a datacenter they took care of all the hardware management, we looked after the software. Mind you this was only about 4U worth.
- BryanRamirez 5y ago
  (I created an account just to upvote and comment on this) Absolutely, nevermind managing OS upgrades and needing to use configuration management to mitigate against drift. It took a lot of effort to make sure that each server was not a special snowflake that could not be reliably reproduced. Also, dealing with vendor warranties, and being on hold with HP (or whoever), then assuring them you're running the latest firmware.. please for the love of god just replace the failed memory/disk/cpu. I found the sheer physicality of computing infrastructure to be a source of exhaustion and burnout. It's a big part of the reason why I'm a software developer now :)
- secabeen 5y ago
  > Once your past a few racks of equipment, you have generator tests and service appointments, redundant AC, Redundant UPS, dual power to each rack, etc. Dual Internet connection, and a link to your other server room that you use for DR, etc. The costs and complexity quickly escalate after a server or two. This is not always the case. For computationally-focused workloads like the OP describes, without direct customer interaction, it may be reasonable to accept risk of downtime in the event of failure. If you are doing computations that take weeks to complete, and you checkpoint regularly, does it really matter if your computation finishes on Saturday morning or Monday morning? If not, you can probably accept 6 hours of downtime once every couple of years and eliminate all of the redundancy overhead described above. In HPC, the general rule of thumb is buy your hardware if you can be sure you'll run compute on it more than ~40% of the time.

stephbu 5y ago

To add to this statement - don’t make any investment unless you have a solid plan to fully utilize it for the full length of the purchase terms. This applies as much to AWS @ minute terms, as it does to a rental @ daily terms, as it does to co-lo capital investment @ yearly terms.

marcosdumay 5y ago
> unless you have a solid plan to fully utilize it for the full length of the purchase Hum... I'd say it's much more reasonable to look at the ROI. Making investments to supply peak demand or to hedge against rare risks is perfectly ok.
- stephbu 5y ago
  Fair point, I guess the point is that unallocated resources - space/power/servers etc. can become huge stealth money sinks, eating budget every hour of every day. Being cognizant of the consumption economics before you stump up for resources is important, as is fitting the investment model to match those economics. Setting utilization/allocation targets are just one way of measuring if those models efficiently match. This is true for any service or resource consumed.