5 ms·

> Colocated systems tend to work very well at first when the original engineers who set it up are all still at the company and it hasn't run long enough to star

> Colocated systems tend to work very well at first when the original engineers who set it up are all still at the company and it hasn't run long enough to start encountering rare failure modes. They quickly become a nightmare when your engineering staff turns over multiple times and nobody can remember who knows how to do what on the colocated system or if the documentation is up to date or not. Everything above really sounds like it's just regurgitating AWS sales person talking points. Sounds like a systemic management / CTO-level problem to me if a company isn't willing to put in place the hiring practices and compensation, documentation systems and operational procedures to deal with that sort of concern. If your core engineering staff is turning over multiple times for arbitrary reasons you have other problems to deal with. > Engineers can't set up, maintain, and do on-call for the colocated setup without subtracting from their primary working hours. If a company can't hire datacenter techs to install hardware, cables, and swap hardware as smart remote hands, maintaining as little as a couple of 45RU cabinets of gear, you also have other management/systemic problems to deal with. I'm looking at this from the point of view of a facilities based bare metal ISP that owns/runs all of its own hardware, and can tell you it's not rocket science.

PragmaticPulp 5y ago
> If you core engineering staff is turning over multiple times for arbitrary reasons you have other problems to deal with. People leave for all sorts of reasons: Moving for family reasons, becoming stay-at-home parents, moving for a spouse's job, retiring, starting their own companies, or even just getting bored and wanting to do something different. Or it could be as simple as getting promoted to a different role. It's unrealistic to make engineering decisions with the assumption that the same engineers will be around and stuck on the same project forever. Like the OP said: Every hour they have to spend working on the colocation setup is an hour they aren't spending on your company's competitive advantage, so you have to hire more engineers (and more managers) to compensate. > If a company can't hire datacenter techs... How many techs do you think you need for reasonable on-call coverage? 3? 6? Add a manager in the mix because you need someone to manage them. The costs add up quickly. It's weird to see people championing colocation as a cost saver and then pivoting to arguments that you just need to hire more engineers and techs and manage them. Employees are expensive. One of the primary benefits of cloud is that you don't have to hire and manage all of these employees to do all of these things at the colo.
- _s 5y ago
  ^^ This 1000x. Humans are incredibly expensive and notoriously unreliable when compared against “machines”; or in this case an API. It’s usually worth paying 2-3x the cost to have someone else manage something for you with a given SLA, because that’s what it will end up being when you decide to bring it in house when you take into account the time and effort needed as well. A “good” & “reliable” Systems Engineering team, that can offer 24/7 support will take around a year to hire and setup, and they need roughly the same amount of time to transition you off AWS in to your system. They probably need closer to 3-5yrs to give the same level of documentation, API’s, tooling, processes, UI’s and training that AWS already provides. Let’s call it 5 years to get to the level of AWS when you started the transition. A decent team of 5-7, including engineers + PO/PM + UX and so forth, is at least $1.5M/yr. That’s $7.5M over 5 years, not including your new hardware and networking costs. Let’s call it $10M. You’re also 5 years behind AWS now, and over that transition you’re still paying AWS, and your development speed has halved as you wait for your new team to build or transition infrastructure. You can trade cost and quality for speed and have everything ready in 2-3 years by setting up a few teams. Add HR support, more contractors etc etc. you’re looking at a $10M+ outlay again, regardless. Or you can keep paying AWS $5M/yr, renegotiate fees often and literally not worry about that headache and focus on your product.
  - Dylan16807 5y ago
    You act like AWS doesn't require you to have a team, or carefully transition infrastructure. Because of that your cost estimate is much higher than the couple additional people you should actually need. > They probably need closer to 3-5yrs to give the same level of documentation, API’s, tooling, processes, UI’s and training that AWS already provides. You don't need to build an internal AWS to manage your own servers.
- midasuni 5y ago
  My team of three looks after hundreds of bits of diverse kit in dozens of locations across the world. I can’t remember the last time I took a call outside of office hours, and even in hours it’s very rare. There’s enough resilience built in that any issues can wait until morning The last major outage was in 2017, before we had a third member of the team. I was on the other side of the world installing a new system, the other was on leave. We had a network issue, OSPF melted and knocked out some services, we were down for about half an hour as I rebooted the core switch pair remotely. (We’ve since redesigned so that doesn’t happen) We get paid nowhere near six figures either. Sure you can be ridiculous, I remember one team I worked on that employed a full time unix contractor (on 3 times the staff wage) to look after 6 servers and deploy a tar ball every few months. I replaced him with a small shell script. Another was a DBA looking after a small oracle database (oracle - which of course is that generations “just use amazon”)
loriverkutya 5y ago
If I have to pay the same amount of money and I can choose to deal with people or deal with a decent sized company (and leave them to deal with their people), I always choose the later.
hagy 5y ago
I worked at a company that migrated a 100 PB Hadoop cluster to GCP for assorted reasons despite many years of success with colocation. I wasn't involved in any of this, but the team's decision process makes sense. You can read through their decision making in these blog posts: * https://liveramp.com/developers/blog/google-cloud-platform-gcp-migration/ https://liveramp.com/developers/blog/google-cloud-platform-g... * https://liveramp.com/developers/blog/migrating-a-big-data-environment-to-the-cloud-part-1/ https://liveramp.com/developers/blog/migrating-a-big-data-en... One big point was challenges of maintaining multiple colocation sites, with cross replication, for disaster recovery. Since Hadoop triple replicates all data within one DC, this requires 6 times the disk storage capacity of data size for dual DCs. In contrast, cloud object storage pricing includes replication within a region with very high availability such that storing once in cloud storage may be acceptable. Further, you also need double the compute, with one of the DCs always standing by should the other fail.
- dilyevsky 5y ago
  HDFS supports RS/XOR erasure coding which gives you same fault tolerance guarantees as 3x at much lower replication factor. This is essentially the same method aws/gcp use under the hood - there’s no magic involved here
  - genewitch 5y ago
    Can I use this over NFS or samba or is it just for Java and Hadoop?
    - dilyevsky 5y ago
      Yes you can* for new projects I would go with ceph+nfsv4 *) https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/ha...
      - genewitch 5y ago
        ceph+nfs was actually what i was going to do. Probably some learning pains but at least i don't have to "lose" half my storage because the sizes aren't paired with anything.
nemothekid 5y ago
You are an ISP, and managing that hardware is your core competency. For every other CTO, saving money by moving to on-prem is picking up pennies in front of steamrollers. Just because you spend $3 million/year on AWS doesn't mean that same organization can turn around can hire 10 sysadmins and do everything themselves; anyone who thinks they can simply execute well just because they have a big checkbook is in for a rude awakening. For a growing company I would always weigh the organizational overhead of moving on-prem to the actual cloud costs. If your goal is in the next year to grow revenue from 50MM to 100MM, then a company wide engineering effort to move to bespoke platform to save 5-10MM just isn't a good use of time (keep in mind you are trying to hire and release new features). It's not that it's impossible; it's that trying to both scale up your business and provide infrastructure to yourself is just unnecessarily increasing the difficulty on your businesses execution.

PragmaticPulp 5y ago

> If you core engineering staff is turning over multiple times for arbitrary reasons you have other problems to deal with. People leave for all sorts of reasons: Moving for family reasons, becoming stay-at-home parents, moving for a spouse's job, retiring, starting their own companies, or even just getting bored and wanting to do something different. Or it could be as simple as getting promoted to a different role. It's unrealistic to make engineering decisions with the assumption that the same engineers will be around and stuck on the same project forever. Like the OP said: Every hour they have to spend working on the colocation setup is an hour they aren't spending on your company's competitive advantage, so you have to hire more engineers (and more managers) to compensate. > If a company can't hire datacenter techs... How many techs do you think you need for reasonable on-call coverage? 3? 6? Add a manager in the mix because you need someone to manage them. The costs add up quickly. It's weird to see people championing colocation as a cost saver and then pivoting to arguments that you just need to hire more engineers and techs and manage them. Employees are expensive. One of the primary benefits of cloud is that you don't have to hire and manage all of these employees to do all of these things at the colo.

_s 5y ago
^^ This 1000x. Humans are incredibly expensive and notoriously unreliable when compared against “machines”; or in this case an API. It’s usually worth paying 2-3x the cost to have someone else manage something for you with a given SLA, because that’s what it will end up being when you decide to bring it in house when you take into account the time and effort needed as well. A “good” & “reliable” Systems Engineering team, that can offer 24/7 support will take around a year to hire and setup, and they need roughly the same amount of time to transition you off AWS in to your system. They probably need closer to 3-5yrs to give the same level of documentation, API’s, tooling, processes, UI’s and training that AWS already provides. Let’s call it 5 years to get to the level of AWS when you started the transition. A decent team of 5-7, including engineers + PO/PM + UX and so forth, is at least $1.5M/yr. That’s $7.5M over 5 years, not including your new hardware and networking costs. Let’s call it $10M. You’re also 5 years behind AWS now, and over that transition you’re still paying AWS, and your development speed has halved as you wait for your new team to build or transition infrastructure. You can trade cost and quality for speed and have everything ready in 2-3 years by setting up a few teams. Add HR support, more contractors etc etc. you’re looking at a $10M+ outlay again, regardless. Or you can keep paying AWS $5M/yr, renegotiate fees often and literally not worry about that headache and focus on your product.
- Dylan16807 5y ago
  You act like AWS doesn't require you to have a team, or carefully transition infrastructure. Because of that your cost estimate is much higher than the couple additional people you should actually need. > They probably need closer to 3-5yrs to give the same level of documentation, API’s, tooling, processes, UI’s and training that AWS already provides. You don't need to build an internal AWS to manage your own servers.
midasuni 5y ago
My team of three looks after hundreds of bits of diverse kit in dozens of locations across the world. I can’t remember the last time I took a call outside of office hours, and even in hours it’s very rare. There’s enough resilience built in that any issues can wait until morning The last major outage was in 2017, before we had a third member of the team. I was on the other side of the world installing a new system, the other was on leave. We had a network issue, OSPF melted and knocked out some services, we were down for about half an hour as I rebooted the core switch pair remotely. (We’ve since redesigned so that doesn’t happen) We get paid nowhere near six figures either. Sure you can be ridiculous, I remember one team I worked on that employed a full time unix contractor (on 3 times the staff wage) to look after 6 servers and deploy a tar ball every few months. I replaced him with a small shell script. Another was a DBA looking after a small oracle database (oracle - which of course is that generations “just use amazon”)

loriverkutya 5y ago

If I have to pay the same amount of money and I can choose to deal with people or deal with a decent sized company (and leave them to deal with their people), I always choose the later.

hagy 5y ago

I worked at a company that migrated a 100 PB Hadoop cluster to GCP for assorted reasons despite many years of success with colocation. I wasn't involved in any of this, but the team's decision process makes sense. You can read through their decision making in these blog posts: * https://liveramp.com/developers/blog/google-cloud-platform-gcp-migration/ https://liveramp.com/developers/blog/google-cloud-platform-g... * https://liveramp.com/developers/blog/migrating-a-big-data-environment-to-the-cloud-part-1/ https://liveramp.com/developers/blog/migrating-a-big-data-en... One big point was challenges of maintaining multiple colocation sites, with cross replication, for disaster recovery. Since Hadoop triple replicates all data within one DC, this requires 6 times the disk storage capacity of data size for dual DCs. In contrast, cloud object storage pricing includes replication within a region with very high availability such that storing once in cloud storage may be acceptable. Further, you also need double the compute, with one of the DCs always standing by should the other fail.

dilyevsky 5y ago
HDFS supports RS/XOR erasure coding which gives you same fault tolerance guarantees as 3x at much lower replication factor. This is essentially the same method aws/gcp use under the hood - there’s no magic involved here
- genewitch 5y ago
  Can I use this over NFS or samba or is it just for Java and Hadoop?
  - dilyevsky 5y ago
    Yes you can* for new projects I would go with ceph+nfsv4 *) https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/ha...
    - genewitch 5y ago
      ceph+nfs was actually what i was going to do. Probably some learning pains but at least i don't have to "lose" half my storage because the sizes aren't paired with anything.

nemothekid 5y ago

You are an ISP, and managing that hardware is your core competency. For every other CTO, saving money by moving to on-prem is picking up pennies in front of steamrollers. Just because you spend $3 million/year on AWS doesn't mean that same organization can turn around can hire 10 sysadmins and do everything themselves; anyone who thinks they can simply execute well just because they have a big checkbook is in for a rude awakening. For a growing company I would always weigh the organizational overhead of moving on-prem to the actual cloud costs. If your goal is in the next year to grow revenue from 50MM to 100MM, then a company wide engineering effort to move to bespoke platform to save 5-10MM just isn't a good use of time (keep in mind you are trying to hire and release new features). It's not that it's impossible; it's that trying to both scale up your business and provide infrastructure to yourself is just unnecessarily increasing the difficulty on your businesses execution.