4 ms·

How to (Re)Build My Infrastructure Team?

ctoinneedofhelp 8y ago
Sorry, I needed more than 2000 characters to give my question the right context.
bryanrasmussen 8y ago
as a general rule if I feel it is difficult to find the right mix of abilities in one individual I want to hire 2 or 3 people and make them work together, that is of course also expensive but might be necessary.
- ctoinneedofhelp 8y ago
  Fully understand that. But beyond the cost, my main concern is communication bandwidth. So productivity. Not even sure 3 people would be as productive as the right guy.
mattbillenstein 8y ago
What is your scale? I think a single engineer given sharp tools and a limited amount of services to deploy can support probably hundreds of systems -- this can be a vast amount of compute and storage given the state of today's hardware.
- ctoinneedofhelp 8y ago
  <100 servers to manage. <100TB of data. <50 technical users.
  - mattbillenstein 8y ago
    Cool, I hear you regarding choice of tech stack, cost constraints, etc -- I manage systems with saltstack, Ubuntu, supervisord -- a very much 12 Factor app architecture, but the new way of doing things is k8s and docker. A huge amount of complexity imho. I think you need to hire programmers and groom them more into devops. They will think the product is the automation software while the artifact is the systems. Traditional sysadmins think the systems are the product and the automation software is the artifact, so they tweak the systems to fix things rather than fixing the software and then letting that fix the systems. So, it boils down to philosophy and what one group thinks is the source of truth vs what the other one values. And maybe you can reach into one of these communities - puppet, ansible, saltstack - find a nerd in that space who shares your sensibilities who you'll have an easier time working with.
    - ctoinneedofhelp 8y ago
      I understand how k8s seduces people: sysadmins feels like "let's deliver this to developers then they'll be autonomous", developers feels like "oh cool, i'm free to run my containers". That's cool for small web shops with a lot of web services. But much less cool when you want to control/understand what's running within those containers. Then you need a lot of CI engineering with precise access control. That said, I couldn't care less about all that container hype. Long running web services are <0.01% of our workload. You confirm my feeling that I should now target good software engineers and teach them how to program the infrastructure. But how would you make them interested in infrastructure? My understanding is that they want to be exposed to the business, closer to where the money is, to get more exposure. And that's normal, that's what smart people do to leverage their skills. You're right I should target my tech stack open source communities. I thought about it so many times. But I just don't know how to communicate what I want without being classified as a spammer. So I never did it. ;)
      - mattbillenstein 8y ago
        Yeah, not an easy task - some of them have job boards or you can do the usual digging keyword search on LinkedIn. I dislike LinkedIn, but everyone is on there and being solicited about job opportunities is pretty much the norm.
        ctoinneedofhelp 8y ago
        I did a lot of hunt on LinkedIn with keywords like Terraform, Python and Puppet. I found that guy here. But years later, I have doubts about LinkedIn efficiency, as most good SWE I found through head hunters did not have the Python keyword on their profile even when they rock at it. Now I believe LinkedIn is only useful to target people who are good at self-marketing.
raincom 8y ago
You have identified some problems - k8s is not fit for your set up, unless you have 1000s of nodes serving an app and tens of such apps. - Software engineers are not going to show any interest in infrastructure as code, as they can make more on the engineering side than on the infra side. Even at Google, I heard, it is hard to become SWE for an SRE, unless he joins as SRE-SWE. This tells that not many want to become SRE-SWE. - What else can you teach to your sysadmin, when he can't learn "owing up to mistakes" in a small shop like yours. Im Mega corps, there is a political cost to "owning up to mistakes" right away. If he can't change, he is not going to learn it. You need someone who wants to learn and show some progress. - Where is your company located?
- ctoinneedofhelp 8y ago
  We are located in south east asia. In a place where it's pretty difficult to find decent software engineers unfortunately.
  - raincom 8y ago
    You have better chances there for finding a software engineer who wants to do infrastructure. Good Luck. If you are in singapore, you can find them easy.
    - ctoinneedofhelp 8y ago
      Thanks :)

Sorry, I needed more than 2000 characters to give my question the right context.

as a general rule if I feel it is difficult to find the right mix of abilities in one individual I want to hire 2 or 3 people and make them work together, that is of course also expensive but might be necessary.

ctoinneedofhelp 8y ago
Fully understand that. But beyond the cost, my main concern is communication bandwidth. So productivity. Not even sure 3 people would be as productive as the right guy.

mattbillenstein 8y ago

What is your scale? I think a single engineer given sharp tools and a limited amount of services to deploy can support probably hundreds of systems -- this can be a vast amount of compute and storage given the state of today's hardware.

ctoinneedofhelp 8y ago
<100 servers to manage. <100TB of data. <50 technical users.
- mattbillenstein 8y ago
  Cool, I hear you regarding choice of tech stack, cost constraints, etc -- I manage systems with saltstack, Ubuntu, supervisord -- a very much 12 Factor app architecture, but the new way of doing things is k8s and docker. A huge amount of complexity imho. I think you need to hire programmers and groom them more into devops. They will think the product is the automation software while the artifact is the systems. Traditional sysadmins think the systems are the product and the automation software is the artifact, so they tweak the systems to fix things rather than fixing the software and then letting that fix the systems. So, it boils down to philosophy and what one group thinks is the source of truth vs what the other one values. And maybe you can reach into one of these communities - puppet, ansible, saltstack - find a nerd in that space who shares your sensibilities who you'll have an easier time working with.
  - ctoinneedofhelp 8y ago
    I understand how k8s seduces people: sysadmins feels like "let's deliver this to developers then they'll be autonomous", developers feels like "oh cool, i'm free to run my containers". That's cool for small web shops with a lot of web services. But much less cool when you want to control/understand what's running within those containers. Then you need a lot of CI engineering with precise access control. That said, I couldn't care less about all that container hype. Long running web services are <0.01% of our workload. You confirm my feeling that I should now target good software engineers and teach them how to program the infrastructure. But how would you make them interested in infrastructure? My understanding is that they want to be exposed to the business, closer to where the money is, to get more exposure. And that's normal, that's what smart people do to leverage their skills. You're right I should target my tech stack open source communities. I thought about it so many times. But I just don't know how to communicate what I want without being classified as a spammer. So I never did it. ;)
    - mattbillenstein 8y ago
      Yeah, not an easy task - some of them have job boards or you can do the usual digging keyword search on LinkedIn. I dislike LinkedIn, but everyone is on there and being solicited about job opportunities is pretty much the norm.
      - ctoinneedofhelp 8y ago
        I did a lot of hunt on LinkedIn with keywords like Terraform, Python and Puppet. I found that guy here. But years later, I have doubts about LinkedIn efficiency, as most good SWE I found through head hunters did not have the Python keyword on their profile even when they rock at it. Now I believe LinkedIn is only useful to target people who are good at self-marketing.

raincom 8y ago

You have identified some problems - k8s is not fit for your set up, unless you have 1000s of nodes serving an app and tens of such apps. - Software engineers are not going to show any interest in infrastructure as code, as they can make more on the engineering side than on the infra side. Even at Google, I heard, it is hard to become SWE for an SRE, unless he joins as SRE-SWE. This tells that not many want to become SRE-SWE. - What else can you teach to your sysadmin, when he can't learn "owing up to mistakes" in a small shop like yours. Im Mega corps, there is a political cost to "owning up to mistakes" right away. If he can't change, he is not going to learn it. You need someone who wants to learn and show some progress. - Where is your company located?

ctoinneedofhelp 8y ago
We are located in south east asia. In a place where it's pretty difficult to find decent software engineers unfortunately.
- raincom 8y ago
  You have better chances there for finding a software engineer who wants to do infrastructure. Good Luck. If you are in singapore, you can find them easy.
  - ctoinneedofhelp 8y ago
    Thanks :)