8 ms·

I did some freelancing cloud work recently and was shocked at what I saw out there. From small one person operations to startups to large multinational Fortune

by coleca 8y ago

I did some freelancing cloud work recently and was shocked at what I saw out there. From small one person operations to startups to large multinational Fortune 500 corporations I saw the same pattern repeated over and over. People using the cloud to spin up infrastructure that have no experience building out infrastructure and making really bad and dangerous mistakes. Amazon, Google, Microsoft and others make it simple to get going but the devil is in the details when it comes to building it right, safe and ready for production. Just a few examples: • Putting RDS SQL Servers on a public IP with no protection • No templating of servers so if it disappears you have no record of how to get a new one running again • Servers with SSH password authentication turned on with passwords of “Password”, because SSH key auth was “too hard for our devs” • No backups because “it’s in the cloud, isn’t the cloud backing it up for me, like iCloud?” I have an AWS Solution Architect Professional certification and agree you could get a cert and still not be qualified to run or design much on AWS but it does put you ahead of much of what I’ve seen out there.

nicoburns 8y ago
Oh wow. Those are some bad ones! I'm normally pretty certification averse (I don't even really value my undergraduate degree much), but I'm thinking of getting the AWS Solution Architect cert, because it seems like the material out there for learning that stuff is genuinely useful (and once you've learnt the material, you may as well get the certificate).
- ldoughty 8y ago
  Cert + personal project, that's my recommendation. Associate's cert is mostly memorization and high level, but doesn't qualify you for being a trusted individual to do the job. It's good to make you aware of the right way to build things in AWS. For $2/month you can make a static website, posted to S3, with a CloudFront front end and web hook to lambda to trigger content refreshes on specific Git actions. I'd hire someone who did this with no professional experience with or without certification for an entry level position. That personal interest and drive is worth 1-2 years experience in my book... And I don't look for more than 6 months experience in entry level (I offer internships/ hourly for that group)
  - nicoburns 8y ago
    I guess so. But that's the easy part of AWS, right? I'm pretty sure (having used AWS a little, but not CloudFront or Lambda) I could start building that now and have it done by this time tomorrow. The tricky part seems like it would be dealing with tens-to-hundreds-to-thousands of machines which should automatically deploy and scale. And things like databases, which might also need to scale up/out at some point, and ideally should do without downtime. That's much harder to try out on a personal project (due to the cost, but also due to the lack of genuine need shaping the implementation)...
    - lostapathy 8y ago
      Let’s be real - how many business projects honestly require tens to thousands of “machines”? And what fraction of IT people work on those projects? There is a huge amount of IT work that never gets to that scale, and of the IT projects that get there, lots don’t actually need to be.
      - brobdingnagians 8y ago
        I guess it's all part of the "silver bullet" and "hype" culture. People want something that will automatically scale their new giga-scale web product that is going to be "the new facebook!", and managers want something that will make them rich without too much work, so they sign up for all the latest cool tools instead of thinking deeply about their real requirements and good old fashioned hard work. Some of the latest cool tools really _are_ amazing and worth every penny and more, but I'm sceptical about some of them...
    - jacurtis 8y ago
      You would be surprised. Most companies I work with are operating on only 1-3 machines, and using the CloudFront infrastructure to distribute content to edge locations around the world. They a handful of Lambda functions to trigger various random tasks like backups, notifications, git pulls, deploys, etc. Oh and S3... tons of S3 storage. I have a client that does $50M a year on a SAAS product using only two EC2 isntances, but 1 of those instances is a developer/beta/test server. Only 1 is used to manage sales across the entire world. They rely heavily on Lambda and SNS & SQS as well. Another client operates eCommerce at around $80M a year revenue using 3 EC2 instances. One of those is a test/dev server and the other 2 are used to distribute load and for redundancy with a load balancer sitting on top. I have 2 other companies that are smaller that generates about $2M and $5M a year respectively that both run a static site hosted on S3, using Lambda for deploys and cloudfront for distribution like mentioned above. Point being, you would be surprised how low key many company's AWS deployments are. --- If I could pin one skill that will stand you head and shoulders above the rest among cloud architects, it is to learn Serverless functions. In this AWS world this is "Lambda". Microsoft calls it "Azure Functions" and Google Cloud Platform calls it "Cloud Functions". If you can write serverless functions AND are AWS certified, that seems to be the hottest thing that I have seen.
whoisthemachine 8y ago
This is also typical of companies that don't use the cloud. Some companies just have poor security practices!
- ldoughty 8y ago
  Or they fire the people that know this stuff now that they are in the cloud :-)
znpy 8y ago
I think that the core of the issue lies in their certification style: simple multiple-choices question, with many of them certifying that you understood that you should use the branded product (example: Amazon RDS instead of managing your own posters cluster in the cloud -- which might best suit your use case). So you're tested on your understanding of the platform but not on your actual competences on it. I've seen this in other areas however: people getting LPIC certifications and failing to run a proper grep. In my experience, Red Hat is one of the few tech companies that get the certification program right by using an hands-on approach: either you actually are capable of performing tasks at a certain level, or you get not certification at all. There are many professionals certified on AWS that aren't really able to perform properly on that platform, for example. Add the fact that plain old system administration I'd dying (less and less needed with the cloud and things like ansible)... And here we are.
- barbecue_sauce 8y ago
  The Kubernetes certifications are also hands on, and actually require a lot of operational knowledge that someone who uses it on a daily basis probably wouldn't even be exposed to (like bootstrapping it from scratch as opposed to using more commonly accepted tools). From what I can tell, AWS recently offloaded their certification program onto a third party, so I doubt it will improve anytime soon.
  - dankohn1 8y ago
    Yes, as part of the Certified Kubernetes Administrator exam, you spin up, configure, and debug 7 clusters. https://www.cncf.io/certification/training/ https://www.cncf.io/certification/training/ (Disclosure: I helped develop the exam.)
    - barbecue_sauce 8y ago
      How stable is the exam at this point? I was looking into it a few months ago, but it seemed that Kubernetes' own APIs and primitives were expanding at the time so I figured I should hold off. (Also heard your interview on Software Engineering Daily the other day, very informative)
blunte 8y ago
Sounds to me like this is an opportunity to sell security/disaster prevention services to the companies you describe. Walk in and show a list of externally visible problems (in a way that clearly does not suggest "hacking"), and propose a project to get them correct.
- mooreds 8y ago
  > Sounds to me like this is an opportunity to sell security/disaster prevention services to the companies you describe. I have an acquaintance that is aiming at that market: https://backstop.it/ https://backstop.it/
avip 8y ago
Never seen any of that in any startup. Did see not using terraform and inf being managed from UI and not reproducible from scratch.
drieddust 8y ago
I work for a large IT outsourcing company and I have seen the opposite of it with our clients. Most of our clients wants to be on cloud but without any change to their habits so we end up building complete traditional data centers in the Azure or AWS. Most applications are simply cloned from existing On-Premises Infrastructure because every project have a 2 weeks deadline and every single stakeholder just wants to stick it in. Patterns like Auto-Scaling etc is just a far fetched dream. Supporting such Infrastructure where every server is a pet trying to survive in a Slaughter House is a deeply painful.
- halbritt 8y ago
  They call this "lift and shift". It's a terrible idea, but cloud providers love it because they end up getting paid a lot of money for idle capacity.
  - drieddust 8y ago
    Exactly and any sane conversation is met with verbal arrows loaded with words like risk mitigation, minimizing business impact, being agile etc.
- ariwilson 8y ago
  IMO this is why Google Cloud was behind AWS (talking 2008-2013 or so). Google bet on PaaS while Amazon bet on IaaS. The dumb corporate money is still behind IaaS, as inefficient as it can be.
  - drieddust 8y ago
    Well a lot of these people are smart BUT who wants to risk his job for an adventure which if successful will help the company and if unsuccessful will probably get him fired.
    - scarface74 8y ago
      That’s why the real money is in consulting. They can outsource blame.
becga 8y ago
I consulted with a place where they brought in a contractor to do AWS stuff. He spent weeks troubleshooting tomcat on EC2, his resume said he knew cloudformation yet when he was asked to write a lambda function using the tool, I had to do it for him. Knew nothing about troubleshooting Postgres on RDS etc. Fast forward a couple of years and he had a couple of AWS certs and was now being employed by some firms to better implement AWS usage. Myself, have no AWS certs myself but have done migration and large project work for several firms. So I think they can definitely be useful for baseline knowledge. This same person I mentioned above also said the company I was at at the time (a startup) should hire him as an Architect yet he doesn't do any app-level coding etc. So I suppose while the cloud bootstrap work is around for now, I definitely see this type of stuff getting consolidated.
- scarface74 8y ago
  There is a difference between knowing AWS and knowing applications that run on AWS. Not knowing Tomcat and Postgres is excusable. I’ve been using C# and MySQL and SQL Server for years. I’ve only done APIs in Node and Python with lambda. The on,y web server I know well is IIS.
thomaskcr 8y ago
On due diligence projects one of my temperature taking questions for AWS is "are you using services besides EC2 and S3?". Of course we'll dig into things, but it does give me a good idea of what that part of the DD engagement is going to look like. It tells me whether they have bothered to learn how to do stuff in the cloud or are just applying their old patterns to the cloud (at that point, they should just have colo/managed servers - they would save money, but usually the driver for the cloud came from the C-Suite in those situations - either way some training and a cost cutting project usually pays for itself in those situations).
foobiekr 8y ago
The worst part of reading your list is that as I read through it I noted my own encounters with each of them, and then some: * what do you mean who installs patches? I think our cloud provider does. * (same, but for "firewall" or "policy") * "I think they turn them off automatically if they're idle" and so on. The "devops" hype has put people who think nothing of operations at all in charge of something they don't understand.
yjftsjthsd-h 8y ago
> because SSH key auth was “too hard for our devs” This always kills me; the worst part is that keys are easier to use after you take 5 minutes to set them up once!
unclebucknasty 8y ago
Some of this is lack of IT experience in general vs with AWS, and just plain laziness. Still, it's interesting because some of it requires that you actively work against AWS to achieve (e.g. publicly accessible RDS). Could be that they gave up on, say, security groups and just found that to be the easiest way to "get it to work" (a frequent overriding directive). In any case, I'd put some of it on Amazon too. Their documentation always struck me as piecemeal and task-oriented; lacking a sense of overall architecture or design. But, I suppose that's what certifications are for.

by coleca 8y ago

nicoburns 8y ago

Oh wow. Those are some bad ones! I'm normally pretty certification averse (I don't even really value my undergraduate degree much), but I'm thinking of getting the AWS Solution Architect cert, because it seems like the material out there for learning that stuff is genuinely useful (and once you've learnt the material, you may as well get the certificate).

ldoughty 8y ago
Cert + personal project, that's my recommendation. Associate's cert is mostly memorization and high level, but doesn't qualify you for being a trusted individual to do the job. It's good to make you aware of the right way to build things in AWS. For $2/month you can make a static website, posted to S3, with a CloudFront front end and web hook to lambda to trigger content refreshes on specific Git actions. I'd hire someone who did this with no professional experience with or without certification for an entry level position. That personal interest and drive is worth 1-2 years experience in my book... And I don't look for more than 6 months experience in entry level (I offer internships/ hourly for that group)
- nicoburns 8y ago
  I guess so. But that's the easy part of AWS, right? I'm pretty sure (having used AWS a little, but not CloudFront or Lambda) I could start building that now and have it done by this time tomorrow. The tricky part seems like it would be dealing with tens-to-hundreds-to-thousands of machines which should automatically deploy and scale. And things like databases, which might also need to scale up/out at some point, and ideally should do without downtime. That's much harder to try out on a personal project (due to the cost, but also due to the lack of genuine need shaping the implementation)...
  - lostapathy 8y ago
    Let’s be real - how many business projects honestly require tens to thousands of “machines”? And what fraction of IT people work on those projects? There is a huge amount of IT work that never gets to that scale, and of the IT projects that get there, lots don’t actually need to be.
    - brobdingnagians 8y ago
      I guess it's all part of the "silver bullet" and "hype" culture. People want something that will automatically scale their new giga-scale web product that is going to be "the new facebook!", and managers want something that will make them rich without too much work, so they sign up for all the latest cool tools instead of thinking deeply about their real requirements and good old fashioned hard work. Some of the latest cool tools really _are_ amazing and worth every penny and more, but I'm sceptical about some of them...
  - jacurtis 8y ago
    You would be surprised. Most companies I work with are operating on only 1-3 machines, and using the CloudFront infrastructure to distribute content to edge locations around the world. They a handful of Lambda functions to trigger various random tasks like backups, notifications, git pulls, deploys, etc. Oh and S3... tons of S3 storage. I have a client that does $50M a year on a SAAS product using only two EC2 isntances, but 1 of those instances is a developer/beta/test server. Only 1 is used to manage sales across the entire world. They rely heavily on Lambda and SNS & SQS as well. Another client operates eCommerce at around $80M a year revenue using 3 EC2 instances. One of those is a test/dev server and the other 2 are used to distribute load and for redundancy with a load balancer sitting on top. I have 2 other companies that are smaller that generates about $2M and $5M a year respectively that both run a static site hosted on S3, using Lambda for deploys and cloudfront for distribution like mentioned above. Point being, you would be surprised how low key many company's AWS deployments are. --- If I could pin one skill that will stand you head and shoulders above the rest among cloud architects, it is to learn Serverless functions. In this AWS world this is "Lambda". Microsoft calls it "Azure Functions" and Google Cloud Platform calls it "Cloud Functions". If you can write serverless functions AND are AWS certified, that seems to be the hottest thing that I have seen.

whoisthemachine 8y ago

This is also typical of companies that don't use the cloud. Some companies just have poor security practices!

ldoughty 8y ago
Or they fire the people that know this stuff now that they are in the cloud :-)

znpy 8y ago

I think that the core of the issue lies in their certification style: simple multiple-choices question, with many of them certifying that you understood that you should use the branded product (example: Amazon RDS instead of managing your own posters cluster in the cloud -- which might best suit your use case). So you're tested on your understanding of the platform but not on your actual competences on it. I've seen this in other areas however: people getting LPIC certifications and failing to run a proper grep. In my experience, Red Hat is one of the few tech companies that get the certification program right by using an hands-on approach: either you actually are capable of performing tasks at a certain level, or you get not certification at all. There are many professionals certified on AWS that aren't really able to perform properly on that platform, for example. Add the fact that plain old system administration I'd dying (less and less needed with the cloud and things like ansible)... And here we are.

barbecue_sauce 8y ago
The Kubernetes certifications are also hands on, and actually require a lot of operational knowledge that someone who uses it on a daily basis probably wouldn't even be exposed to (like bootstrapping it from scratch as opposed to using more commonly accepted tools). From what I can tell, AWS recently offloaded their certification program onto a third party, so I doubt it will improve anytime soon.
- dankohn1 8y ago
  Yes, as part of the Certified Kubernetes Administrator exam, you spin up, configure, and debug 7 clusters. https://www.cncf.io/certification/training/ https://www.cncf.io/certification/training/ (Disclosure: I helped develop the exam.)
  - barbecue_sauce 8y ago
    How stable is the exam at this point? I was looking into it a few months ago, but it seemed that Kubernetes' own APIs and primitives were expanding at the time so I figured I should hold off. (Also heard your interview on Software Engineering Daily the other day, very informative)

blunte 8y ago

Sounds to me like this is an opportunity to sell security/disaster prevention services to the companies you describe. Walk in and show a list of externally visible problems (in a way that clearly does not suggest "hacking"), and propose a project to get them correct.

mooreds 8y ago
> Sounds to me like this is an opportunity to sell security/disaster prevention services to the companies you describe. I have an acquaintance that is aiming at that market: https://backstop.it/ https://backstop.it/

avip 8y ago

Never seen any of that in any startup. Did see not using terraform and inf being managed from UI and not reproducible from scratch.

drieddust 8y ago

I work for a large IT outsourcing company and I have seen the opposite of it with our clients. Most of our clients wants to be on cloud but without any change to their habits so we end up building complete traditional data centers in the Azure or AWS. Most applications are simply cloned from existing On-Premises Infrastructure because every project have a 2 weeks deadline and every single stakeholder just wants to stick it in. Patterns like Auto-Scaling etc is just a far fetched dream. Supporting such Infrastructure where every server is a pet trying to survive in a Slaughter House is a deeply painful.

halbritt 8y ago
They call this "lift and shift". It's a terrible idea, but cloud providers love it because they end up getting paid a lot of money for idle capacity.
- drieddust 8y ago
  Exactly and any sane conversation is met with verbal arrows loaded with words like risk mitigation, minimizing business impact, being agile etc.
ariwilson 8y ago
IMO this is why Google Cloud was behind AWS (talking 2008-2013 or so). Google bet on PaaS while Amazon bet on IaaS. The dumb corporate money is still behind IaaS, as inefficient as it can be.
- drieddust 8y ago
  Well a lot of these people are smart BUT who wants to risk his job for an adventure which if successful will help the company and if unsuccessful will probably get him fired.
  - scarface74 8y ago
    That’s why the real money is in consulting. They can outsource blame.

becga 8y ago

I consulted with a place where they brought in a contractor to do AWS stuff. He spent weeks troubleshooting tomcat on EC2, his resume said he knew cloudformation yet when he was asked to write a lambda function using the tool, I had to do it for him. Knew nothing about troubleshooting Postgres on RDS etc. Fast forward a couple of years and he had a couple of AWS certs and was now being employed by some firms to better implement AWS usage. Myself, have no AWS certs myself but have done migration and large project work for several firms. So I think they can definitely be useful for baseline knowledge. This same person I mentioned above also said the company I was at at the time (a startup) should hire him as an Architect yet he doesn't do any app-level coding etc. So I suppose while the cloud bootstrap work is around for now, I definitely see this type of stuff getting consolidated.

scarface74 8y ago
There is a difference between knowing AWS and knowing applications that run on AWS. Not knowing Tomcat and Postgres is excusable. I’ve been using C# and MySQL and SQL Server for years. I’ve only done APIs in Node and Python with lambda. The on,y web server I know well is IIS.

thomaskcr 8y ago

On due diligence projects one of my temperature taking questions for AWS is "are you using services besides EC2 and S3?". Of course we'll dig into things, but it does give me a good idea of what that part of the DD engagement is going to look like. It tells me whether they have bothered to learn how to do stuff in the cloud or are just applying their old patterns to the cloud (at that point, they should just have colo/managed servers - they would save money, but usually the driver for the cloud came from the C-Suite in those situations - either way some training and a cost cutting project usually pays for itself in those situations).

foobiekr 8y ago

The worst part of reading your list is that as I read through it I noted my own encounters with each of them, and then some: * what do you mean who installs patches? I think our cloud provider does. * (same, but for "firewall" or "policy") * "I think they turn them off automatically if they're idle" and so on. The "devops" hype has put people who think nothing of operations at all in charge of something they don't understand.

yjftsjthsd-h 8y ago

> because SSH key auth was “too hard for our devs” This always kills me; the worst part is that keys are easier to use after you take 5 minutes to set them up once!

unclebucknasty 8y ago

Some of this is lack of IT experience in general vs with AWS, and just plain laziness. Still, it's interesting because some of it requires that you actively work against AWS to achieve (e.g. publicly accessible RDS). Could be that they gave up on, say, security groups and just found that to be the easiest way to "get it to work" (a frequent overriding directive). In any case, I'd put some of it on Amazon too. Their documentation always struck me as piecemeal and task-oriented; lacking a sense of overall architecture or design. But, I suppose that's what certifications are for.