5 ms·

Hmm really? My experience is that, with k8s I end up learning all the complexities of k8s in addition to AWS. Besides, personally I find AWS console much easi

Hmm really? My experience is that, with k8s I end up learning all the complexities of k8s in addition to AWS. Besides, personally I find AWS console much easier to understand. I don't get why people hate it.

frompdx 6y ago
> I don't get why people hate it. Because it is hard to manage the configuration. It's why tools like terraform exist. Anecdote. I worked for a small company that was later acquired. It turned out one of the long time employees had set up the company's AWS account using his own Amazon account. Bad on it's own. We built out the infra in AWS. A lot of it was "click-ops". There was no configuration management. Not even CloudFormation (which is not all that great in my opinion). Acquiring company realizes mistake after the fact. Asks employee to turn over account. Employee declines. Acquiring company bites the bullet and shells out a five figure sum to employee to "buy" his account. Could have been avoided with some form of config management.
- cmckn 6y ago
  Sounds like the exact nightmare a previous employer was living. AWS' (awful) web UI convinces the faint of heart to click through wizards for everything. If you're not using version control for _anything_ related to your infrastructure...you have my thoughts and prayers. +1 for "click-ops", perfectly put.
  - frompdx 6y ago
    > you have my thoughts and prayers. Pretty much. The lesson learned for me was to always have version control for the complete stack including the infra for the stack. I like terraform for this. Terragrunt at least solves the issue of modules for terraform reducing the verbosity. Assume things could go wrong and you will need to redeploy EVERYTHING. I've been there.
- bsder 6y ago
  > Acquiring company realizes mistake after the fact. Asks employee to turn over account. Employee declines. Acquiring company bites the bullet and shells out a five figure sum to employee to "buy" his account. Could have been avoided with some form of config management. That is completely the wrong lesson from this anecdote. 1) The acquiring company didn't do proper due diligence. Sorry, this is diligence 101--where are the accounts and who has the keys? 2) Click-Ops is FINE. In a startup, you do what you need now and the future can go to hell because the company may be bankrupt tomorrow. You fix your infra when you need to in a startup. 3) Long-time employee seemed to have exactly the right amount of paranoia regarding his bosses. The fact that the buyout appears to have killed his job and paid so little that he was willing to torch his reputation and risk legal action for merely five figures says something.
  - tly_alex 6y ago
    Make sense. This is a classic example of bad operation management leads to unexpected outcome. Could have be solved much easier with proper configuration management.
- lonelappde 6y ago
  Why not just migrate the resources? Spin up new instances, load data from snapshots, get back to work.
  - tcgv 6y ago
    Probably because paying 5 figures to the account owner was cheaper, faster and safer than recreating the production env from scratch
  - tly_alex 6y ago
    Migration on resources taking live traffic is not an easy thing. Before migrating the traffic over to a different end-point, the time and the engineering work to make sure the new endpoints works is money and cost. Also there could be statefull data in the original account and to do a live migration of data with new data coming in at X tps is absolutely hard work.
- valuearb 6y ago
  Only 5 figures? That’s only what, a 100 hours of consulting? Sounds like the tiniest acquisition mistake I’ve ever heard of.
verdverm 6y ago
How do you view all the VMs in a project across the globe at the same time? Do you need to manage keys when ssh'n into a VM? Do you know what the purpose of all the products are? If you don't know one, are you able to at least have an idea what it's for without going to documentation? The have also directly opposed many efforts for Kubernetes, even to their own customers, until they realized they couldn't win. Only then did they cave, and they are really doing the bare minimum. The most significant contribution to OSS they have made was a big middle finger to Elastic search...
- yongjik 6y ago
  Of course everyone's experience is different, but in my case... > How do you view all the VMs in a project across the globe at the same time? I'm not sure what it's got to do with k8s? I can't see jobs that belong to different k8s clusters at the same time, either. > Do you need to manage keys when ssh'n into a VM? Well, in k8s everybody who has access to the cluster can "ssh" into each pod as root and do whatever they want, or at least that's how I've seen it, but I'm not sure it's an improvement. > Do you know what the purpose of all the products are? If you don't know one, are you able to at least have an idea what it's for without going to documentation? Man, if I got a dime every time someone asked "Does anyone know who owns this kubernetes job?", I'll have... hmm maybe a dollar or two... Of course k8s can be properly managed, but IMHO, whether it is properly managed is orthogonal to whether it's k8s or vanilla AWS.
  - arejaytee 6y ago
    Beleive they were referring to AWS not k8
  - kronin 6y ago
    If you're running pods as root, you're doing it wrong. That was a no-no with docker, and it's still a no-no for kubernetes. People still run non-containerized services as root too...
    - yongjik 6y ago
shiftpgdn 6y ago
AWS console is very carefully designed to let you create traps that result in spending huge buckets of money if you're not paying attention.
- Aeolun 6y ago
  EC2 instances are fairly predictable.
  - verdverm 6y ago
    And when they fail (especially when the underlying hardware fails... Does AWS live migrate your workloads with zero downtime like Google?
    - ec109685 6y ago
      Google can’t migrate if the underlying hardware fails quick enough. I don’t think AWS has talked about live migrate, but given stability of their VMs and rareness of “we need to restart it notices”, it seems like they have something.
      - verdverm 6y ago
        I've "experienced" a hard drive failure on both platforms. On AWS, I was getting pagerduty'd because the solo gateway box was down, we couldn't ssh in so after an hour of no progress with dubugging or support, we just hit the reset button and hoped for the best. Fortunately this worked and later we where told there has a disk failure. On GCP, I didn't even know and only discovered it in the logs when I was looking for other audit reasons. Turns out your long-running google VMs are being migrated all of the time and you had no idea. They actually have a policy / SLA around it, basically saying they refresh their entire fleet of servers every 6 weeks iirc. Honestly, if AWS is not doing something like this, I'd have increased concerns about leaky security neighbors (i.e. someone who has a VM running for multiple years without software updates. Hopefully you should be protected on shared servers, but it is software afterall)
dragonwriter 6y ago
> Besides, personally I find AWS console much easier to understand. I don't get why people hate it. The console is fine as a learning tool for deployment/management, and for occasional experimentation, monitoring, and troubleshooting, but any IaC tool is vastly more manageable for non-toy deployments where you need repeatability and consistency and/or the ability to manage more than a very small number of resources.
tly_alex 6y ago
Console is suppose to just be a web UI to quickly make a change or explore the feature. Any high quality engineering team should avoid make changes through console because it's not testable and repeatable. For code change, use codedeploy or container through ECS. For configuration and infrastructure changes, CloudFormation should be the right tool.

frompdx 6y ago

> I don't get why people hate it. Because it is hard to manage the configuration. It's why tools like terraform exist. Anecdote. I worked for a small company that was later acquired. It turned out one of the long time employees had set up the company's AWS account using his own Amazon account. Bad on it's own. We built out the infra in AWS. A lot of it was "click-ops". There was no configuration management. Not even CloudFormation (which is not all that great in my opinion). Acquiring company realizes mistake after the fact. Asks employee to turn over account. Employee declines. Acquiring company bites the bullet and shells out a five figure sum to employee to "buy" his account. Could have been avoided with some form of config management.

cmckn 6y ago
Sounds like the exact nightmare a previous employer was living. AWS' (awful) web UI convinces the faint of heart to click through wizards for everything. If you're not using version control for _anything_ related to your infrastructure...you have my thoughts and prayers. +1 for "click-ops", perfectly put.
- frompdx 6y ago
  > you have my thoughts and prayers. Pretty much. The lesson learned for me was to always have version control for the complete stack including the infra for the stack. I like terraform for this. Terragrunt at least solves the issue of modules for terraform reducing the verbosity. Assume things could go wrong and you will need to redeploy EVERYTHING. I've been there.
bsder 6y ago
> Acquiring company realizes mistake after the fact. Asks employee to turn over account. Employee declines. Acquiring company bites the bullet and shells out a five figure sum to employee to "buy" his account. Could have been avoided with some form of config management. That is completely the wrong lesson from this anecdote. 1) The acquiring company didn't do proper due diligence. Sorry, this is diligence 101--where are the accounts and who has the keys? 2) Click-Ops is FINE. In a startup, you do what you need now and the future can go to hell because the company may be bankrupt tomorrow. You fix your infra when you need to in a startup. 3) Long-time employee seemed to have exactly the right amount of paranoia regarding his bosses. The fact that the buyout appears to have killed his job and paid so little that he was willing to torch his reputation and risk legal action for merely five figures says something.
- tly_alex 6y ago
  Make sense. This is a classic example of bad operation management leads to unexpected outcome. Could have be solved much easier with proper configuration management.
lonelappde 6y ago
Why not just migrate the resources? Spin up new instances, load data from snapshots, get back to work.
- tcgv 6y ago
  Probably because paying 5 figures to the account owner was cheaper, faster and safer than recreating the production env from scratch
- tly_alex 6y ago
  Migration on resources taking live traffic is not an easy thing. Before migrating the traffic over to a different end-point, the time and the engineering work to make sure the new endpoints works is money and cost. Also there could be statefull data in the original account and to do a live migration of data with new data coming in at X tps is absolutely hard work.
valuearb 6y ago
Only 5 figures? That’s only what, a 100 hours of consulting? Sounds like the tiniest acquisition mistake I’ve ever heard of.

verdverm 6y ago

How do you view all the VMs in a project across the globe at the same time? Do you need to manage keys when ssh'n into a VM? Do you know what the purpose of all the products are? If you don't know one, are you able to at least have an idea what it's for without going to documentation? The have also directly opposed many efforts for Kubernetes, even to their own customers, until they realized they couldn't win. Only then did they cave, and they are really doing the bare minimum. The most significant contribution to OSS they have made was a big middle finger to Elastic search...

yongjik 6y ago
Of course everyone's experience is different, but in my case... > How do you view all the VMs in a project across the globe at the same time? I'm not sure what it's got to do with k8s? I can't see jobs that belong to different k8s clusters at the same time, either. > Do you need to manage keys when ssh'n into a VM? Well, in k8s everybody who has access to the cluster can "ssh" into each pod as root and do whatever they want, or at least that's how I've seen it, but I'm not sure it's an improvement. > Do you know what the purpose of all the products are? If you don't know one, are you able to at least have an idea what it's for without going to documentation? Man, if I got a dime every time someone asked "Does anyone know who owns this kubernetes job?", I'll have... hmm maybe a dollar or two... Of course k8s can be properly managed, but IMHO, whether it is properly managed is orthogonal to whether it's k8s or vanilla AWS.
- arejaytee 6y ago
  Beleive they were referring to AWS not k8
- kronin 6y ago
  If you're running pods as root, you're doing it wrong. That was a no-no with docker, and it's still a no-no for kubernetes. People still run non-containerized services as root too...
  - yongjik 6y ago

shiftpgdn 6y ago

AWS console is very carefully designed to let you create traps that result in spending huge buckets of money if you're not paying attention.

Aeolun 6y ago
EC2 instances are fairly predictable.
- verdverm 6y ago
  And when they fail (especially when the underlying hardware fails... Does AWS live migrate your workloads with zero downtime like Google?
  - ec109685 6y ago
    Google can’t migrate if the underlying hardware fails quick enough. I don’t think AWS has talked about live migrate, but given stability of their VMs and rareness of “we need to restart it notices”, it seems like they have something.
    - verdverm 6y ago
      I've "experienced" a hard drive failure on both platforms. On AWS, I was getting pagerduty'd because the solo gateway box was down, we couldn't ssh in so after an hour of no progress with dubugging or support, we just hit the reset button and hoped for the best. Fortunately this worked and later we where told there has a disk failure. On GCP, I didn't even know and only discovered it in the logs when I was looking for other audit reasons. Turns out your long-running google VMs are being migrated all of the time and you had no idea. They actually have a policy / SLA around it, basically saying they refresh their entire fleet of servers every 6 weeks iirc. Honestly, if AWS is not doing something like this, I'd have increased concerns about leaky security neighbors (i.e. someone who has a VM running for multiple years without software updates. Hopefully you should be protected on shared servers, but it is software afterall)

dragonwriter 6y ago

> Besides, personally I find AWS console much easier to understand. I don't get why people hate it. The console is fine as a learning tool for deployment/management, and for occasional experimentation, monitoring, and troubleshooting, but any IaC tool is vastly more manageable for non-toy deployments where you need repeatability and consistency and/or the ability to manage more than a very small number of resources.

tly_alex 6y ago

Console is suppose to just be a web UI to quickly make a change or explore the feature. Any high quality engineering team should avoid make changes through console because it's not testable and repeatable. For code change, use codedeploy or container through ECS. For configuration and infrastructure changes, CloudFormation should be the right tool.