7 ms·

Shout out to AWS batch, where if you delete the role assigned to a compute cluster the cluster itself becomes impossible to delete. Found this out after using

by orf 4y ago

Shout out to AWS batch, where if you delete the role assigned to a compute cluster the cluster itself becomes impossible to delete. Found this out after using AWS nuke

OJFord 4y ago
That's a weird one, but surely an aws-nuke bug? It must already use a deliberate order - there's plenty of resources that need anything linked/constituent deleted first - so that order is/was just not correct for those?
- bilekas 4y ago
  Nah I've seen this too. It is considered an expected behavior. In a way it makes sense. But it's not well documented.
  - OJFord 4y ago
    But that's what I mean, if that's how it works, nuke needs to account for it. I assume it already does for other dependencies, because there are a lot of them - including that do make more (obvious) sense. You're not the only one to complain about the documentation though, so it's easy to see how aws-nuke apparently missed it!
- Marazan 4y ago
  No, Batch's interaction with IAM roles and permission is super weird and not at all documented. It is easy to screw it up.
  - dekhn 4y ago
    Aws batch is a terrible product
    - aaaaaaaaata 4y ago
      AWS is a terrible product, unless you happen to be in specific corners they've bothered to document/make their UI architecture not a horror show.
  - OJFord 4y ago
    What I mean is 'yes that's a weird interaction' but, if that's how the AWS service works (however weird) then it's an aws-nuke bug that the ordering isn't correct (to account for that weirdness)? Though 'not at all documented' makes that harder of course. I haven't used it.
  - dotancohen 4y ago
    I've never used AWS Batch but heard only good things about it. If you could elaborate - i.e. warn me - I would very much appreciate it. What should I know before using AWS Batch?
    - Marazan 4y ago
      First is the problem as described. If youndelete the ComputeEnvironment Role you can't delete the compute environment itself. If you male a mistake setting up the Batch ComputeEnvironment using cloidformation and it rollbacks then the cloidformation error message is useless (resource failed to stabilise) and no trace is left behind for you to check. Two, Batch needs to use servicelinkedroles. If you create an AWS thing that needs a servicelinkedrole via the console it gets creates for you automatically. If you created via CLI or Cloidformation it does not. So if someone has created a batch env in your account before it will probably be there but if you are setting up in a new account you might have no clue whu things arent working. Compute Environments and JobSpecs that use Fargate are configured differenrly, requiring a extra IAM role, from EC2 batch envs.
      - dotancohen 4y ago
        Thank you! You've probably saved me much head scratching down the line.
bilekas 4y ago
Not sure why but I laughed out loud reading this. Edit : I've ran into this but it's considered a feature not a bug!
iasay 4y ago
There's loads of stuff like this. If you detach an EC2 instance from a virtual tape library and destroy it you can't delete the tape library any more. Even AWS support couldn't delete it. This is fine until you have 60TB of tapes online and are paying for it. Fortunately we found an EBS snapshot of the VM.
- TedDoesntTalk 4y ago
  This is my worst nightmare. I had a situation like this where I was able to essentially ignore the entire AWS account and cancel the attached credit card. Amazon somehow linked this AWS account to my retail Amazon.com account and started booking my personal credit card.
Marazan 4y ago
Even better if you have your cloud formation with the iam role in the same stack and there is something wrong the the rollback can happen such that the iam role gets deleted standing the compute environment and failing the rollback.
- zxcvbn4038 4y ago
  Update the stack to use a different role with cloudformation in its trust policy, then you can delete the stack with the new policy. Has to be two operations to work.
zxcvbn4038 4y ago
Try logging into the console as root, that lets you delete anything. You can also try updating the cluster to have a new role.
- orf 4y ago
  That’s… not the issue. The cluster gets stuck in a deleting state forever. And at which point you’re unable to update it.
  - zxcvbn4038 4y ago
    Time to open a support ticket, I’ve had a few things get into a bad state where they could not be updated or deleted, the backend team can step in, but first line support need to see the usual solutions fail before they escalate to them.
jiggawatts 4y ago
This is because neither AWS nor Azure use referential integrity in any of their "cloud scale" databases. For example, Azure uses some hideous JavaScript-based document DB where things like renames, moves, and deletes are hit & miss at best. A never-ending whack-a-mole of bugs and issues. Remember boys and girls: Being "cloud scale" means data corruption and referential integrity violation!
- orf 4y ago
  I don’t think it is - the issue is that the Batch service needs to assume your role in order to clean up associated cluster resources like auto scaling groups. If the role is deleted, it can’t do this.
  - jiggawatts 4y ago
    That's what referential integrity violation means: an essential related piece of information can be deleted without the parent/using object also being deleted.
    - orf 4y ago
      Sure, in an abstract sense maybe, but in a cross-service context it’s got nothing to do with their “cloud scale databases”. In fact, being able to delete a role without removing any associated resources is a feature, not a bug. And how would you even ensure referential integrity in this case - you would achieve the same effect by modifying the assume role policy but keeping the role around. You could craft a policy that only allows Batch to assume the role on a Tuesday for example.
      - XorNot 4y ago
        Wedging a cloud resource so you can't delete it is always a bug.
        orf 4y ago
        Yes, in the service that provides the AWS resource. Because it didn’t handle the fact that the role it’s using might be deleted or otherwise rendered un-assumable for a variety of different reasons at any point in time. Which is a feature. Not a bug.
bergenty 4y ago
Sounds like it’s a bug in AWS nuke as well then.
suifbwish 4y ago
The idea that we can actually delete anything from a non physical system the hyper visor of which we do not control is absolute nonsense. Unless you have control over the physical system and the hypervisor all you can do is destroy your ability to access the information. We can never have confidence of how it truly works back end.

by orf 4y ago

Shout out to AWS batch, where if you delete the role assigned to a compute cluster the cluster itself becomes impossible to delete. Found this out after using AWS nuke

OJFord 4y ago

That's a weird one, but surely an aws-nuke bug? It must already use a deliberate order - there's plenty of resources that need anything linked/constituent deleted first - so that order is/was just not correct for those?

bilekas 4y ago
Nah I've seen this too. It is considered an expected behavior. In a way it makes sense. But it's not well documented.
- OJFord 4y ago
  But that's what I mean, if that's how it works, nuke needs to account for it. I assume it already does for other dependencies, because there are a lot of them - including that do make more (obvious) sense. You're not the only one to complain about the documentation though, so it's easy to see how aws-nuke apparently missed it!
Marazan 4y ago
No, Batch's interaction with IAM roles and permission is super weird and not at all documented. It is easy to screw it up.
- dekhn 4y ago
  Aws batch is a terrible product
  - aaaaaaaaata 4y ago
    AWS is a terrible product, unless you happen to be in specific corners they've bothered to document/make their UI architecture not a horror show.
- OJFord 4y ago
  What I mean is 'yes that's a weird interaction' but, if that's how the AWS service works (however weird) then it's an aws-nuke bug that the ordering isn't correct (to account for that weirdness)? Though 'not at all documented' makes that harder of course. I haven't used it.
- dotancohen 4y ago
  I've never used AWS Batch but heard only good things about it. If you could elaborate - i.e. warn me - I would very much appreciate it. What should I know before using AWS Batch?
  - Marazan 4y ago
    First is the problem as described. If youndelete the ComputeEnvironment Role you can't delete the compute environment itself. If you male a mistake setting up the Batch ComputeEnvironment using cloidformation and it rollbacks then the cloidformation error message is useless (resource failed to stabilise) and no trace is left behind for you to check. Two, Batch needs to use servicelinkedroles. If you create an AWS thing that needs a servicelinkedrole via the console it gets creates for you automatically. If you created via CLI or Cloidformation it does not. So if someone has created a batch env in your account before it will probably be there but if you are setting up in a new account you might have no clue whu things arent working. Compute Environments and JobSpecs that use Fargate are configured differenrly, requiring a extra IAM role, from EC2 batch envs.
    - dotancohen 4y ago
      Thank you! You've probably saved me much head scratching down the line.

bilekas 4y ago

Not sure why but I laughed out loud reading this. Edit : I've ran into this but it's considered a feature not a bug!

iasay 4y ago

There's loads of stuff like this. If you detach an EC2 instance from a virtual tape library and destroy it you can't delete the tape library any more. Even AWS support couldn't delete it. This is fine until you have 60TB of tapes online and are paying for it. Fortunately we found an EBS snapshot of the VM.

TedDoesntTalk 4y ago
This is my worst nightmare. I had a situation like this where I was able to essentially ignore the entire AWS account and cancel the attached credit card. Amazon somehow linked this AWS account to my retail Amazon.com account and started booking my personal credit card.

Marazan 4y ago

Even better if you have your cloud formation with the iam role in the same stack and there is something wrong the the rollback can happen such that the iam role gets deleted standing the compute environment and failing the rollback.

zxcvbn4038 4y ago
Update the stack to use a different role with cloudformation in its trust policy, then you can delete the stack with the new policy. Has to be two operations to work.

zxcvbn4038 4y ago

Try logging into the console as root, that lets you delete anything. You can also try updating the cluster to have a new role.

orf 4y ago
That’s… not the issue. The cluster gets stuck in a deleting state forever. And at which point you’re unable to update it.
- zxcvbn4038 4y ago
  Time to open a support ticket, I’ve had a few things get into a bad state where they could not be updated or deleted, the backend team can step in, but first line support need to see the usual solutions fail before they escalate to them.

jiggawatts 4y ago

This is because neither AWS nor Azure use referential integrity in any of their "cloud scale" databases. For example, Azure uses some hideous JavaScript-based document DB where things like renames, moves, and deletes are hit & miss at best. A never-ending whack-a-mole of bugs and issues. Remember boys and girls: Being "cloud scale" means data corruption and referential integrity violation!

orf 4y ago
I don’t think it is - the issue is that the Batch service needs to assume your role in order to clean up associated cluster resources like auto scaling groups. If the role is deleted, it can’t do this.
- jiggawatts 4y ago
  That's what referential integrity violation means: an essential related piece of information can be deleted without the parent/using object also being deleted.
  - orf 4y ago
    Sure, in an abstract sense maybe, but in a cross-service context it’s got nothing to do with their “cloud scale databases”. In fact, being able to delete a role without removing any associated resources is a feature, not a bug. And how would you even ensure referential integrity in this case - you would achieve the same effect by modifying the assume role policy but keeping the role around. You could craft a policy that only allows Batch to assume the role on a Tuesday for example.
    - XorNot 4y ago
      Wedging a cloud resource so you can't delete it is always a bug.
      - orf 4y ago
        Yes, in the service that provides the AWS resource. Because it didn’t handle the fact that the role it’s using might be deleted or otherwise rendered un-assumable for a variety of different reasons at any point in time. Which is a feature. Not a bug.

bergenty 4y ago

Sounds like it’s a bug in AWS nuke as well then.

suifbwish 4y ago

The idea that we can actually delete anything from a non physical system the hyper visor of which we do not control is absolute nonsense. Unless you have control over the physical system and the hypervisor all you can do is destroy your ability to access the information. We can never have confidence of how it truly works back end.