Delete cloudformation stack with AWS::IAM::Policy resource errors intermittently

[robgodfrey]

I have bumped into the following issue when working with moto and cloudformation stacks.

Where a cloudformation template contains both an IAM role and a separate Inline policy, deletion of the stack with moto can intermittently fail.

The following code demonstrates the intermittent failure. The test runs the same code 10 times, but varies the logical resource id for the inline policy to demostrate the issue. When running you should see some tests pass and some tests fail. This is because the delete method in moto/cloudformation/parsing.py iterates over a set of resources to delete, the ordering of which isn't consistent between runs.

The ordering of resources matters. If the role is deleted first then the inline policy is left dangling and when moto tries to delete the inline policy an error like this is raised:

botocore.exceptions.ClientError: An error occurred (NoSuchEntity) when calling the DeleteStack operation: Role inline_policy_issue-Role-47BAI5OU6Q50 not found.

Note if the policy is deleted first, there is no error raised.

Varying the logical resource id for the inline policy resource on different runs causes the ordering of the remaining_resources set to vary between runs.

from moto import mock_aws

import boto3
import pytest


def template_body(policy_logical_id):
    return f"""
---
AWSTemplateFormatVersion: '2010-09-09'

Resources:
  Role:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: StandardTaskPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Sid: AllowCloudwatchLogging
                Effect: Allow
                Action:
                  - logs:CreateLogStream
                  - logs:DescribeLogGroups
                  - logs:DescribeLogStreams
                Resource: '*'

  {policy_logical_id}:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: AdditionalInlinePolicy
      PolicyDocument: 
        Version: "2012-10-17"
        Statement:
          - Sid: AllowCloudwatchLogging
            Effect: Allow
            Action:
              - logs:PutLogEvents
            Resource: '*'
      Roles:
        - !Ref Role
"""


@pytest.mark.parametrize("run_number", range(10))
@mock_aws
def test_delete_cfn_stack(run_number):
    cfn = boto3.client("cloudformation", region_name="us-east-1")

    cfn.create_stack(
        StackName="inline_policy_issue",
        TemplateBody=template_body(f"Policy{run_number}")
    )

    cfn.delete_stack(StackName="inline_policy_issue")

[robgodfrey]

A temporary workaround is to add DeletionPolicy: Retain to the inline policy resource.

[bblommers]

Hi @robgodfrey, thanks for raising this. Sounds like we have two options here:

• refactor the data model so that inline policies are not left dangling
• rework the deletion process to ignore missing inline policies if the request comes from CF

Marking it as a bug either way. Do you want to implement a fix for this yourself?

[robgodfrey]

We could look at the moto.iam.models.InlinePolicy.unapply_policy logic.

    def unapply_policy(self, backend: "IAMBackend") -> None:
        if self.user_names:
            for user_name in self.user_names:
                backend.delete_user_policy(user_name, self.policy_name)
        if self.role_names:
            for role_name in self.role_names:
                backend.delete_role_policy(role_name, self.policy_name)
        if self.group_names:
            for group_name in self.group_names:
                backend.delete_group_policy(group_name, self.policy_name)

The inline policy potentially references users, groups and roles that may or may not exist in the cloudformation delete scenario. In each of the for loops a check could be added to verify existence of the user, group or role before trying to detach the policy.

The checks could alternatively be added to the delete_user_policy / delete_role_policy / delete_group_policy methods if that makes sense.