Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Troubleshoot "Deploy Amazon ECS task definition" step of staging deployment #436

Closed
2 tasks
MattPereira opened this issue Aug 23, 2023 · 2 comments
Closed
2 tasks

Troubleshoot "Deploy Amazon ECS task definition" step of staging deployment #436

MattPereira opened this issue Aug 23, 2023 · 2 comments
Labels
bug Something isn't working feature: Architecture Pertains to project architecture and settings feature: GitHub Actions Pertains to all GHAs (and other GH automations) PBV: dev all issues for engineering roles (devops, backend, frontend, db) role: devops Pertains to cd/ci tasks size: 1pt Can be done in 4-6 hours
Milestone
05 - Project Setu...

Comments

@MattPereira
Copy link
Contributor

MattPereira commented Aug 23, 2023 •
edited
Loading

Overview

Our Deploy to Amazon ECS stage environment workflow is failing because the Build, push image to ECR and depoy to ECS job is failing on the Deploy Amazon ECS task definition step with the following error:

Error: Resource is not in the state servicesStable

Action Items

  • Investigate the deployment's progress in the Amazon ECS console
  • Fix it

Resources

See ECS Service Configurations 
{
    "services": [
        {
            "serviceArn": "arn:aws:ecs:us-west-2:035866691871:service/incubator-prod/civictechjobs-fullstack-stage",
            "serviceName": "civictechjobs-fullstack-stage",
            "clusterArn": "arn:aws:ecs:us-west-2:035866691871:cluster/incubator-prod",
            "loadBalancers": [
                {
                    "targetGroupArn": "arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850",
                    "containerName": "civictechjobs-fullstack-stage",
                    "containerPort": 8000
                }
            ],
            "serviceRegistries": [],
            "status": "ACTIVE",
            "desiredCount": 1,
            "runningCount": 1,
            "pendingCount": 0,
            "launchType": "FARGATE",
            "platformVersion": "LATEST",
            "taskDefinition": "arn:aws:ecs:us-west-2:035866691871:task-definition/civictechjobs-fullstack-stage:88",
            "deploymentConfiguration": {
                "deploymentCircuitBreaker": {
                    "enable": false,
                    "rollback": false
                },
                "maximumPercent": 200,
                "minimumHealthyPercent": 100
            },
            "deployments": [
                {
                    "id": "ecs-svc/5001592045582031217",
                    "status": "PRIMARY",
                    "taskDefinition": "arn:aws:ecs:us-west-2:035866691871:task-definition/civictechjobs-fullstack-stage:88",
                    "desiredCount": 1,
                    "pendingCount": 0,
                    "runningCount": 0,
                    "failedTasks": 3,
                    "createdAt": "2023-08-23T18:20:03.176000-07:00",
                    "updatedAt": "2023-08-23T18:21:02.853000-07:00",
                    "launchType": "FARGATE",
                    "platformVersion": "1.4.0",
                    "networkConfiguration": {
                        "awsvpcConfiguration": {
                            "subnets": [
                                "subnet-08c26edd1afc2b9d7",
                                "subnet-03202f3bf9a24c1a5"
                            ],
                            "securityGroups": [
                                "sg-0dd1d411163070e57"
                            ],
                            "assignPublicIp": "ENABLED"
                        }
                    },
                    "rolloutState": "IN_PROGRESS",
                    "rolloutStateReason": "ECS deployment ecs-svc/5001592045582031217 in progress."
                },
                {
                    "id": "ecs-svc/0327562177077820032",
                    "status": "ACTIVE",
                    "taskDefinition": "arn:aws:ecs:us-west-2:035866691871:task-definition/civictechjobs-fullstack-stage:87",
                    "desiredCount": 0,
                    "pendingCount": 0,
                    "runningCount": 1,
                    "failedTasks": 0,
                    "createdAt": "2023-08-21T19:24:57.421000-07:00",
                    "updatedAt": "2023-08-23T18:21:10.224000-07:00",
                    "launchType": "FARGATE",
                    "platformVersion": "1.4.0",
                    "networkConfiguration": {
                        "awsvpcConfiguration": {
                            "subnets": [
                                "subnet-08c26edd1afc2b9d7",
                                "subnet-03202f3bf9a24c1a5"
                            ],
                            "securityGroups": [
                                "sg-0dd1d411163070e57"
                            ],
                            "assignPublicIp": "ENABLED"
                        }
                    },
                    "rolloutState": "COMPLETED",
                    "rolloutStateReason": "ECS deployment ecs-svc/0327562177077820032 completed."
                }
            ],
            "roleArn": "arn:aws:iam::035866691871:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS",
            "events": [
                {
                    "id": "f611dbbd-dd97-49b7-8890-ec0f111a1e9e",
                    "createdAt": "2023-08-23T18:27:56.521000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "ff172b52-dc9d-4891-806b-dcb9be267a4f",
                    "createdAt": "2023-08-23T18:27:56.442000-07:00",
                    "message": "(service civictechjobs-fullstack-stage, taskSet ecs-svc/5001592045582031217) has begun draining connections on 1 tasks."
                },
                {
                    "id": "07707180-2c6a-423c-b1d4-98f001bf2eba",
                    "createdAt": "2023-08-23T18:27:56.436000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "f65d006f-b7ad-4213-a23d-c1cf2246c0f1",
                    "createdAt": "2023-08-23T18:27:47.377000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has stopped 1 running tasks: (task 26feeb2c30cf4d7cb3ae44fd0e17719b)."
                },
                {
                    "id": "daaa976e-1f13-4319-aa21-bff10a7002bf",
                    "createdAt": "2023-08-23T18:27:47.331000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) (port 8000) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850) due to (reason Health checks failed)."
                },
                {
                    "id": "d1e6d1be-4b2d-4545-a1f9-61ca07c504c8",
                    "createdAt": "2023-08-23T18:26:48.993000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "88429445-28ee-44f8-b42c-28fc65e94b13",
                    "createdAt": "2023-08-23T18:26:20.072000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has started 1 tasks: (task 26feeb2c30cf4d7cb3ae44fd0e17719b)."
                },
                {
                    "id": "b0f6d89a-21ea-426a-abbe-d2615566162c",
                    "createdAt": "2023-08-23T18:24:51.790000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "8877a3cb-75cb-412b-acaa-671b4db3dce4",
                    "createdAt": "2023-08-23T18:24:51.713000-07:00",
                    "message": "(service civictechjobs-fullstack-stage, taskSet ecs-svc/5001592045582031217) has begun draining connections on 1 tasks."
                },
                {
                    "id": "ead331d0-8aec-480d-95e3-7c6056746a19",
                    "createdAt": "2023-08-23T18:24:51.709000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "aa1572cf-879b-409a-80ec-bd7ef7ae5b88",
                    "createdAt": "2023-08-23T18:24:41.806000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has stopped 1 running tasks: (task b86cc6e059ac4ccc836df453acff193b)."
                },
                {
                    "id": "dd99f8a5-97e5-4a33-b525-9d05d8838f53",
                    "createdAt": "2023-08-23T18:24:41.752000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) (port 8000) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850) due to (reason Health checks failed)."
                },
                {
                    "id": "fa8f612b-5d25-4cd3-9600-babe16ea3b61",
                    "createdAt": "2023-08-23T18:23:52.373000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "7fefb6a9-37e4-405b-99fd-be74b8c44768",
                    "createdAt": "2023-08-23T18:23:14.147000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has started 1 tasks: (task b86cc6e059ac4ccc836df453acff193b)."
                },
                {
                    "id": "52e7434b-df8e-4dba-bed0-ddcc7d42639d",
                    "createdAt": "2023-08-23T18:21:51.134000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "adde7aa3-119f-461e-a051-beff822b537d",
                    "createdAt": "2023-08-23T18:21:51.064000-07:00",
                    "message": "(service civictechjobs-fullstack-stage, taskSet ecs-svc/5001592045582031217) has begun draining connections on 1 tasks."
                },
                {
                    "id": "1fa7c123-365a-4cd3-9896-8a72ffe435d5",
                    "createdAt": "2023-08-23T18:21:51.060000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "786d3aa3-b74c-4a52-85db-a745aafc616c",
                    "createdAt": "2023-08-23T18:21:40.859000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has stopped 1 running tasks: (task 03b30478d89a4debb9844a03c82b8ce8)."
                },
                {
                    "id": "714a77e4-80f6-4c4a-b6fb-ca08617ca4ee",
                    "createdAt": "2023-08-23T18:21:40.815000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) (port 8000) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850) due to (reason Health checks failed)."
                },
                {
                    "id": "4629db40-2bfc-44c7-bb44-5548b6036c26",
                    "createdAt": "2023-08-23T18:20:51.208000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "376d9f44-e760-49e0-a455-e9d57bbdaae6",
                    "createdAt": "2023-08-23T18:20:22.468000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has started 1 tasks: (task 03b30478d89a4debb9844a03c82b8ce8)."
                },
                {
                    "id": "0cb11a63-b5b6-4985-9856-1c6f299e0f93",
                    "createdAt": "2023-08-23T15:42:57.852000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has reached a steady state."
                },
                {
                    "id": "607dcb6c-42c7-4d00-8c8d-8be1dd833afc",
                    "createdAt": "2023-08-23T09:42:47.761000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has reached a steady state."
                },
                {
                    "id": "5f311f82-717e-4b3c-bd2d-b48f359977cb",
                    "createdAt": "2023-08-23T03:42:47.105000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has reached a steady state."
                },
                {
                    "id": "c65a4849-6771-436a-9ad3-72aa5255eeaa",
                    "createdAt": "2023-08-22T21:42:45.371000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has reached a steady state."
                },
                {
                    "id": "bb989f2b-5845-4750-8e75-da4457e18d11",
                    "createdAt": "2023-08-22T15:42:09.984000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has reached a steady state."
                },
                {
                    "id": "8fff1bd4-c279-4949-8ed4-38db7cfda79e",
                    "createdAt": "2023-08-22T09:41:36.784000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has reached a steady state."
                },
                {
                    "id": "c85d3180-efab-405d-8a61-e53fe6a78000",
                    "createdAt": "2023-08-22T03:41:14.578000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has reached a steady state."
                },
                {
                    "id": "a0e2e7fc-a373-44ea-b7f8-90af2c00b944",
                    "createdAt": "2023-08-21T21:40:49.477000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has reached a steady state."
                },
                {
                    "id": "64551eac-516b-4182-b4bb-4574aa9a89b7",
                    "createdAt": "2023-08-21T21:40:49.476000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) (deployment ecs-svc/0327562177077820032) deployment completed."
                },
                {
                    "id": "84af6e60-6214-43e5-a85e-c23081f4fb14",
                    "createdAt": "2023-08-21T21:38:04.024000-07:00",
                    "message": "(service civictechjobs-fullstack-stage, taskSet ecs-svc/0234257862204896283) has begun draining connections on 1 tasks."
                },
                {
                    "id": "e74c1bd6-3a5c-48ee-85b1-11ab4d146ea4",
                    "createdAt": "2023-08-21T21:38:04.020000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "43231990-a210-4af6-a633-9ef01b6784de",
                    "createdAt": "2023-08-21T21:37:53.863000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has stopped 1 running tasks: (task 25ae85dadc4648c9a1da50170d725084)."
                },
                {
                    "id": "38e618f9-0b28-40ea-87c8-897b540ca9a0",
                    "createdAt": "2023-08-21T21:36:48.169000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "a2137de0-243c-47ea-8532-344c804a8b7e",
                    "createdAt": "2023-08-21T21:36:10.532000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has started 1 tasks: (task 5b31f0d6c4214b9c8d428d85e135cb61)."
                },
                {
                    "id": "b0b3119b-d84d-40f5-bee4-38e873a207b6",
                    "createdAt": "2023-08-21T21:34:43.743000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "49777f5d-e7d8-4449-9a0e-d0bf214ec251",
                    "createdAt": "2023-08-21T21:34:43.546000-07:00",
                    "message": "(service civictechjobs-fullstack-stage, taskSet ecs-svc/0327562177077820032) has begun draining connections on 1 tasks."
                },
                {
                    "id": "ef8b41f2-91fc-4dd2-917a-c29b6989f392",
                    "createdAt": "2023-08-21T21:34:43.542000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "082fe707-d92e-4740-a00d-3efe1cd20f68",
                    "createdAt": "2023-08-21T21:34:33.439000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has stopped 1 running tasks: (task c5240d966ae54cf0b79924e0e3087012)."
                },
                {
                    "id": "94f710d7-65b6-4a23-b585-c92c57899d3c",
                    "createdAt": "2023-08-21T21:34:33.405000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) (port 8000) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850) due to (reason Health checks failed)."
                },
                {
                    "id": "2e28fce9-e799-4a67-9dbd-4190ff54e4b3",
                    "createdAt": "2023-08-21T21:33:35.204000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "bf720268-9ae6-451a-b71d-06f0402eed50",
                    "createdAt": "2023-08-21T21:33:05.160000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has started 1 tasks: (task c5240d966ae54cf0b79924e0e3087012)."
                },
                {
                    "id": "bf6fe7ef-bd1c-4612-97af-8b2c2abe0a68",
                    "createdAt": "2023-08-21T21:31:47.344000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "b566e92c-0cfa-4ff6-aac3-7f95fb8da654",
                    "createdAt": "2023-08-21T21:31:47.189000-07:00",
                    "message": "(service civictechjobs-fullstack-stage, taskSet ecs-svc/0327562177077820032) has begun draining connections on 1 tasks."
                },
                {
                    "id": "0a142569-9b57-4c1d-8543-2505ebeba005",
                    "createdAt": "2023-08-21T21:31:47.184000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "75285cc9-a6fd-48b6-8c4c-448ec308ec25",
                    "createdAt": "2023-08-21T21:31:37.289000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has stopped 1 running tasks: (task 03b989af2fed4e79b3a8fc22ed7ce9e0)."
                },
                {
                    "id": "8973b429-03aa-44ba-85ee-e3d3ffe1cef3",
                    "createdAt": "2023-08-21T21:31:37.255000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) (port 8000) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850) due to (reason Health checks failed)."
                },
                {
                    "id": "272ee94d-c071-4580-bb32-ef81c333cdf3",
                    "createdAt": "2023-08-21T21:30:39.157000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "18fde06e-b93e-44cd-8a22-018ad361f877",
                    "createdAt": "2023-08-21T21:30:09.603000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has started 1 tasks: (task 03b989af2fed4e79b3a8fc22ed7ce9e0)."
                },
                {
                    "id": "0662c61a-4f71-4c6e-bc3b-156535e7fb46",
                    "createdAt": "2023-08-21T21:28:50.381000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "71e6286b-280b-4e8a-9eb7-c32da64580d8",
                    "createdAt": "2023-08-21T21:28:50.211000-07:00",
                    "message": "(service civictechjobs-fullstack-stage, taskSet ecs-svc/0327562177077820032) has begun draining connections on 1 tasks."
                },
                {
                    "id": "45737304-2d15-44eb-9df1-b9a41d9e1b60",
                    "createdAt": "2023-08-21T21:28:50.206000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "047d6881-9343-4470-8fb9-842f70be36d0",
                    "createdAt": "2023-08-21T21:28:41.273000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has stopped 1 running tasks: (task 127678b8242f4a7ba0de0eb413031899)."
                },
                {
                    "id": "65b69586-faea-4f13-b311-baa22911cc21",
                    "createdAt": "2023-08-21T21:28:41.235000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) (port 8000) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850) due to (reason Health checks failed)."
                },
                {
                    "id": "6969594a-e7f3-4d26-b0c5-e6a23bcf86e5",
                    "createdAt": "2023-08-21T21:27:41.567000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "149522db-b96d-4021-9a4e-cd1b16b742cf",
                    "createdAt": "2023-08-21T21:27:12.343000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has started 1 tasks: (task 127678b8242f4a7ba0de0eb413031899)."
                },
                {
                    "id": "851026d0-fe86-4659-b8d1-9a2605faf31a",
                    "createdAt": "2023-08-21T21:25:19.180000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "e5f9c536-2b35-404f-b5bb-afc0b9dd7a2e",
                    "createdAt": "2023-08-21T21:25:19.007000-07:00",
                    "message": "(service civictechjobs-fullstack-stage, taskSet ecs-svc/0327562177077820032) has begun draining connections on 1 tasks."
                },
                {
                    "id": "fe69b0d9-55b1-44be-8eac-6c8df3581ff6",
                    "createdAt": "2023-08-21T21:25:19.002000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "9095a90a-2582-471f-a46b-1083dbdd1986",
                    "createdAt": "2023-08-21T21:25:09.455000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has stopped 1 running tasks: (task 8675883c0b3641b6ab950aae6bf46403)."
                },
                {
                    "id": "7398979d-1b78-4465-910c-d4a7c7246c8c",
                    "createdAt": "2023-08-21T21:25:09.415000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) (port 8000) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850) due to (reason Health checks failed)."
                },
                {
                    "id": "c6f51010-b3c5-47c0-b279-ad868c975c9d",
                    "createdAt": "2023-08-21T21:24:13.035000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "42396617-59e9-4c2a-8b48-da5fdd0f9318",
                    "createdAt": "2023-08-21T21:23:44.110000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has started 1 tasks: (task 8675883c0b3641b6ab950aae6bf46403)."
                },
                {
                    "id": "0924221e-66b9-4a2d-8e2f-0db6c608cfa2",
                    "createdAt": "2023-08-21T21:22:24.471000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "f3c4c4e9-c264-4962-a805-d00c173eb359",
                    "createdAt": "2023-08-21T21:22:24.316000-07:00",
                    "message": "(service civictechjobs-fullstack-stage, taskSet ecs-svc/0327562177077820032) has begun draining connections on 1 tasks."
                },
                {
                    "id": "5143f629-0f4c-47c5-b1fe-a0dee2bd6270",
                    "createdAt": "2023-08-21T21:22:24.311000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "f5876d69-ca41-4714-89bf-c62e00621650",
                    "createdAt": "2023-08-21T21:22:14.670000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has stopped 1 running tasks: (task ca675154981d4ebd81b7a97747141a2e)."
                },
                {
                    "id": "02eceef2-24ff-4f64-bcfa-ca96ebbd1e10",
                    "createdAt": "2023-08-21T21:22:14.627000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) (port 8000) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850) due to (reason Health checks failed)."
                },
                {
                    "id": "7c5e3ed4-bba1-4261-8123-83c468bb1ec1",
                    "createdAt": "2023-08-21T21:21:16.359000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "cd9c3be5-bd98-42e9-b0c9-cfce4d1227ee",
                    "createdAt": "2023-08-21T21:20:47.975000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has started 1 tasks: (task ca675154981d4ebd81b7a97747141a2e)."
                },
                {
                    "id": "9adb0025-243e-4bf7-be5f-0781d583accd",
                    "createdAt": "2023-08-21T21:19:28.593000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "dbe76dcc-6557-4743-83dd-e441e0dac940",
                    "createdAt": "2023-08-21T21:19:28.445000-07:00",
                    "message": "(service civictechjobs-fullstack-stage, taskSet ecs-svc/0327562177077820032) has begun draining connections on 1 tasks."
                },
                {
                    "id": "d3668a18-0777-4317-82a1-f03e77e2f481",
                    "createdAt": "2023-08-21T21:19:28.441000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "1422fd67-c730-41a5-9c3b-b66859c9a219",
                    "createdAt": "2023-08-21T21:19:17.857000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has stopped 1 running tasks: (task 67aa4deb60c74c14b94b4ea4f6548f39)."
                },
                {
                    "id": "43eb7835-ea89-4170-96f9-6b0781693aac",
                    "createdAt": "2023-08-21T21:19:17.807000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) (port 8000) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850) due to (reason Health checks failed)."
                },
                {
                    "id": "3f6820cd-4574-4fc7-a750-0d556fe4d046",
                    "createdAt": "2023-08-21T21:18:20.325000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "e7078dad-dd9b-40f4-83fe-3aba6fc7fe3d",
                    "createdAt": "2023-08-21T21:17:52.264000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has started 1 tasks: (task 67aa4deb60c74c14b94b4ea4f6548f39)."
                },
                {
                    "id": "6c179e63-2479-40c4-ae99-4453653e4797",
                    "createdAt": "2023-08-21T21:16:34.154000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "59566e09-e694-4649-a96f-58ed2bd523b9",
                    "createdAt": "2023-08-21T21:16:34.015000-07:00",
                    "message": "(service civictechjobs-fullstack-stage, taskSet ecs-svc/0327562177077820032) has begun draining connections on 1 tasks."
                },
                {
                    "id": "ff670917-714a-4c3b-b3dc-1ae9522da445",
                    "createdAt": "2023-08-21T21:16:34.010000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "21142e6e-1ecc-4ce6-84ee-76064743e462",
                    "createdAt": "2023-08-21T21:16:24.134000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has stopped 1 running tasks: (task 5e78743286d8450a9b0bf004d753b11e)."
                },
                {
                    "id": "fa92ae17-9d74-481b-a02f-d9bfededfa5d",
                    "createdAt": "2023-08-21T21:16:24.090000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) (port 8000) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850) due to (reason Health checks failed)."
                },
                {
                    "id": "f16a3f23-3e15-4377-97c3-aaa0ac420939",
                    "createdAt": "2023-08-21T21:15:35.145000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "deb51d39-67f4-40bf-87c8-cc7541311787",
                    "createdAt": "2023-08-21T21:15:05.982000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has started 1 tasks: (task 5e78743286d8450a9b0bf004d753b11e)."
                },
                {
                    "id": "388ac130-21e7-4910-86d7-35fe6fa3181e",
                    "createdAt": "2023-08-21T21:13:46.647000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "29c13baf-1679-432f-9968-a8b5d7120705",
                    "createdAt": "2023-08-21T21:13:46.496000-07:00",
                    "message": "(service civictechjobs-fullstack-stage, taskSet ecs-svc/0327562177077820032) has begun draining connections on 1 tasks."
                },
                {
                    "id": "25ed067d-470d-4382-8010-a1b768f235c4",
                    "createdAt": "2023-08-21T21:13:46.489000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "9fa289da-d1cb-4ad7-b7ee-04eb591f1dc4",
                    "createdAt": "2023-08-21T21:13:35.996000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has stopped 1 running tasks: (task 60b144a32eb04235a7dd52dfaf3f6c6f)."
                },
                {
                    "id": "6320677e-b3e0-4a39-b161-cdfd464692bd",
                    "createdAt": "2023-08-21T21:13:35.955000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) (port 8000) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850) due to (reason Health checks failed)."
                },
                {
                    "id": "6fd023e4-ac97-4385-9a87-557748a60ca9",
                    "createdAt": "2023-08-21T21:12:48.774000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "1ad38052-118e-4cfb-9d09-8a4c60e59996",
                    "createdAt": "2023-08-21T21:12:00.214000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has started 1 tasks: (task 60b144a32eb04235a7dd52dfaf3f6c6f)."
                },
                {
                    "id": "7b1e7513-b515-4e87-aa21-c3d770f5daeb",
                    "createdAt": "2023-08-21T21:10:40.699000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "012f0338-7cf2-4bc5-b1f7-ce74171b3323",
                    "createdAt": "2023-08-21T21:10:40.551000-07:00",
                    "message": "(service civictechjobs-fullstack-stage, taskSet ecs-svc/0327562177077820032) has begun draining connections on 1 tasks."
                },
                {
                    "id": "b4b57099-2729-4c0c-801f-0ecc338b3070",
                    "createdAt": "2023-08-21T21:10:40.547000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "9b417238-a63e-46b5-ae44-108faa5ae765",
                    "createdAt": "2023-08-21T21:10:30.674000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has stopped 1 running tasks: (task dfbca03fec214159a4e23128c1ec6dfe)."
                },
                {
                    "id": "1158f2ea-e9ce-4cbd-b310-df220e7c01e4",
                    "createdAt": "2023-08-21T21:10:30.634000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) (port 8000) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850) due to (reason Health checks failed)."
                },
                {
                    "id": "172541a7-7d00-418b-bd51-6bbea87a1ac1",
                    "createdAt": "2023-08-21T21:09:32.556000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "ab830290-231c-4f13-800e-343383ed59a7",
                    "createdAt": "2023-08-21T21:09:05.319000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) has started 1 tasks: (task dfbca03fec214159a4e23128c1ec6dfe)."
                },
                {
                    "id": "191e0ae4-194e-41fd-970e-0b422515bb3e",
                    "createdAt": "2023-08-21T21:07:38.412000-07:00",
                    "message": "(service civictechjobs-fullstack-stage) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/civictechjobs-fullstack-stage/555a47e25c5a1850)"
                },
                {
                    "id": "46ce6dfa-fdb0-4ca2-b832-9d1f4a5e2729",
                    "createdAt": "2023-08-21T21:07:38.271000-07:00",
                    "message": "(service civictechjobs-fullstack-stage, taskSet ecs-svc/0327562177077820032) has begun draining connections on 1 tasks."
                }
            ],
            "createdAt": "2022-11-08T13:31:00.650000-08:00",
            "placementConstraints": [],
            "placementStrategy": [],
            "networkConfiguration": {
                "awsvpcConfiguration": {
                    "subnets": [
                        "subnet-08c26edd1afc2b9d7",
                        "subnet-03202f3bf9a24c1a5"
                    ],
                    "securityGroups": [
                        "sg-0dd1d411163070e57"
                    ],
                    "assignPublicIp": "ENABLED"
                }
            },
            "healthCheckGracePeriodSeconds": 0,
            "schedulingStrategy": "REPLICA",
            "deploymentController": {
                "type": "ECS"
            },
            "createdBy": "arn:aws:iam::035866691871:user/noor_civictechjobs",
            "enableECSManagedTags": false,
            "propagateTags": "NONE",
            "enableExecuteCommand": true
        }
    ],
    "failures": []
}
@MattPereira MattPereira added bug Something isn't working size: 2pt Can be done in 7-12 hours role: devops Pertains to cd/ci tasks labels Aug 23, 2023
@MattPereira MattPereira self-assigned this Aug 23, 2023
@MattPereira MattPereira added size: 1pt Can be done in 4-6 hours feature: GitHub Actions Pertains to all GHAs (and other GH automations) and removed size: 2pt Can be done in 7-12 hours labels Aug 23, 2023
@MattPereira MattPereira changed the title Troubleshoot "Deploy Amazon ECS task definition" step of deployment action Troubleshoot "Deploy Amazon ECS task definition" step of staging deployment Aug 23, 2023
@JasonEb JasonEb mentioned this issue Aug 24, 2023
Closed
@MattPereira
Copy link
Contributor Author

The Deploy Amazon ECS task definition step hangs for approximately 30 minutes before failing. However, the failure is not preventing changes from being deployed.

@MattPereira MattPereira removed their assignment Aug 28, 2023
@ExperimentsInHonesty ExperimentsInHonesty added the PBV: dev all issues for engineering roles (devops, backend, frontend, db) label Jun 12, 2024
@nooriaali9
Copy link
Member

Closing this issue - since it is outdated/ not a priority anymore - based off of #552

@ExperimentsInHonesty ExperimentsInHonesty added the feature: Architecture Pertains to project architecture and settings label Oct 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working feature: Architecture Pertains to project architecture and settings feature: GitHub Actions Pertains to all GHAs (and other GH automations) PBV: dev all issues for engineering roles (devops, backend, frontend, db) role: devops Pertains to cd/ci tasks size: 1pt Can be done in 4-6 hours
Projects
P: CTJ: Project Board
Archived in project
Milestone
05 - Project Setup - Technical Standa...
Development

No branches or pull requests
5 participants
@nooriaali9 @ExperimentsInHonesty @kcoronel @MattPereira @luca-navarrete