Operating the application

Day Ten #

An uploaded asset now crosses several independently recoverable boundaries. SQS buffers workflow admission, Step Functions owns processing, EventBridge routes terminal facts, and SNS delivers subscriber copies. Each boundary can fail while the browser and the other services keep working.

Today we introduce Amazon CloudWatch as the operator's view of this system. A dashboard brings workflow outcomes, admission health, queue age, delivery failures, and recent lifecycle events together. Alarms turn specific conditions into operational notifications, including recovery messages.

CloudWatch observes these services. Step Functions still owns processing state, EventBridge still owns application routing, SNS still owns notification fan-out, and SQS still owns buffered work. An alarm never changes an asset's DynamoDB status or automatically redrives a message.

Goal #

For each preview, provision one dashboard and alarms for:

• failed and timed-out workflow executions;
• starter Lambda errors and throttles;
• sustained admission and integration queue age;
• visible messages in every existing delivery or admission DLQ;
• failed lifecycle SNS deliveries.

Use the existing rotating customer managed key for a separate encrypted alarm topic. Reuse the optional notificationEmail context value to subscribe the operator, with a separate confirmation. Include recent audit events through a bounded CloudWatch Logs Insights query.

The implementation adds lib/asset-monitoring.ts.

Choose signals that identify a boundary #

A small application can start with the service consoles and manual inspection. That works while someone actively watches each upload. A dashboard becomes useful when an operator needs a common view; alarms become useful when a failure must attract attention without an open browser tab. For a one-off demonstration, individual console graphs may be sufficient.

These initial thresholds are explicit operating choices, rather than AWS service limits:

Signal	Statistic over five minutes	Alarm condition	First response
Workflow ExecutionsFailed	Sum	At least 1 in one period	Inspect the failed execution and its task logs
Workflow ExecutionsTimedOut	Sum	At least 1 in one period	Inspect history and reconcile metadata
Starter Errors or Throttles	Sum, separate alarms	At least 1 in one period	Inspect logs, concurrency, and admission backlog
Admission oldest-message age	Maximum	At least 300 seconds in two periods	Check starter permissions and workflow quotas
Integration oldest-message age	Maximum	At least 3,600 seconds in two periods	Drain the demonstration queue or repair its consumer
Each DLQ's visible message count	Maximum	At least 1 in one period	Repair the named boundary before redrive
Lifecycle SNS NumberOfNotificationsFailed	Sum	At least 1 in one period	Inspect subscription and destination health

Use Sum for event counts and Maximum for queue gauges. Summing repeated queue-depth samples would not tell us how many distinct messages are waiting. Two breaching periods dampen short queue-age spikes; detection still includes service publication and CloudWatch evaluation latency.

The integration queue from Day Nine deliberately has no application consumer. Leaving demonstration messages there will eventually trigger its backlog alarm. Read and delete test messages when finished, or attach a real consumer before applying a production latency target.

Attach monitoring to the existing resources #

Import AssetMonitoring from ./asset-monitoring in lib/stack.ts. Instantiate it after the application resources exist:

new AssetMonitoring(this, 'Monitoring', {
  workflow: assetWorkflow,
  starter: workflowStarter,
  admissionQueue: assetsQueue,
  integrationQueue,
  deadLetterQueues: {
    Admission: deadLetterQueue,
    LifecycleAudit: lifecycleDeliveryDeadLetterQueue,
    NotificationRule: notificationRuleDeliveryDeadLetterQueue,
    IntegrationSubscription: integrationDeliveryDeadLetterQueue,
    ...(emailDeliveryDeadLetterQueue ? { EmailSubscription: emailDeliveryDeadLetterQueue } : {}),
  },
  notificationTopic,
  notificationKey,
  lifecycleLogs: lifecycleLogGroup,
  notificationEmail: notificationEmail ? String(notificationEmail) : undefined,
});

Passing resource constructs lets CDK select the actual deployed dimensions. The workflow metrics use StateMachineArn, Lambda metrics use FunctionName, SQS uses QueueName, and SNS uses TopicName. Other previews cannot contribute to these alarms.

The four mandatory DLQs preserve different recovery instructions. Enabling lifecycle email adds its fifth DLQ and an associated alarm automatically. There are 11 alarms without email and 12 with it.

Keep operational messages on their own topic #

The lifecycle topic carries EventBridge asset envelopes, including detail.status. CloudWatch produces an alarm-state message with a different schema. A separate topic keeps those messages out of the machine integration queue and avoids applying the lifecycle email filter to alarm notifications.

The construct derives an alarm-name prefix from the stack name. Both the SNS policy and the existing key's additional policy statement limit CloudWatch access to alarms under that prefix in the same account and Region:

const alarmPrefix = `${stack.stackName}-Assets-`;
const alarmSource = stack.formatArn({
  service: 'cloudwatch', resource: 'alarm', resourceName: `${alarmPrefix}*`,
  arnFormat: cdk.ArnFormat.COLON_RESOURCE_NAME,
});
const conditions = {
  StringEquals: { 'aws:SourceAccount': stack.account },
  ArnLike: { 'aws:SourceArn': alarmSource },
};
props.notificationKey.addToResourcePolicy(new iam.PolicyStatement({
  principals: [new iam.ServicePrincipal('cloudwatch.amazonaws.com')],
  actions: ['kms:Decrypt', 'kms:GenerateDataKey*'],
  resources: ['*'],
  conditions,
}));
const alarmTopic = new sns.Topic(this, 'AlarmTopic', {
  displayName: 'Asset operational alarms',
  masterKey: props.notificationKey,
  enforceSSL: true,
});
alarmTopic.addToResourcePolicy(new iam.PolicyStatement({
  principals: [new iam.ServicePrincipal('cloudwatch.amazonaws.com')],
  actions: ['sns:Publish'],
  resources: [alarmTopic.topicArn],
  conditions,
}));
if (props.notificationEmail) {
  alarmTopic.addSubscription(new subscriptions.EmailSubscription(props.notificationEmail));
}

The KMS policy's Resource: '*' refers to the key whose policy contains the statement. It grants the named service the listed operations on that key. Topic publication is separately scoped to the alarm topic ARN. These permissions are both necessary for encrypted notifications; an alarm can enter ALARM even if notification publication fails. AWS documents the key permissions needed by service publishers^[1] and source conditions for alarm notifications^[2].

With notificationEmail omitted, the dashboard, alarms, topic, and topic policies still exist. Operators inspect state in CloudWatch; email begins only after configuration and confirmation. With it supplied, confirm the new operational subscription even if the lifecycle subscription was confirmed in Day Nine. Operational email is unfiltered and receives both alarm and recovery transitions.

Email provides awareness rather than a durable incident record. Check alarm action history if delivery does not arrive, then check topic permissions, key permissions, and subscription confirmation. Teams requiring acknowledged paging can add a subscriber to this topic.

Define alarm evaluation once #

The constructor defines period as cdk.Duration.minutes(5). A helper applies the same comparison, missing-data policy, and notification actions:

const alarms: cloudwatch.Alarm[] = [];
const alarm = (
  name: string, metric: cloudwatch.IMetric, threshold: number,
  description: string, evaluationPeriods = 1,
) => {
  const item = new cloudwatch.Alarm(this, name, {
    alarmName: `${alarmPrefix}${name}`,
    alarmDescription: description,
    metric,
    threshold,
    evaluationPeriods,
    datapointsToAlarm: evaluationPeriods,
    comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
    treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
  });
  item.addAlarmAction(new actions.SnsAction(alarmTopic));
  item.addOkAction(new actions.SnsAction(alarmTopic));
  alarms.push(item);
};

CloudWatch treats missing observations here as non-breaching because previews spend much of their time idle. Missing data is not evidence that processing works. A disabled producer or a failed telemetry path may remain quiet; an end-to-end canary would be a separate capability. AWS describes the evaluation behavior in missing-data handling^[3].

SNS actions normally run on transitions into the configured state. An alarm that stays in ALARM does not send a fresh incident email every five minutes. Recovery email means the evaluated condition cleared; the operator must still verify that retained work was handled.

Observe buffered failures as well as exceptions #

The starter catches failures for individual SQS messages and returns batchItemFailures. That invocation can return successfully while messages need another attempt. Lambda Errors alone cannot cover this path. Admission age and the admission DLQ make the waiting work visible. See AWS's partial batch response behavior^[4].

The queue alarms use the same five-minute period, with a maximum statistic:

const admissionAge = props.admissionQueue.metricApproximateAgeOfOldestMessage(maximum);
const integrationAge = props.integrationQueue.metricApproximateAgeOfOldestMessage(maximum);
alarm('AdmissionBacklog', admissionAge, 300,
  'Admission has waited at least five minutes in two periods; inspect starter and workflow quotas.', 2);
alarm('IntegrationBacklog', integrationAge, 3600,
  'Subscriber has waited at least one hour in two periods; drain the demonstration queue or repair its consumer.', 2);

const dlqMetrics = Object.entries(props.deadLetterQueues).map(([name, queue]) => {
  const metric = queue.metricApproximateNumberOfMessagesVisible({ ...maximum, label: name });
  alarm(`${name}NotEmpty`, metric, 1,
    `Inspect ${name} messages and repair its delivery boundary before redrive; export before teardown.`);
  return metric;
});

Monitor DLQ occupancy rather than NumberOfMessagesSent: automatic SQS redrive does not increment that metric in the same way as a direct send. Queue metrics are approximate, and poison messages can affect oldest-message age. Use both backlog and DLQ signals, then inspect actual messages. AWS explains these details in SQS metrics^[5].

A DLQ alarm fires after a message reaches that queue. EventBridge or SNS may still be retrying upstream for an extended period. A queue cannot record a message if its own delivery policy is broken. These alarms cover retained failures and SNS delivery errors; they do not prove every event arrived. Audit history, workflow history, and destination checks remain part of investigation.

Read the dashboard and logs #

The dashboard is named <stack-name>-<region>-assets. CloudWatch dashboard names are account-wide, so including the Region prevents same-named stacks in different Regions from sharing a dashboard accidentally.

It contains an alarm status row, workflow outcome counts, starter errors and throttles, both queue ages, all DLQ depths, lifecycle SNS publication and failure counts, and recent audit events. Graphs reuse the alarm metrics rather than recreating different statistics for display.

The audit widget runs this Logs Insights query against AssetLifecycleEvents:

fields @timestamp, detail.assetId, detail.status, detail.reason
| sort @timestamp desc
| limit 20

For one asset, add filter detail.assetId = "YOUR_ASSET_ID" in the Logs Insights console. The query reads only the lifecycle audit group. It does not inspect private image bytes, presigned URLs, or authentication tokens.

That audit group and the workflow log group already retain seven days in previews and one month in production. Workflow logging remains at ERROR; use Standard Workflow execution history to inspect successful state transitions. Apply a narrow time range when querying. Returning 20 rows limits the display, while scanned bytes still determine Logs Insights cost.

An invalid image follows the explicit rejection branch and ends in Succeed. It appears as a successful execution with a rejected lifecycle event, rather than a failed execution. Step Functions metrics are operational signals delivered on a best-effort basis, so use execution history and DynamoDB when reconciling individual assets. See Step Functions metric semantics^[6].

SNS's failed-delivery metric is also protocol-dependent: retries can contribute differently for different endpoint types. Treat it as a delivery-health signal rather than a count of lost assets. Successful email delivery to an endpoint does not prove a human read it. See SNS monitoring metrics^[7].

Deploy and verify #

Deploy an existing preview. The shared AssetSeriesAuth stack from Day Seven must already exist. Use your configured deployment profile and the same Region throughout:

npm test
PR_NUMBER=123
STACK="Stack-PR${PR_NUMBER}"
REGION=us-east-1

npx cdk deploy "$STACK" \
  --app "npx ts-node bin/preview.ts" \
  --profile dev --region "$REGION" \
  -c pr="$PR_NUMBER" \
  -c notificationEmail=operator@example.com \
  --outputs-file cdk-outputs.json

The existing GitHub preview workflow also passes its optional NOTIFICATION_EMAIL secret into this context value. CDK context is configuration, not secret storage: the email address appears in the deployed subscription and template.

Resolve the dashboard and inspect the preview's alarms:

DASHBOARD=$(aws cloudformation describe-stacks \
  --stack-name "$STACK" --region "$REGION" --profile dev \
  --query "Stacks[0].Outputs[?OutputKey=='AssetDashboardName'].OutputValue | [0]" \
  --output text)

aws cloudwatch get-dashboard --dashboard-name "$DASHBOARD" \
  --region "$REGION"

aws cloudwatch describe-alarms --alarm-name-prefix "${STACK}-Assets-" \
  --region "$REGION" \
  --query 'MetricAlarms[].{name:AlarmName,state:StateValue,reason:StateReason}'

Open that dashboard in CloudWatch. Sign in to the preview, upload a valid PNG or JPEG, and request its private download. After metrics arrive, inspect successful workflow counts and the ready audit event. Upload bytes that disagree with the declared image type and inspect the rejected event; the workflow failure alarm should remain clear.

Confirm the operational SNS subscription, then test the alert channel in this disposable preview:

ALARM=$(aws cloudformation describe-stacks \
  --stack-name "$STACK" --region "$REGION" --profile dev \
  --query "Stacks[0].Outputs[?OutputKey=='AssetWorkflowFailedAlarmName'].OutputValue | [0]" \
  --output text)

aws cloudwatch set-alarm-state --alarm-name "$ALARM" \
  --state-value ALARM --state-reason 'Preview notification drill' \
  --region "$REGION"

aws cloudwatch describe-alarm-history --alarm-name "$ALARM" \
  --history-item-type Action --region "$REGION"

Run this when the alarm is OK, so it creates a transition. Verify the action succeeded and the operational email arrived. CloudWatch resumes metric evaluation and can quickly restore the actual state. This tests state actions, topic permissions, encryption, and email delivery; it does not test the metric threshold.

For a real metric test, resolve AssetsDeadLetterQueueName from stack outputs and obtain its URL with aws sqs get-queue-url. In an otherwise empty disposable preview DLQ, send one diagnostic message using aws sqs send-message --queue-url "$DLQ_URL" --message-body '{"diagnostic":"cloudwatch-drill"}' with the same profile and Region. Wait for metric publication and verify AdmissionNotEmpty enters ALARM. Receive that diagnostic message and delete it using its receipt handle. Confirm recovery after subsequent evaluations. Inspect and preserve any real messages before cleanup; never purge a shared queue to run this drill.

The synthesis tests in tests/monitoring.test.ts cover previews with and without email. They verify metric dimensions, thresholds, missing-data handling, notification actions, encrypted publication permissions, dashboard coverage, and an alarm for every actual DLQ. Deployment verification checks the behavior that synthesis cannot establish.

Cost, quotas, and cleanup #

We introduced one dashboard and 11 standard metric alarms per preview, or 12 with lifecycle email enabled. Multiple open previews multiply those resources even when application traffic is low. Basic service metrics avoid adding custom metric series; dashboard, alarm, query, SNS delivery, and KMS usage still have their own pricing dimensions. Check the deployment Region and account's current allowances on the CloudWatch pricing page^[8].

The existing KMS key is reused. Logs Insights queries incur scan charges; keep the dashboard's six-hour default range or narrow it during investigation. Existing logs continue to incur ingestion and storage charges under their existing retention policies.

Check CloudWatch alarm and dashboard quotas before multiplying the design across a large preview fleet. Alarm API throttling and metric publication latency also matter when running bulk drills. Keep metric dimensions tied to infrastructure resources: an assetId dimension on custom metrics would create an expensive series per asset.

Normal preview teardown removes the dashboard, alarms, alarm topic, subscription, queues, and preview log groups. The existing key enters its seven-day deletion waiting period. Export incident evidence and retained messages first:

npx cdk destroy "$STACK" \
  --app "npx ts-node bin/preview.ts" \
  --profile dev --region "$REGION" \
  -c pr="$PR_NUMBER"

Deletion removes alarm configuration and notifications rather than repairing any outstanding incident. Production retains the existing data resources according to the stack's policies, but these new dashboard and alarm resources follow stack deletion. Keep monitoring deployed for as long as retained resources still serve users.

Well-Architected Framework #

• Operational Excellence: each alarm identifies a boundary and a first investigation step; the dashboard connects metrics to audit events.
• Reliability: backlog and DLQ signals complement execution failures, including partial SQS batch failures.
• Security: the operational topic uses encryption, TLS enforcement, and source-scoped service permissions.
• Performance Efficiency: managed service metrics observe the application without adding work to its upload path.
• Cost Optimization: finite query ranges, resource-scoped dimensions, and an explicit per-preview alarm count make monitoring spend reviewable.
• Sustainability: service metrics and a shared dashboard avoid a continuously polling custom monitoring worker.

Source code #

Reference implementation (opens in a new tab)

References

1. key permissions needed by service publishers (opens in a new tab) · Back
2. source conditions for alarm notifications (opens in a new tab) · Back
3. missing-data handling (opens in a new tab) · Back
4. partial batch response behavior (opens in a new tab) · Back
5. SQS metrics (opens in a new tab) · Back
6. Step Functions metric semantics (opens in a new tab) · Back
7. SNS monitoring metrics (opens in a new tab) · Back
8. CloudWatch pricing page (opens in a new tab) · Back

Continue exploring

• Application events Publish versioned asset lifecycle events Shared tags: aws, cdk · Same collection
• Workflow orchestration Turn asset processing into an observable Step Functions workflow Shared tags: aws, cdk · Same collection
• Notifications and fan-out Fan out asset lifecycle events to durable and human subscribers Shared tags: aws, cdk · Same collection