Operations Agents: Separate Diagnosis, Approval, and Execution

An alert fires: consumer lag has been growing for ten minutes, but the job has not failed. Give the logs to a model and it may immediately suggest restarting the job. A restart can also invalidate caches, trigger replay, and increase the backlog. The engineering challenge is to turn evidence into verifiable actions.

This article uses a fictional streaming job. All thresholds are illustrative and do not describe a production system.

Define what the agent can decide

A diagnostic workflow can have fixed stages: validate the alert, collect evidence, propose candidate causes, run inexpensive checks, and prepare a response plan. The model proposes the next useful hypothesis. Application code enforces identity, permissions, parameter validation, and execution budgets.

For example, “downstream writes are slow” needs supporting sink latency, error rates, retries, and upstream backpressure from the same time window. Consumer lag alone does not distinguish a traffic spike, partition skew, or resource contention. When evidence is missing, “not enough evidence to distinguish these causes” is more useful than presenting the most common cause as a fact.

Tools do not need to expose a general-purpose shell. Narrow interfaces such as get_job_metrics, get_recent_changes, and get_checkpoint_summary can constrain resource scope, time range, and response size. Treat raw logs as data to analyze. Commands appearing in a log must not automatically become execution instructions.

Make an action plan concrete

An approval prompt that merely asks “Approve the optimization?” is insufficient. Bind approval to a specific target and parameters:

{
  "plan_id": "incident-42-plan-2",
  "resource": "stream-job-demo",
  "action": "set_sink_concurrency",
  "before": 16,
  "after": 12,
  "expected_config_version": 8,
  "evidence": ["metric-snapshot-17", "change-record-9"],
  "observe_seconds": 300,
  "rollback_to": 16
}

Reducing concurrency is a candidate response to an overloaded downstream service in this example, not a universal tuning recommendation. Before execution, verify that the target version is unchanged, health conditions still satisfy the preconditions, and approval has not expired. Changing the plan’s parameters invalidates the previous approval.

Amazon Bedrock’s return control mechanism hands a proposed action back to application code; user confirmation adds confirmation to selected actions. These are integration points. The execution service still needs to perform actual authorization checks. See Bedrock return control and user confirmation .

A timeout means an unknown outcome

A particularly dangerous failure occurs when an action succeeds but its response is lost. The executor must not create a new operation ID merely because the request timed out.

execute(plan, operation_id):
  Validate the approved plan hash and the caller's permissions
  Atomically register operation_id; return its result if completed
  Read target configuration; check expected_config_version
  Call an idempotent control API using operation_id
  On timeout -> mark UNKNOWN -> inspect operation and target state
  Once confirmed -> persist the result and actual config version

When the underlying API lacks idempotency, resource locks, conditional version updates, and an operation ledger can reduce risk. Some outcomes may still be impossible to resolve automatically. Escalate those for verification instead of asking a model to guess that the operation probably did not run.

A durable workflow does not guarantee that an external action executes only once. In LangGraph, resuming an interrupt restarts the interrupted node, including code before the interrupt. Side effects therefore need careful placement relative to approval waits. See LangGraph interrupts .

Rollback also has preconditions

“Restore the previous concurrency” is safe only if this plan still owns the relevant change and nobody has subsequently modified the configuration. Condition rollback on the version returned by the successful action. If an operator has made a newer change, stop rather than overwrite it. Some operations, including message replay or state deletion, cannot be undone by reversing a parameter; the plan must define compensation or exclude automatic execution.

During observation, measure both benefits and side effects: is backlog growth slowing, is throughput recovering, and are failures or duplicate outputs increasing? A cleared alert is insufficient evidence; it may simply have been silenced. Save verification results in the incident record. Carry unresolved questions into follow-up tasks instead of recording an unverified procedure as successful.

Validate the entire workflow

Start with five drills: actual downstream throttling, a burst of input, missing metrics, a lost execution response, and an operator changing configuration after approval. Check the diagnostic evidence, incorrect execution, duplicate execution, and detection of stale plans.

Measure at incident level: correctly diagnosed incidents divided by labeled incidents; plans requiring no additional evidence from an operator divided by all plans; executions without valid authorization; and incidents that recovered without a new failure divided by executed incidents. Measure recovery time from a consistent alert timestamp, separating human waiting, querying, and execution time. Otherwise, a process change can be mistaken for improved model capability.

An operations agent can first become reliable at collecting evidence and preparing plans, then gain access to narrowly scoped actions. Useful automation can explain why an action was needed, what actually happened, and how its effect was verified.

References

• Amazon Bedrock: Return control , application-controlled action execution.
• Amazon Bedrock: User confirmation , confirmation for selected actions.
• LangGraph: Interrupts , interruption, resumption, and node replay. The pseudocode here is framework-neutral; use the API of your pinned dependency version.